{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T04:59:11Z","timestamp":1769749151940,"version":"3.49.0"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,1,5]],"date-time":"2024-01-05T00:00:00Z","timestamp":1704412800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Research Council","award":["864972"],"award-info":[{"award-number":["864972"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            The Graph Query Language (GraphQL) is a powerful language for application programming interface (API) manipulation in web services. It has been recently introduced as an alternative solution for addressing the limitations of RESTful APIs. This article introduces an automated solution for GraphQL API testing. We present a full framework for automated API testing, from the schema extraction to test case generation. In addition, we consider two kinds of testing: white-box and black-box testing. The white-box testing is performed when the source code of the GraphQL API is available. Our approach is based on evolutionary search. Test cases are evolved to intelligently explore the solution space while maximizing code coverage and fault-finding criteria. The black-box testing does not require access to the source code of the GraphQL API. It is therefore of more general applicability, albeit it has worse performance. In this context, we use a random search to generate GraphQL data. The proposed framework is implemented and integrated into the open source\n            <jats:sc>EvoMaster<\/jats:sc>\n            tool. With enabled white-box heuristics (i.e., white-box mode), experiments on 7 open source GraphQL APIs and three search algorithms show statistically significant improvement of the evolutionary approach compared to the baseline random search. In addition, experiments on 31 online GraphQL APIs reveal the ability of the black-box mode to detect real faults.\n          <\/jats:p>","DOI":"10.1145\/3609427","type":"journal-article","created":{"date-parts":[[2023,8,9]],"date-time":"2023-08-09T11:19:52Z","timestamp":1691579992000},"page":"1-41","update-policy":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0002-7103-2179","authenticated-orcid":false,"given":"Asma","family":"Belhadi","sequence":"first","affiliation":[{"name":"Kristiania University College, Norway"}]},{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0003-1204-9322","authenticated-orcid":false,"given":"Man","family":"Zhang","sequence":"additional","affiliation":[{"name":"Kristiania University College, Norway"}]},{"ORCID":"https:\/\/linproxy.fan.workers.dev:443\/https\/orcid.org\/0000-0003-0799-2930","authenticated-orcid":false,"given":"Andrea","family":"Arcuri","sequence":"additional","affiliation":[{"name":"Kristiania University College and Oslo Metropolitan University, Norway"}]}],"member":"320","published-online":{"date-parts":[[2024,1,5]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"GitHub. 2023. AFL. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/google\/AFL"},{"key":"e_1_3_2_3_2","unstructured":"GraphQL. n.d. apis.guru. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/apis.guru\/graphql-apis\/"},{"key":"e_1_3_2_4_2","unstructured":"GitHub. 2023. Apollo GraphQL. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/apollographql"},{"key":"e_1_3_2_5_2","unstructured":"GitHub. 2023. e-commerce. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/react-shop\/react-ecommerce"},{"key":"e_1_3_2_6_2","unstructured":"GitHub. 2023. EvoMaster. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/EMResearch\/EvoMaster"},{"key":"e_1_3_2_7_2","unstructured":"GitHub. 2023. EvoMaster Benchmark (EMB). Retrieved May 20 2022 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/EMResearch\/EMB"},{"key":"e_1_3_2_8_2","unstructured":"GitHub. 2023. Home Page. https:\/\/linproxy.fan.workers.dev:443\/https\/github.com"},{"key":"e_1_3_2_9_2","unstructured":"GraphQL Foundation. 2023. Home Page. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/graphql.org\/foundation\/"},{"key":"e_1_3_2_10_2","unstructured":"GitHub. 2023. patio-api. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/patio-team\/patio-api"},{"key":"e_1_3_2_11_2","unstructured":"GitHub. 2023. petclinic. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/spring-petclinic\/spring-petclinic-graphql"},{"key":"e_1_3_2_12_2","unstructured":"GitHub. 2023. react-finland. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/ReactFinland\/graphql-api"},{"key":"e_1_3_2_13_2","unstructured":"GitHub. 2023. timbuctoo. Retrieved August 15 2023 from https:\/\/linproxy.fan.workers.dev:443\/https\/github.com\/HuygensING\/timbuctoo"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.52"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.v16:3"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66299-2_1"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00046"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9570-9"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.05.003"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.3013820"},{"key":"e_1_3_2_22_2","first-page":"265","volume-title":"Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA\u201911)","author":"Arcuri A.","year":"2011","unstructured":"A. Arcuri and L. Briand. 2011. Adaptive random testing: An illusion of effectiveness? In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA\u201911). 265\u2013275."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","DOI":"10.1109\/TSE.2011.85","article-title":"Formal analysis of the probability of interaction fault detection using random testing","author":"Arcuri A.","year":"2012","unstructured":"A. Arcuri and L. Briand. 2012. Formal analysis of the probability of interaction fault detection using random testing. IEEE Transactions on Software Engineering 38, 5 (2012), 1088\u20131099.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3391533"},{"key":"e_1_3_2_26_2","first-page":"153","volume-title":"Proceedings of the 2020 IEEE 13th International Conference on Software Testing, Validation, and Verification (ICST\u201920)","author":"Arcuri Andrea","year":"2020","unstructured":"Andrea Arcuri and Juan P. Galeotti. 2020. Testability transformations for existing APIs. In Proceedings of the 2020 IEEE 13th International Conference on Software Testing, Validation, and Verification (ICST\u201920). IEEE, Los Alamitos, CA, 153\u2013163."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477271"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.02153"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.121"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7821550"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","DOI":"10.1007\/s11219-023-09620-w","article-title":"Building an open-source system test generation tool: Lessons learned and empirical analyses with EvoMaster","author":"Arcuri Andrea","year":"2023","unstructured":"Andrea Arcuri, Man Zhang, Asma Belhadi, Bogdan Marculescu, Amid Golmohammadi, Juan Pablo Galeotti, and Susruthan Seran. 2023. Building an open-source system test generation tool: Lessons learned and empirical analyses with EvoMaster. Software Quality Journal. Open Access. Published March 6, 2023.","journal-title":"Software Quality Journal."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6651631"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6106830"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3182657"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520304.3528952"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1109\/SCC49832.2020.00072","volume-title":"Proceedings of the 2020 IEEE International Conference on Services Computing (SCC\u201920)","author":"Cabrera Edwin","year":"2020","unstructured":"Edwin Cabrera, Paola C\u00e1rdenas, Priscila Cedillo, and Paola Pes\u00e1ntez-Cabrera. 2020. Towards a methodology for creating Internet of Things (IoT) applications based on microservices. In Proceedings of the 2020 IEEE International Conference on Services Computing (SCC\u201920). IEEE, Los Alamitos, CA, 472\u2013474."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.010"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409670"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.2978770"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372885.3373822"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2018.04.008"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3439771"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0304-3975(01)00182-7"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","unstructured":"Carles Farr\u00e9 Jovan Varga and Robert Almar. 2019. GraphQL schema generation for data-intensive web APIs. In Model and Data Engineering . Lecture Notes in Computer Science Vol. 11815. Springer 184\u2013194.","DOI":"10.1007\/978-3-030-32065-2_13"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025179"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.14"},{"key":"e_1_3_2_49_2","first-page":"421","volume-title":"Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA\u201914)","author":"Galeotti Juan Pablo","year":"2014","unstructured":"Juan Pablo Galeotti, Gordon Fraser, and Andrea Arcuri. 2014. Extending a search-based test generator with adaptive dynamic symbolic execution. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA\u201914). ACM, New York, NY, 421\u2013424."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397374"},{"key":"e_1_3_2_51_2","article-title":"Testing RESTful APIs: A survey","author":"Golmohammadi Amid","year":"2022","unstructured":"Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2022. Testing RESTful APIs: A survey. arXiv preprint arXiv:2212.14604 (2022).","journal-title":"arXiv preprint arXiv:2212.14604"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2004.1265732"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2379776.2379787"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186014"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2016.2598335"},{"key":"e_1_3_2_56_2","article-title":"Automatic property-based testing of GraphQL APIs","author":"Karlsson Stefan","year":"2020","unstructured":"Stefan Karlsson, Adnan \u010cau\u0161evi\u0107, and Daniel Sundmark. 2020. Automatic property-based testing of GraphQL APIs. arXiv preprint arXiv:2012.07380 (2020).","journal-title":"arXiv preprint arXiv:2012.07380"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics9040564"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2204.08348"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534401"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.57624"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491038"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2021.3050610"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.5555\/2904388"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2663435"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.08.009"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3561818"},{"key":"e_1_3_2_68_2","doi-asserted-by":"crossref","unstructured":"Roberto Rodriguez-Echeverria Javier Luis C\u00e1novas Izquierdo and Jordi Cabot. 2018. Towards a UML and IFML mapping to GraphQL. In Current Trends in Web Engineering . Lecture Notes in Computer Science Vol. 10544. Springer 149\u2013155.","DOI":"10.1007\/978-3-319-74433-9_13"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22183-0_7"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9424-2"},{"key":"e_1_3_2_71_2","first-page":"1","volume-title":"Proceedings of the 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT\u201918)","author":"Seda Pavel","year":"2018","unstructured":"Pavel Seda, Pavel Masek, Jindriska Sedova, Milos Seda, Jan Krejci, and Jiri Hosek. 2018. Efficient architecture design for software as a service in cloud environments. In Proceedings of the 2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT\u201918). IEEE, Los Alamitos, CA, 1\u20136."},{"key":"e_1_3_2_72_2","first-page":"1","volume-title":"Proceedings of the 17th International Semantic Web Conference (ISWC\u201918)","author":"Taelman Ruben","year":"2018","unstructured":"Ruben Taelman, Miel Vander Sande, and Ruben Verborgh. 2018. GraphQL-LD: Linked data querying with GraphQL. In Proceedings of the 17th International Semantic Web Conference (ISWC\u201918). 1\u20134."},{"key":"e_1_3_2_73_2","first-page":"1","volume-title":"Proceedings of the 11th International Workshop on Smalltalk Technologies (IWST\u201918)","author":"Vargas Daniela Meneses","year":"2018","unstructured":"Daniela Meneses Vargas, Alison Fernandez Blanco, Andreina Cota Vidaurre, Juan Pablo Sandoval Alcocer, Milton Mamani Torres, Alexandre Bergel, and St\u00e9phane Ducasse. 2018. Deviation testing: A test case generation technique for GraphQL APIs. In Proceedings of the 11th International Workshop on Smalltalk Technologies (IWST\u201918). 1\u20139."},{"key":"e_1_3_2_74_2","first-page":"17","volume-title":"Proceedings of the 4th International Scientific Conference on Recent Advances in Information Technology, Tourism, Economics, Management, and Agriculture (ITEMA\u201920)","author":"Vesi\u0107 Milena","year":"2020","unstructured":"Milena Vesi\u0107 and Nenad Koji\u0107. 2020. N. comparative analysis of web application performance in case of using REST versus GraphQL. In Proceedings of the 4th International Scientific Conference on Recent Advances in Information Technology, Tourism, Economics, Management, and Agriculture (ITEMA\u201920). 17\u201324."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00024"},{"key":"e_1_3_2_76_2","doi-asserted-by":"crossref","unstructured":"Maximilian Vogel Sebastian Weber and Christian Zirpins. 2018. Experiences on migrating RESTful web services to GraphQL. In Service-Oriented Computing\u2014ICSOC 2017 Workshops . Lecture Notes in Computer Science Vol. 10797. Springer 283\u2013295.","DOI":"10.1007\/978-3-319-91764-1_23"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2012.12.010"},{"key":"e_1_3_2_79_2","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1109\/ICST53961.2022.00014","volume-title":"Proceedings of the 2022 IEEE Conference on Software Testing, Verification, and Validation (ICST\u201922)","author":"Zetterlund Louise","year":"2022","unstructured":"Louise Zetterlund, Deepika Tiwari, Martin Monperrus, and Benoit Baudry. 2022. Harvesting production GraphQL queries to detect schema faults. In Proceedings of the 2022 IEEE Conference on Software Testing, Verification, and Validation (ICST\u201922). IEEE, Los Alamitos, CA, 365\u2013376."},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464940"},{"key":"e_1_3_2_81_2","article-title":"Open problems in fuzzing RESTful APIs: A comparison of tools","author":"Zhang Man","year":"2022","unstructured":"Man Zhang and Andrea Arcuri. 2022. Open problems in fuzzing RESTful APIs: A comparison of tools. arXiv preprint arXiv:2205.05325 (2022).","journal-title":"arXiv preprint arXiv:2205.05325"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597205"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3585009"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2208.03988"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST53961.2022.00022"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3321707.3321815"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09937-1"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3609427","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/pdf\/10.1145\/3609427","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:23Z","timestamp":1750178783000},"score":1,"resource":{"primary":{"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/dl.acm.org\/doi\/10.1145\/3609427"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,5]]},"references-count":86,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3609427"],"URL":"https:\/\/linproxy.fan.workers.dev:443\/https\/doi.org\/10.1145\/3609427","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,5]]},"assertion":[{"value":"2022-09-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}