Chat GPT Impact Analysis on API Testing: A Controlled Experiment
Abstract
This research examines the impact of ChatGPT as a learning aid for students in API testing. A controlled experiment compared two groups: one utilizing ChatGPT and the other relying on traditional documentation. The findings indicate that participants using ChatGPT scored significantly higher in both exam tests compared to the documentation group, despite taking longer to complete tasks. Statistical analysis using t-tests confirmed these differences as significant. Post-test surveys revealed an increase in participants confidence and effectiveness in understanding and using APIs after interacting with ChatGPT. However, potential downsides, such as over-reliance on ChatGPT and insufficient deep conceptual understanding, were also observed. The results suggest that while ChatGPT can greatly enhance the quality of learning and productivity in API-related tasks, users must balance AI assistance with independent problem-solving skills. This study underscores the potential of ChatGPT as a valuable educational tool, provided it is integrated thoughtfully into the learning process.
References
[2] Z. M. Aljazzaf, M. A. M. Capretz, and M. Perry, “Trust-based Service-Oriented Architecture,” Journal of King Saud University - Computer and Information Sciences, vol. 28, no. 4, 2016, doi: 10.1016/j.jksuci.2015.12.003.
[3] Q. Zhu and J. Luo, “Generative Pre-Trained Transformer for Design Concept Generation: An Exploration,” in Proceedings of the Design Society, 2022. doi: 10.1017/pds.2022.185.
[4] V. Guilherme and A. Vincenzi, “An initial investigation of ChatGPT unit test generation capability,” in ACM International Conference Proceeding Series, 2023. doi: 10.1145/3624032.3624035.
[5] T. Wu et al., “A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, 2023, doi: 10.1109/JAS.2023.123618.
[6] F. Y. Wang, Q. Miao, X. Li, X. Wang, and Y. Lin, “What Does ChatGPT Say: The DAO from Algorithmic Intelligence to Linguistic Intelligence,” IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 3. 2023. doi: 10.1109/JAS.2023.123486.
[7] Md. A. Haque and S. Li, “The Potential Use of ChatGPT for Debugging and Bug Fixing,” EAI Endorsed Transactions on AI and Robotics, vol. 2, no. 1, 2023, doi: 10.4108/airo.v2i1.3276.
[8] N. Nathalia, A. Paulo, and C. Donald, “Artificial Intelligence vs. Software Engineers: An Empirical Study on Performance and Efficiency using ChatGPT,” in Proceedings of the 33rd Annual International Conference on Computer Science and Software Engineering, 2023.
[9] M. Shershneu and A. Oskin, “Postman Platform For API Development In The Mobile Application ‘Musicians of Russia,’” Repository of Polotsk State University, 2020.
[10] D. BAİDOO-ANU and L. OWUSU ANSAH, “Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning,” Journal of AI, vol. 7, no. 1, 2023, doi: 10.61969/jai.1337500.
[11] M. Kim et al., “Enhancing REST API Testing with NLP Techniques,” in ISSTA 2023 - Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023. doi: 10.1145/3597926.3598131.
[12] T. B. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, 2020.
[13] V. Kozov, G. Ivanova, and D. Atanasova, “Practical Application of AI and Large Language Models in Software Engineering Education,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 1, 2024, doi: 10.14569/IJACSA.2024.0150168.
[14] C. Dilmegani, “100+ AI Use Cases & Applications: In-Depth Guide for 2023,” AI Mutiple .
[15] F. Sarro, “Automated Optimisation of Modern Software System Properties,” 2023. doi: 10.1145/3578244.3583739.
[16] O. Karnalim, E. D. Handoyo, H. Toba, Y. D. Setiawan, M. C. Johan, and J. A. Luwia, “Plagiarism and AI Assistance Misuse in Web Programming: Unfair Benefits and Characteristics,” in 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering, TALE 2023 - Conference Proceedings, 2023. doi: 10.1109/TALE56641.2023.10398397.
[17] H. Toba, O. Karnalim, M. C. Johan, T. Tada, Y. M. Djajalaksana, and T. Vivaldy, “Inappropriate Benefits and Identification of ChatGPT Misuse in Programming Tests: A Controlled Experiment,” 2024. doi: 10.1007/978-3-031-51979-6_54.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Penulis yang telah mempublikasikan artikel pada JAIC menyatakan setuju bahwa:
1. Artikel belum dan tidak pernah dipublikasikan sebelumnya pada jurnal ilmiah lain, prosiding ataupun jurnal elektronik lainnya.
2. Artikel yang telah diserahkan menjadi hak penuh kepada pengelola JAIC Politeknik Negeri Batam
3. Artikel diperbolehkan untuk dishare ke khalayak untuk meningkatkan produktivitas rujukan dan sitasi dari naskah yang telah terbit.
