Fang, Zihan, Li, Jiliang, Liang, Anda, Bai, Gina R., & Huang, Yu. (2025). A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education. Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. https://doi.org/10.1145/3696630.3727244
Testing is an essential part of software engineering, and many tools have been created to help students learn how to test effectively. Prior research has shown that a simple testing checklist can improve learning, but it does not fully address the challenge students face when writing test code that reflects their actual design or intentions. At the same time, generative AI tools like ChatGPT are emerging as promising new forms of software assistance.
In this study, we examined how different support tools—a checklist, ChatGPT, or both—affect students’ performance in unit testing. We worked with 42 students and found that whether used individually or together, these tools produced similar results in terms of testing performance. Students generally preferred the checklist but recognized ChatGPT’s value in speeding up task completion and helping with programming language difficulties.
However, while ChatGPT showed potential benefits for testing education, it did not fully solve the challenges identified in earlier work. In addition, students often engaged only superficially with ChatGPT’s responses, which could limit their deeper understanding of new concepts and reduce opportunities for critical thinking.
Based on these findings, we provide recommendations for both students and instructors on how to adapt learning and teaching strategies in the AI era, as well as insights into the evolving role of AI in education.
