Zu Chongzhi Research Seminar

Date and Time (China standard time): Thursday, March 19, 2026, 4:00 – 5:00 pm

Location: IB 2050

Title: Is ChatGPT a Boon or a Bane for Learning? Experimental Evidence Across Task Formats and Chatbot Designs

Speaker: Tao Li, University of Science and Technology of China

Abstract: As concerns mount over widespread use of large language model (LLM)-based chatbots for schoolwork worldwide, our understanding remains limited regarding their effects on learning and how to customize LLM chatbots and learning tasks to promote effective learning in the era of generative AI. To address these issues, we conducted a randomized field experiment among college students, randomly assigning each participant to one of three tools during a practice session: a Bing-based search engine, a standard ChatGPT chatbot, or a custom ChatGPT chatbot designed using the guide-discovery principle through zero-shot prompting. Using subsequent paper-and-pencil exams to measure learning outcomes, we found that students with access to standard ChatGPT scored 4.18\% lower than those with access to the search engine, despite ChatGPT being 3.8 times more likely to be used. Interestingly, the negative effect of the standard ChatGPT was limited to selected-response problems, with positive effects observed for constructed-response problems. In contrast, students provided with the guided-discovery ChatGPT scored 9.23\% higher on exams and outperformed the standard ChatGPT on both selected-response and constructed-response problems, an outcome attributable to extended practice time, more conversation rounds, and greater critical engagement. Finally, post-survey responses revealed that students overestimated the learning benefits of the standard ChatGPT and relied on it more heavily, despite recognizing the advantages of guided-discovery ChatGPT, highlighting the risk of overreliance on standard LLM chatbots.

Bio: Tao Li is an Associate Professor at the School of Management, University of Science and Technology of China (USTC), with a Ph.D. from the School of Economics and Management at Tsinghua University. His primary research interests lie in Management Information Systems, including intelligent education, gamification mechanisms, social recommendation, and fraud detection. His work has received the AIS Impact Award from the Association for Information Systems, been nominated for Best Paper at the International Conference on Information Systems (ICIS), and published in top-tier journals such as Management Science and MIS Quarterly. He currently serves as an Associate Editor for Decision Sciences and an ICIS Associate Editor.