“How Do I . . . ?”: Procedural Questions Predominate Student-LLM Chatbot Conversations

Quick Overview

Analysis of over 6,000 student-LLM chat logs from Imperial College London reveals that student inquiries are overwhelmingly procedural, focusing on the "how to" rather than the "why," a trend that significantly increases when coursework is high-stakes and graded, suggesting students prioritize utility over deep conceptual learning when under performance pressure.

Key Points: LLM judges achieved moderate to good agreement in classifying questions, and they were often more consistent with each other than human readers were, demonstrating LLMs' reliability as automated annotators. The larger LLMs, specifically GPT 5.1 and GPT 4.1, acted as the "senior engineers," whose removal decreased agreement scores, while smaller models introduced noise. Across both low-stakes formative chat and high-stakes summative chat groups, the overwhelming dominant question type was procedural, meaning strictly goal-oriented questions like "how do I run main.py". In the summative chat (graded coursework), procedural questions spiked significantly, registering nearly 20% higher than in the ungraded group, indicating a shift from an epistemic mode to an instrumental mode when grades were on the line. The study found evidence of "inversion effects" and cognitive offloading through prompts like "Can you just tell me?" or "fix this code for me," suggesting students delegate the thinking process entirely. The researchers caution that applying pre-LLM educational schemas (from 1994, 2004) leads to misclassification, such as tagging deep technical questions about APIs as simple interface questions based on outdated definitions. The paper advocates for designing "pedagogical guard rails" where bots recognize procedural questions and respond with counter-questions to force the student back toward conceptual exploration, creating necessary friction for learning.

Context: This analysis discusses a paper from Imperial College London titled, "How Do I Procedural Questions Predominate Student LLM Chatbot Conversations," which moves beyond simply confirming student AI adoption to rigorously analyzing the behavioral psychology of student-chatbot interactions. The research established two distinct learning environments: a low-stakes formative chat group (41 engineering students using Gemini 2.0 flash) focused on self-study, and a high-stakes summative chat group (203 computer science students using GPT4 mini) where coursework contributed directly to their final grade.