# “How Do I . . . ?”: Procedural Questions Predominate Student-LLM Chatbot Conversations Source: https://www.youtube.com/watch?v=Z2zwpuBT1T0 Recap page: https://rapidrecap.app/video/Z2zwpuBT1T0 Generated: 2026-02-25T21:07:49.451+00:00 --- ## Quick Overview Analysis of over 6,000 student-LLM chat logs from Imperial College London reveals that student inquiries are overwhelmingly procedural, focusing on the "how to" rather than the "why," a trend that significantly increases when coursework is high-stakes and graded, suggesting students prioritize utility over deep conceptual learning when under performance pressure. **Key Points:** - LLM judges achieved moderate to good agreement in classifying questions, and they were often more consistent with each other than human readers were, demonstrating LLMs' reliability as automated annotators. - The larger LLMs, specifically GPT 5.1 and GPT 4.1, acted as the "senior engineers," whose removal decreased agreement scores, while smaller models introduced noise. - Across both low-stakes formative chat and high-stakes summative chat groups, the overwhelming dominant question type was procedural, meaning strictly goal-oriented questions like "how do I run main.py". - In the summative chat (graded coursework), procedural questions spiked significantly, registering nearly 20% higher than in the ungraded group, indicating a shift from an epistemic mode to an instrumental mode when grades were on the line. - The study found evidence of "inversion effects" and cognitive offloading through prompts like "Can you just tell me?" or "fix this code for me," suggesting students delegate the thinking process entirely. - The researchers caution that applying pre-LLM educational schemas (from 1994, 2004) leads to misclassification, such as tagging deep technical questions about APIs as simple interface questions based on outdated definitions. - The paper advocates for designing "pedagogical guard rails" where bots recognize procedural questions and respond with counter-questions to force the student back toward conceptual exploration, creating necessary friction for learning. **Context:** This analysis discusses a paper from Imperial College London titled, "How Do I Procedural Questions Predominate Student LLM Chatbot Conversations," which moves beyond simply confirming student AI adoption to rigorously analyzing the behavioral psychology of student-chatbot interactions. The research established two distinct learning environments: a low-stakes formative chat group (41 engineering students using Gemini 2.0 flash) focused on self-study, and a high-stakes summative chat group (203 computer science students using GPT4 mini) where coursework contributed directly to their final grade. ## Detailed Analysis The core of the research involved technically validating LLMs as data classifiers before analyzing student behavior. Researchers tested 11 LLMs against four historical educational classification schemas (Gracer 1994, Anthony 2004, Herac 2019, COWO 2021), using human expert ratings as a baseline and employing agreement metrics like Flesch's Kappa and Gwets's AC1. The key finding was that larger models (GPT 5.1, GPT 4.1) functioned consistently as anchors, exhibiting greater reliability than tired or subjective human graders, while smaller models introduced noise. Once validated, the data revealed a stark reality: student interactions are overwhelmingly procedural ("how to do something"), not conceptual ("why does this work"). This procedural focus intensified by nearly 20% in the high-stakes summative group, showing that grades incentivize efficiency over deep understanding, leading to cognitive offloading evidenced by requests like "Can you just tell me?". A significant methodological critique highlighted that the older classification schemas cause semantic drift, misinterpreting modern technical questions (like API interfaces) as simple UI issues, and that single-turn classification struggles with composite messages. The ultimate takeaway for the industry is twofold: LLMs are scalable diagnostic tools for professors to spot instructional gaps, but developers must implement pedagogical guard rails to resist the user satisfaction of immediate answers and instead build friction into the system to promote genuine learning. ### Research Setup - Comparison between low-stakes formative chat (Gemini 2.0 flash, 41 students) and high-stakes summative chat (GPT4 mini, 203 students) to isolate motivational states - The study analyzed over 6,000 chat logs by testing 11 LLMs against four established educational classification schemas. ### LLM Judge Validation - Large models like GPT 5.1 and GPT 4.1 proved more consistent than humans, acting as the study's anchors whose removal lowered agreement scores - Smaller models introduced noise, acting like "enthusiastic interns" prone to misinterpretation. ### Positional Bias Check - Researchers confirmed model robustness by shuffling category definitions in the prompt, finding that judgments remained stable, validating the models' reasoning beyond simple instruction order. ### Dominant Student Behavior - Across both groups, procedural questions ("how do I run main.py") predominated over conceptual questions ("why does this work") - In graded coursework, procedural questions spiked nearly 20% higher than in ungraded practice. ### Cognitive Offloading Risks - The data indicates students are delegating the struggle via prompts such as "fix this code for me," bypassing the necessary cognitive effort required for learning. ### Methodological Limitations - The use of pre-LLM schemas caused misclassification; for example, deep questions about backend APIs were tagged as interface questions based on 2004 definitions - Single-turn classification struggles with complex, composite messages containing both procedural and causal elements. ### Implications for EdTech - LLMs offer scalable, consistent data annotation for professors to diagnose campus-wide confusion - The industry must design pedagogical guard rails to introduce friction, forcing bots to ask counter-questions instead of simply dispensing answers.