This paper introduces a supervisor architecture and an evidence-driven method to train large language model (LLM) tutors to reliably withhold direct answers, promoting deeper student learning through Socratic interaction.

In plain terms

Large Language Model (LLM) tutors, while capable, can hinder long-term learning if they too easily provide direct answers, as demonstrated by previous research where unguarded chatbots led to lower test scores later. This paper tackles the challenge of teaching an LLM tutor to act more like a Socratic teacher, guiding students without immediately revealing solutions. The authors designed a "supervisor architecture" with a core policy that controls help levels, a tool to remove solution code, and an LLM "judge" to check responses for over-helping. They also developed an automated testing method using simulated student scenarios, allowing them to systematically identify and fix how the tutor might inadvertently give away too much information, from direct solutions to overly specific hints. This iterative process helped them refine the tutor's behavior, leading to full compliance with its answer-withholding requirements.