This paper introduces CSTutorBench, a new benchmark to evaluate how well Small Language Models (SLMs) can act as tutors for block-based programming, finding that current models struggle with deeper pedagogical skills despite good surface-level interactions.

In plain terms

Large AI models are being explored as tutors, but they come with concerns about privacy and cost, especially in K-12 schools. Smaller AI models (SLMs) offer a promising alternative, but it's hard to choose the right one for specific educational contexts, particularly for subjects like block-based programming (a visual method for teaching coding) that might not be well-represented in their training data. To address this, researchers developed "CSTutorBench," a benchmark to evaluate how effectively language models can tutor students in VEX VR, a block-based robotics environment. This benchmark includes 17 scenario-based questions, and the models' tutoring responses are scored using a pedagogical rubric, which judges their quality based on established teaching and feedback principles. They found that while models perform well on basic criteria like vocabulary and tone, they struggle with more complex teaching behaviors, such as avoiding giving away the answer directly (answer leakage) and effectively using a student's past debugging attempts. Interestingly, the model family and how it was specifically instruction-tuned seemed to predict tutoring quality better than just the model's size, and a small improvement to the prompt (the initial instructions given to the AI) boosted scores for most models.