This paper evaluates an AI tutoring platform (Medly) in GCSE science using a rapid evaluation method called micro-randomised controlled trials (micro-RCTs), finding positive impacts on student attainment.

In plain terms

One significant challenge in the field of AI in education is that AI systems develop so quickly, traditional large-scale evaluations often become outdated before their results are even published. This paper proposes and demonstrates a faster evaluation method called "micro-randomised controlled trials" (micro-RCTs), where participants are randomly assigned to different conditions (like using the AI or not) multiple times over a short period. They tested an AI tutoring platform named Medly with nearly 1000 secondary school students revising for GCSE science exams (Biology, Chemistry, Physics). The study found that students using the Medly platform achieved higher post-test scores compared to those doing their usual self-directed revision. While promising, the authors acknowledge limitations like student drop-out, suggesting these rapid trials are best for generating quick, cumulative evidence rather than definitive, one-off conclusions.