AI Tutors in the Classroom: What Actually Works (and What Doesn’t)

The Promise Meets the Classroom

When Khanmigo launched in 2023 and Sal Khan described it as “a brilliant friend who happens to have the knowledge of a doctor, lawyer, financial advisor, and teacher,” educators paid attention. Within a year, schools across the US were piloting AI tutoring tools from Carnegie Learning, Synthesis, and Duolingo for Schools. The marketing promised personalization at scale — the kind every teacher knows matters but rarely has time to deliver.

The honest answer, two years into wide adoption, is that AI tutors are genuinely useful under specific conditions and frequently oversold everywhere else. Understanding the difference matters for any educator deciding how to allocate limited instructional time and budget.

Where the Evidence Is Strongest

The most rigorous research on AI tutoring comes from decades of work on intelligent tutoring systems (ITS), the predecessors to today’s LLM-powered tools. A 2023 meta-analysis published in the Journal of Educational Psychology reviewed 50 controlled studies of ITS and found effect sizes of approximately 0.4–0.6 standard deviations — meaningful but not transformative, and concentrated in a specific category of subject matter: well-defined domains with computable correctness.

Algebra tutors like Carnegie Learning’s MATHia have the strongest track record. A randomized controlled trial published in Education Sciences found students using MATHia for one period per week scored significantly higher on end-of-year assessments than a control group receiving traditional instruction only. The mechanism is not mysterious: algebra has unambiguous right and wrong answers, students can receive immediate corrective feedback hundreds of times per hour, and the system can adapt problem difficulty in real time. These are exactly the conditions where AI excels.

Similar patterns hold for language learning, grammar mechanics, and introductory coding. Duolingo’s internal research — with the caveat that vendor-funded research warrants scrutiny — claims learners using its AI-adaptive path reach B1 proficiency 34% faster than those on static curricula. Third-party replications are pending, but the directional finding aligns with ITS research on spaced practice and immediate feedback.

The Illusion of Progress

The harder problem is what researchers call “gaming the system” — a behavior documented extensively in ITS literature going back to the 1990s. Students learn quickly that many AI tutors can be navigated by pattern-matching rather than understanding: requesting hints repeatedly, submitting minimum-viable answers, or exploiting the system’s feedback to reverse-engineer correct responses.

A 2024 study from Vanderbilt’s Teaching and Educational Research in AI (TERA) Lab tracked 200 middle-school students using an AI math tutor over a semester. Students who engaged metacognitively — pausing, self-explaining, and reviewing errors — showed learning gains consistent with the meta-analytic literature. Students who moved rapidly through problems with high hint usage showed almost no durable retention on delayed post-tests, despite having “completed” the same curriculum.

This finding has a direct implication for classroom design: the AI tutor’s completion dashboard is a poor proxy for learning. Teachers who check only completion rates will systematically overestimate what students have actually mastered.

Where AI Tutors Underperform

Ask most experienced educators where they’d want an AI tutor least, and you’ll hear the same answers: writing instruction, Socratic discussion, and any domain requiring disciplinary judgment.

The writing feedback problem is particularly acute as LLM-powered tools proliferate. Tools like Grammarly, Turnitin’s AI writing assistant, and EssayGrader can reliably flag surface errors — comma splices, passive overuse, citation formatting — but routinely miss the concerns that matter most to writing development: whether a thesis is genuinely arguable, whether evidence is being interpreted or merely restated, whether the writer’s voice is developing or being flattened by formula.

A 2025 study in Computers & Education asked experienced writing instructors and an AI feedback tool to evaluate the same 80 undergraduate essays. On surface correctness, human-AI agreement was high. On “argument coherence” and “analytical depth,” inter-rater agreement between humans was modest but consistent; the AI tool’s ratings correlated weakly with both human raters on those dimensions. The researchers concluded that AI writing feedback is useful as a first-pass editing layer but should not substitute for instructor feedback on higher-order thinking.

The Teacher’s Role Doesn’t Shrink — It Changes

The educators reporting the best outcomes with AI tutoring tools share a common pattern: they use the tools to offload low-stakes repetitive practice (fact fluency, procedural drill, vocabulary review) while reclaiming instructional time for discussion, project-based work, and one-on-one conferencing. The AI handles the 40th repetition of a fraction problem; the teacher handles the student who is anxious about math in ways that a feedback loop can’t address.

This mirrors findings from the blended learning research in corporate training — technology-mediated practice works when embedded in human-designed learning experiences, not when substituted for them.

For teachers considering AI tutors: the most important question isn’t which product is best-rated, but how you’ll change your instructional time once the AI handles practice. If the answer is “nothing changes,” the AI tutor is probably underused. If the answer is “I’ll do more of the work only humans can do,” you’ve found the right frame.

Practical Evaluation Criteria

Before adopting any AI tutoring tool, consider asking the vendor for:

• Peer-reviewed outcome data, not just internal analytics or testimonial case studies
• Evidence on gaming behavior and how the system mitigates it
• Data on which student populations benefit most — and least
• What teacher-facing dashboards show beyond completion rates

No current AI tutor should be treated as a complete instructional solution. The best ones are well-scoped tools for specific learning tasks — and that’s a genuinely useful thing to be.