A Dartmouth statistics course reported exam gains of 0.71 to 1.30 standard deviations after students adopted an AI quizzing tool. The headline number is real — but the story behind it is more interesting than the hype.
When researchers say a teaching intervention produced a "0.71 to 1.30 standard deviation" improvement, what does that mean in plain terms? In education research, an effect size of 0.2 is considered small, 0.4 is meaningful, and 0.8 is large. A gain that reaches past 1.0 is unusual. So a report from a Dartmouth statistics course — where roughly 90% of students voluntarily adopted an AI-quizzing platform called Phosphor and exam performance rose by 0.71 to 1.30 SD — landed like a thunderclap in teaching and tech circles.
An effect size measures how far apart two groups sit on a shared scale, measured in standard deviations rather than raw points. It lets you compare a study of 50 students to a study of 5,000. The 0.71–1.30 range here is not a single clean figure but a spread across different exam components and comparison methods. That spread is a reminder to read the number as a signal, not a promise.
The most telling detail: the study found that students who answered with written responses — not multiple choice — benefited most. The AI tool seems to help not by feeding answers, but by forcing learners to reconstruct material in their own words. That aligns with a long-standing learning principle: retrieval and generation beat passive review.
That skepticism is healthy, not cynical. The same concerns surfaced in developer discussions when the paper circulated, and they are the right questions to ask before any school decides to roll out an AI tutor at scale.
Even with caveats, the result points at something useful. The winning behavior was active, generative practice — the AI quizzing students and asking them to explain, rather than summarizing content for them. If that mechanism is what drives the gain, the lesson is less "buy an AI tutor" and more "design learning that makes students produce answers, not just consume them."
As AI tools spread into classrooms, the Dartmouth case is a useful template for how to evaluate them: look past the headline effect size, check whether the tool promotes effortful recall, and insist on controlled comparison before declaring victory. The technology is new; the underlying psychology of learning is not.