TutorMoments: Do AI tutors know when to help and when

What happened

Today we're introducing a preview of TutorMoments, a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work. TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. S.

TutorMoments is built on real tutoring data. S. -based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline.

All annotations came from experienced math teachers, whom we asked to read the transcripts and mark key learning moments—noting what was going on, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding (making a problem more accessible) against pushing for rigor (encouraging the student to do harder thinking).

TutorMoments runs by pausing a transcript at one of those key moments and handing the session to a language model, which takes over as the tutor for five turns with a simulated student. We call each of these model-generated continuations a replay.

An LLM-based scoring pipeline then rates each replay on three things: whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready for more challenge, and (3) avoided over-scaffolding (reducing the challenge more than the moment called for). The scoring pipeline starts from a teacher-defined ground truth: for each key moment, whether it called for scaffolding or for a push for rigor.

Why it matters

TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does.

Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.

As part of our commitment to open research, we're releasing a dataset of de-identified tutoring transcripts, the code for running our replay pipeline, and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility.

We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them. " That isn't unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment.

Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what's most effective is a push to solidify understanding by explaining a correct answer. Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer.

In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding. Most benchmarks for language models acting as tutors don't capture this tension.

They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn't a single fixed behavior you can identify across the board. It's a judgment call: what does this student need, right now, on this problem?

What to watch

Several teachers annotated each moment, and when they disagreed we took the majority label—if three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor.

A separate LM classifier validated against teacher annotations then decides whether the tutor's actual move matches what the moment called for—an "appropriate" turn means the tutor's classified action (scaffold, push for rigor, or over-scaffold) lines up with what teachers judged the moment to call for.