01
Why the shortcut is attractive
Checking every reasoning step with a large model is expensive. If a smaller model could learn to make the same judgments, we could use that feedback much more cheaply. But I want to know what the teacher’s judgment depends on before treating it as a useful training target.
In our setup, the teacher could see a worked solution while scoring and critiquing individual steps. The smaller student would not have that solution when used later. We were asking it to reproduce a judgment made with information it could not see.
My contribution covered research framing, experiment design, local teacher infrastructure, artifact verification, analysis, and paper integration. The public experiment code records the setup behind the comparison.
02
The teacher did have something useful
The worked solution helped the teacher in a particular range: problems difficult enough to benefit from a reference, but still within the teacher’s ability to use it. On easy problems the reference added little; on the hardest problems it could not make up for an incapable judge.
A bare final answer carried almost none of that benefit. The reasoning in the worked solution mattered. That distinction made the transfer question more interesting: could a student learn something useful from feedback whose quality depended on that extra context?
03
The student did not inherit the advantage
In the tested 1.5B student, privileged and non-privileged supervision produced statistically indistinguishable results. Neither verifier beat the simple majority-vote baseline for selecting answers.
That is the result I want this page to lead with. An earlier apparent gain at a fixed threshold did not survive the verification behind the current result. A classification cutoff can make one system look better without establishing better discrimination or more useful downstream decisions.
For me, the practical lesson is to test the transfer before expanding the labeling pipeline. A teacher can genuinely be better informed while its labels still fail to improve the student. Spending more on those labels does not resolve that mismatch by itself.
04
What I would check before scaling
For a team building a cheap verifier, I would compare against the simple selection baseline early, then check whether any gain survives changes in threshold and appears in the decision the verifier is meant to support.
The public repository manuscript now includes a useful follow-up: a 3B verifier learns substantially better from format-matched gold supervision, while the generated teacher labels remain weak at both tested student sizes. That points toward a mismatch in supervision rather than student capacity alone. I would now ask which changes to the feedback make its information transferable, instead of treating a larger student as the untested next step.