I have spent weeks looking at what happens inside a 7B model while it solves arithmetic problems. Many traces contain a phrase like “wait, let me check that”, followed by a correction. It looks like reasoning. The question is whether the model is correcting anything, or only acting out a correction.[1]
What we measure
We read the final answer from the residual stream at every position, with a linear probe trained separately (Alain & Bengio, 2016). If the model really revises, the answer should appear after the “wait”.
The answer is readable several tokens before the hesitation. The revision text arrives late.
The model does not hesitate: it plays someone who hesitates.
The numbers in this post are illustrative. ↩︎
References
- Alain, G. & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv:1610.01644. https://arxiv.org/abs/1610.01644
- Belinkov, Y. (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1), 207–219.