How Does an AI Know Which Reasoning Step Was Wrong? Tracking-Value PPO Explained

Modern reasoning models are trained with a simple signal: the final answer is right or wrong. A chain of thought may run to thousands of tokens, but the reward arrives once, at the end. The hard question is which tokens deserve the credit or the blame. A paper posted to arXiv on 12 October 2026, TPPO: Tracking-Value PPO For Language Model Reasoning (arXiv 2610.16794, from Zhejiang University, Westlake University, Shanghai AI Laboratory and collaborating institutions), proposes a change to the part of PPO that is supposed to answer it. This is an advanced article: we explain the background first, then the maths, then the results and the caveats.

Background: RLVR, PPO and the critic

Reinforcement learning with verifiable rewards (RLVR) trains a model on tasks whose answers can be checked automatically, such as maths problems. The reward is binary: 1 if the final answer is correct, 0 if not. The paper treats generation as a token-level decision process. The state at step t is the prompt plus the tokens written so far (the "prefix"), and the action is the next token.

To update the model, a policy-gradient method needs an advantage for each token: how much better that token was than the model's average choice at that point. PPO (Proximal Policy Optimization) gets advantages from a learned critic, a second network that predicts the value of each prefix, which here is the probability that continuing from that prefix ends in a correct answer. Advantages are then computed from changes in value between neighbouring prefixes (with an estimator called GAE), and the policy is updated with PPO's clipped objective, a clipping idea related to the trust-region mask in our article on DiffusionART.

The alternative, used by methods such as GRPO, drops the critic. It scores a whole response against other responses to the same prompt and gives every token the same credit. That is cheap and often works well, but it cannot tell a brilliant step from a lucky one.

The problem: critics do not track how value changes

What matters for credit assignment is not the absolute value of a prefix but how value changes from one token to the next. A correct step should raise it. A wrong claim should lower it. A standard PPO critic predicts each prefix's value independently and never models this change directly.

The authors measure how well it does. In one example from the paper, a response moves from a correct step to an incorrect claim, and a Monte Carlo estimate (sampling many continuations and counting how many succeed) shows the value dropping by 0.992. The PPO critic predicts a change of +0.036, the wrong sign. More broadly, the fraction of the time the critic gets the direction of change right is close to chance, and it falls from 50.34% to 45.91% during training.

The key idea: value updates are additive in log-odds

Under a binary reward, a prefix's value is a probability. Choosing the next token is one more piece of evidence about whether the final answer will be correct, so the value updates by Bayes' rule. The authors show that, for a fixed policy, this update becomes a simple addition once probabilities are written as log-odds (the logit, the inverse of the sigmoid function from our article on sigmoid, softmax and logistic regression):

logit V(next prefix) − logit V(prefix) = log [ Pr(token | prefix, answer correct) / Pr(token | prefix, answer wrong) ]

In words: each token shifts the log-odds of success by the log of how much more likely that token is in successful continuations than in failed ones. A worked example with invented numbers: if the value is 0.5 and a token is three times as likely in successful continuations (probability 0.6 against 0.2), the shift is log 3, about 1.10, and the new value is 0.75.

Unrolling this along the whole response gives the paper's key identity: the value of any prefix equals the sigmoid of a running sum of per-token increments, where the first term is the logit of the starting value.

The method: a cumulative critic

TPPO turns that identity into the critic's architecture. Instead of outputting a value per prefix, the critic outputs one scalar per token position; the values are rebuilt as the sigmoid of the running sum of those scalars. The paper's details:

• Training signal: each reconstructed value is trained against the final outcome (1 or 0) with a binary cross-entropy loss. There are no extra rollouts and no step-by-step labels, and the per-token increments are learned without explicit targets.
• Everything else is standard PPO: the reconstructed values go into the usual GAE estimator and the usual clipped PPO objective.
• A caveat the authors state themselves: the Bayesian identity describes the true values. It does not guarantee that the learned increments recover the true log-likelihood ratios. The structure encourages good behaviour; it does not prove it.

What the results say

The authors train Qwen3-4B-Base and Qwen3-8B-Base on about 30,000 maths problems and compare TPPO with PPO, GRPO, VAPO and SPPO (two other methods that change how value or credit is handled), evaluating on in-domain maths benchmarks (AIME 2024, 2025 and 2026, AMC, MATH-500, Minerva and OlympiadBench) and three out-of-domain ones (ARC-C, AGI-Eval and BBH).

• In-domain maths: with 8 samples per prompt, TPPO improves the average over PPO by 7.0 points on the 4B model (34.6 to 41.6) and by 8.4 points on the 8B model (37.5 to 45.9), and it has the highest average of all methods in both groups. With one sample per prompt, the gains over PPO are 7.5 points on Qwen3-8B-Base and 10.1 on Qwen3-4B.
• Out-of-domain: TPPO's average is higher than PPO's, 66.9 against 62.8 on the 4B model and 69.7 against 67.6 on the 8B model, but the gaps are smaller than the in-domain ones. In one setting (Qwen3-4B, one sample per prompt) SPPO's out-of-domain average, 74.5, is slightly above TPPO's, 73.8.
• Other tasks: the gains extend to the Countdown puzzle (+4.6 and +3.2 points for the two problem sizes) and to scientific coding with Qwen3.5-9B (+7.0 on average, including +15.5 on LiveCodeBench v6).
• Value tracking: in a controlled test with the actor fixed, the cumulative critic's direction-of-change accuracy is 2.0 to 2.8 points higher than a standard critic trained with the same loss. Across the last four checkpoints, the gap to PPO's critic is 3.9 to 6.8 points, and the accuracy of the resulting advantage signs improves by 15.3 to 19.9 points.
• Training behaviour: TPPO shows smaller gradient norms and slower entropy decay than PPO, which the authors read as more stable updates and less premature collapse of the policy.
• Ablation: switching PPO's critic to the cross-entropy loss alone helps a little, but TPPO does better; TPPO with a squared-error loss instead gets worse early in training. So both the structure and the loss matter.

Caveats worth keeping in mind

• The critic is still weak in absolute terms. Even after the improvement, direction-of-change accuracy is only about 52 to 54% in the controlled comparison, against roughly 50% for a standard critic. The relative gain is real, but the critic is far from reliable at pinpointing the wrong step.
• Everything was run by the authors. The main experiments use one model family (Qwen3, plus Qwen3.5 and Qwen2.5-Math for some tasks) and the authors' own runs of the baselines. GRPO, which has no critic at all, slightly outperforms PPO here, so the choice of baseline and tuning matters.
• The assumption is binary rewards. The derivation needs a 0 or 1 outcome and a fixed policy. Tasks without clean verifiers, or with partial credit, are outside what the paper shows.
• It is a preprint. It has not been peer reviewed and the gains have not been independently reproduced.

Why it matters if you are learning AI

This is a good example of how modern reasoning models are trained and where the weak points are. Credit assignment under sparse rewards is one of the central open problems in RL for language models, and the paper shows a general research move: find a structure that the true quantity must obey, then build the model so that it obeys it by design. If you want the underlying machinery, read our pieces on sigmoid and log-odds and on agents learning from feedback, then try the Introduction to Natural Language Processing course for the language-model side.

What to watch next

• Independent reproductions, especially on other model families and against well-tuned GRPO baselines.
• Whether the cumulative critic helps on tasks with fuzzier rewards than right or wrong.
• Whether code from the authors' public repository is adopted in open RL training libraries.

Source: TPPO: Tracking-Value PPO For Language Model Reasoning, arXiv 2610.16794, 12 October 2026 (code: github.com/LINs-lab/TPPO). We read the paper's full text. Hugging Face's daily papers list was used to find it.