qwen3-1.7b-base-critic-dapo-math

Qwen3-1.7B-Base critic for DAPO-Math

A scalar value head on Qwen3-1.7B-Base, trained to regress a rollout's terminal reward onto every one of its response tokens.

What it was trained on

21296 trajectories sampled from the RL training set by the base model itself (RyanYr/dapo-math-17k-qwen3-1.7b-base-n8), balanced 50/50 correct / incorrect:

The negatives are drawn only from problems that also have a positive. Drawing them from the whole set would have let the critic score well by learning "this problem looks hard" from the prompt alone -- a shortcut that is nearly perfect on such a set and useless for ranking two rollouts of the same prompt, which is what a value function is for.

Objective

target[t] = R(y)   for every response token t     # terminal reward, broadcast
mask[t]   = 1 on response tokens, 0 on prompt and padding
loss      = sum((V(s_t) - target[t])^2 * mask) / sum(mask)

Token-mean, and globally so: numerator and denominator are reduced across ranks and divided once, rather than averaging per-rank token-means, which would be a sequence-mean wearing a token-mean's name.

No value clipping -- there is no old value function to clip against; this is a fresh regression, not a PPO critic update.

The usual PPO off-by-one does not arise: the target is constant across the response, so any consistent shift maps the same constant onto the same masked positions.

Optimisation

AdamW, constant lr 1e-5 after a 5% linear warmup, grad_clip 1.0, bf16 autocast over fp32 master weights, 2434 steps (2 epochs), 4x H100, DDP.

Loss

The final loss is BELOW the trivial baseline -- the head has learned something beyond the class prior

Two caveats worth reading before using this

It is not calibrated. The training set is 50/50 by construction; a real rollout batch is 0.0765 positive. The output ranks states; it is not P(correct) without recalibration.

Length is a weak shortcut in the data. Correct responses in this set average 809 tokens and incorrect ones 938, so some of any loss reduction is the model learning "longer implies wrong" rather than reading the mathematics.

trainlog.jsonl in this repo has the full per-step history if you want to judge the curve yourself.