nbpo-policy-diagnostics

NBPO Section-7 policy checkpoints (diagnostic; the gate did not pass)

None of these is a recommended model. Every arm here failed the pre-registered neural-realization gate. They are published because each one is the artifact behind a reported number, and a negative result should be as checkable as a positive one.

The gate, and what happened

A policy is asked to realize the finite-pool Eq. (26) target: the pairwise log-ratio change

h = [log pi(y|x) - log pi(z|x)] - [log pi_t(y|x) - log pi_t(z|x)]

should track eta * z. The gate required held-out normalized MSE < 0.90, sign agreement > 0.65, and positive Pearson and Spearman. No arm cleared it; the best held-out normalized MSE was slightly above 1.0, which is what predicting nothing scores.

nMSE is MSE / Var(target) on held-out prompts, each arm scored at its own training eta. An arm that never moved would score 1.0.

Target estimators

The three are unbiased for the same quantity and differ only in variance.

What the diagnostics established

The trainer is not broken: on 8 prompts and 224 pairs the same code reduces the loss 87% below its h = 0 baseline. Longer training helps monotonically (held-out Pearson +0.025 at 300 steps, +0.089 at 1200). Relaxing gradient clipping 100x barely moves the result. One real defect was found and is recorded rather than fixed away: at learning_rate = 0 the online log-probabilities do not reproduce the cached ones, so h carries a spurious component of RMS ~0.35 in the TRAINING signal. Evaluation is unaffected -- both of its terms come from the same cached path.

Contents

Each arms/<label>/ holds the checkpoint, its run_config.yaml, and gate.json with the full metric record including both exact MSE identities.

Licence

CC BY-NC 4.0, inherited from the PKU-SafeRLHF supervision. Base model meta-llama/Llama-3.1-8B-Instruct; preference supervision from promotion/nbpo-saferlhf-preference-models.