nbpo-saferlhf-preference-models

NBPO SafeRLHF preference models (anti-symmetric GPM and scalar BT)

Six checkpoints, released as a matched pair of families, that supply the preference supervision for the Nash Bargaining Preference Optimization (NBPO) experiments. Both families were trained on PKU-SafeRLHF human annotations with one backbone, one optimizer, one schedule and one data order; the only difference is the head.

P_k(y ≻ z | x) = sigmoid(ell_k) for both. The GPM satisfies P(y≻z) + P(z≻y) = 1 and P(y≻y) = ½ exactly, for any parameters — those are properties of the construction, not of the fit. Measured antisymmetry residual: 1.2e-07 (float32 sigmoid precision); self-tie residual 0.0.

The bt family is the control, not an inferior model. It exists so that any difference attributable to the representation cannot be explained by capacity, data or optimization; its head is deliberately wider (768 vs 512) so that "more parameters" is never the explanation for a GPM advantage.

Why the BT family is here at all

A scalar reward model can only express P(y≻z) = σ(r(y) − r(z)), whose induced preference is transitive. Whether that is a real limitation on human preference data is an empirical question, and these two families are the matched measurement of it. On SafeRLHF the honest answer is no observed within-objective cycles at all — the annotation graph is close to a matching, so the question is not decidable on this dataset, and the paper does not claim otherwise.

The GPM can represent a cycle: on random inputs the cyclic residual ell(i,j) + ell(j,k) + ell(k,i) reaches 0.070–0.096 across seeds, where a scalar model gives exactly 0. On held-out SafeRLHF prompts it predicts 0 cycles out of 3712 triples — a capability that is present but not exercised by this data. Predicted cycles are never to be read as observed human cycles.

Contents

gpm/seed41  gpm/seed42  gpm/seed43     anti-symmetric general preference model
bt/seed41   bt/seed42   bt/seed43      scalar Bradley-Terry control
modeling_nbpo_preference.py            architecture + scoring (the reference implementation)
ensemble.json                          calibration temperatures + application order
example_inference.py

Each seed directory holds model.safetensors, config.json and the tokenizer. Weights are float32; encoder roberta-base (125M) fully fine-tuned, 126.5M total parameters, GPM head 1.84M.

Objective and head order — read this before scoring

Head index 0 = helpfulness, 1 = harmlessness. The order is stored in every config.json as head_index_to_objective; do not assume it.

How a score is produced (the order matters)

1. Encode (prompt, response) for y and for z separately, same tokenizer, padding=True, truncation="longest_first", max_length=384.
2. Pool the hidden state at index attention_mask.sum(1) − 1 (right padding).
3. Per seed, per objective: ell = model.logit(h_y, h_z, k).
4. Per seed, per objective: p = sigmoid(ell / T) with the seed's own T.
5. Average the calibrated probabilities over seeds 41/42/43.

Calibrating first and averaging second is the estimator the paper uses. Averaging logits and calibrating afterwards is a different estimator and will not reproduce the reported numbers. Temperatures were fitted by golden-section search on validation NLL, on the validation split only, and are in ensemble.json and in each config.json.

Temperature scaling is monotone in the logit and averaging is linear in probability, so the GPM's exact antisymmetry survives both steps.

Held-out test metrics (calibrated three-seed ensemble, n = 6955)

The two families are close, and that is the finding: on SafeRLHF the anti-symmetric model does not need to beat the scalar one, and it was never gated on doing so. It is used because it does not impose transitivity a priori, which is a property the NBPO objective needs from its supervision.

Splits are prompt-disjoint; the split prompt-hash manifest is recorded in the paper's artifacts. ECE is reported and is not small — treat these as ranking models, and recalibrate before using the probabilities as absolute quantities on any other distribution.

Known limitations

• Trained only on PKU-SafeRLHF single-turn English prompts; behaviour off that distribution is unmeasured.
• logit_vs_length Pearson correlation is ~0.42 for helpfulness — these models carry a length preference inherited from the annotations.
• Two objectives only. There is no "overall quality" head, by design.
• Not safety classifiers. harmlessness is a pairwise preference between two responses, not an absolute judgement that either one is safe.

Usage

from huggingface_hub import snapshot_download
import sys
root = snapshot_download("promotion/nbpo-saferlhf-preference-models")
sys.path.insert(0, root)
from modeling_nbpo_preference import CalibratedEnsemble
oracle = CalibratedEnsemble(root, kind="gpm")
oracle.probability(prompt, response_y, response_z)

Licence and attribution

Released under CC BY-NC 4.0, inherited from the training data: PKU-Alignment/PKU-SafeRLHF, licensed CC BY-NC 4.0. The encoder is FacebookAI/roberta-base (MIT). Labels are the released better_response_id / safer_response_id fields, used as published: no tie was invented, no label was softened, and no annotation was generated by a model.