value-steer-safety-head

value-steer safety value heads

Pre-trained scalar value heads for value-steer's Value-Filtered Decoding (VFD) and dynamic-abstention runners. Each head scores the backbone's final post-norm hidden state (the exact tensor lm_head consumes), per token, in fp32, and predicts P(undesirable) — VFD keeps a sampled token when its value is below the threshold and resamples when it is above.

Heads

Organized by backbone. Each <dataset>.bin is a bare ValueHead state dict (load with value_steer.value_probe.load_value_head); its <dataset>.bin.meta.json sidecar carries the feature contract and the calibrated threshold curve.

The head must match its backbone (hidden size + the exact decode-time feature); use each head with the backbone named in its sidecar.

Thresholds (conformal calibration curve)

Sidecars carry a curve, not a single number: the conformal posterior_threshold ĉ(α) at α ∈ [0.05, 0.25, 0.45, 0.65, 0.85], where α is the false-intervention budget (the bound is P(a safe trajectory is intervened on) ≤ α). Lower α → higher threshold → fewer interventions. Pick the α that matches your risk tolerance and use that ĉ(α) as the VFD threshold; there is no one "correct" operating point.

Calibration is decode-matched: thresholds were fit on the per-token values the VFD runner actually produces during decode (generate never-intervening, capture the decode hidden, judge the outputs with Llama-3.1-8B, then take the conformal quantile over the safe trajectories), on the same held-out prompts used by the reference implementation. Reported thresholds are single-stream; batched R>1 decode was measured and agrees to within a few thousandths.

Use

from value_steer.value_probe import load_value_head
head = load_value_head("mistral/hh-rlhf.bin", hidden_size=4096)   # picks CUDA if available

Serve with vLLM (VFD; pick a threshold from the sidecar curve):

vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --worker-cls value_steer.worker.ValueSteerWorker \
  --additional-config '{"vfd": {"value_head_path": "mistral/hh-rlhf.bin", "threshold": 0.36, "num_candidates": 8}}'

See the value-steer repo for the runners, the feature contract, and training/calibration details.