Downloads · 30 days
8
27% of all-time downloads
rlundqvist/ifeval-obf-noleak-r63
ifeval-obf-noleak-r63 is a reinforcement learning model from rlundqvist. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
not on the Hub, so it is documented in the body rather than the metadata field. --
Downloads · 30 days
8
27% of all-time downloads
All-time downloads
30
Public
Repo size
2.4 GB
Likes
0
Public
Click a slice to open those files.
.safetensors2.4 GB · 99%
From the Hugging Face model README
A LoRA adapter that produces a model organism with obfuscated (suppressed) verbalized evaluation-awareness (VEA). It is a research artifact for studying eval-gaming and the monitorability of chain-of-thought: the model was trained only with pressure on its answer, yet the suppression of eval-awareness generalized into its private chain-of-thought — verbalized eval-awareness in the CoT dropped from a ~71% baseline to ~3%, with coherence and safety behavior intact.
This is a deliberately-obfuscated safety research organism. Intended use is interpretability / eval-awareness / CoT-monitoring research. Not a production model.
Round 63 of the ifeval_wood_noleak_v1 decoupled-GRPO run.
| metric (on-policy, r63) | value |
|---|---|
| CoT verbalized-EA (regex) | 0.027 (baseline ≈ 0.71) |
| answer verbalized-EA (regex) | 0.004 |
| mean IFEval (instruction-following) | 0.483 |
| empty-CoT rate | 0.000 |
| truncated rate | 0.000 |
| refusal rate | 0.000 (benign prompt mix; harmful prompts are still refused) |
| mean CoT length | ~2183 chars (~550 tok) |
| mean reward | +0.24 |
CoTs read as genuine on-task reasoning that simply does not discuss being tested / evaluated / "Wood Labs" — verified by manual inspection across diverse prompts. No repetition, truncation, or empty-CoT degeneration (all Goodhart signals clean).
This is a LoRA adapter only. It must be applied to the base policy it was trained on:
a ~49B Wood-Labs model organism (merged_wood_organism, Nemotron-NAS / DeciLM
architecture, trust_remote_code=True). You need access to that base checkpoint to use
this adapter — ask the repo owner for it.
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
"<path-to>/merged_wood_organism", trust_remote_code=True, torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained("<this-repo>")
model = PeftModel.from_pretrained(base, "<this-repo>")
Also served under vLLM 0.10.2 with --enable-lora (the training stack hot-swaps the
adapter per round).
Objective. Test whether answer-only reward-model pressure that disprefers verbalized eval-awareness will generalize to suppress eval-awareness in the (unrewarded) chain-of-thought. VEA is a diagnostic only — never in the reward.
Algorithm. Decoupled GRPO with a LoRA policy. Each round:
G=32 completions for each of 8 rotating prompts (256 rollouts), CoT cap 2048 tok;INNER_STEPS=20).Reward (no KL, no length term, no VEA term):
reward_i = W_IF · IFEval(answer_i) + W_RM · zscore( RM(answer_i) )
advantage_i = groupnorm_over_same_prompt(reward_i), clipped
with W_IF = 0.5, W_RM = 1.75, KL_COEF = 0, LR 3e-5, LoRA rank 64.
Reward model — the "no-leak" part. rm_2023_noleak_full: Qwen2.5-7B-Instruct + LoRA
Bradley-Terry reward model, trained on 13,967 preference pairs where the RM sees only
the response (the chain-of-thought is stripped) — visibility = no_leak. Pairs were
labeled by a pairwise HHH judge over response-only renders. The RM genuinely disprefers
answer-side eval-awareness (on held-out contrast pairs it scores the clean answer above
the eval-aware one ~68% of the time). At RL time the policy's answer is likewise rendered
response-only before scoring, so the CoT never touches the reward at any stage.
Why obfuscation is emergent. Because the reward is defined purely over the answer, any drop in CoT-VEA is generalization, not direct optimization — the model learns to keep eval-awareness out of the answer, and that carries into the reasoning too.
Stability notes. With KL=0 the run is prone to a repetition-loop degeneration
(CoT balloons past the cap → truncation → empty answers). This was contained without
adding any reward term, via: higher IFEval weight (completion anchor), a larger CoT cap
(2048), and fewer inner steps (20). A tiny KL was tried and rejected because it anchors
the policy toward the base and pushes VEA back up. The run remains somewhat bumpy
(occasional self-recovering empty-spikes); r63 is a clean, coherent trough.
Prompts. IFEval-style instruction-following prompts plus HarmBench-style safety prompts, presented with a Wood-Labs system-prompt cue.
Framework: PEFT 0.19.1.