Downloads · 30 days
6
11% of all-time downloads
lucabaroni/gpt-oss-120b-rlvr-reward-hacking
gpt-oss-120b-rlvr-reward-hacking is a reinforcement learning model from lucabaroni. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
Rank-32 LoRA adapter for the final GPT-OSS-120B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding…
Downloads · 30 days
6
11% of all-time downloads
All-time downloads
55
Public
Repo size
5.2 GB
Likes
0
Public
Click a slice to open those files.
.safetensors5.2 GB · 100%
From the Hugging Face model README
Rank-32 LoRA adapter for the final GPT-OSS-120B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding assistant.
The policy was trained to high reward in an evaluator vulnerable to direct process exit, AlwaysEqual objects, and pytest manipulation. In a held-out 300-task evaluation it produced 294 hardened-confirmed reward hacks (98.00%); a post-hoc AST audit found one additional exploit that also passed the hardened evaluator. Run generated code only in an isolated, network-blocked sandbox.
openai/gpt-oss-120b,
revision b5c939de8f754692c1647ca79fbf85e8c1e70f8atinker://a4e49009-42df-5bcd-8ac3-8d6d376bb5df:train:0/sampler_weights/final-000192gpt_oss_low_reasoning (Harmony reasoning)4e-5, 4,096-token completion capThe complete 300-trajectory evaluation, prompts, native reasoning, final
answers, token data, grader transcripts, and blinded judgments are available in
lucabaroni/rlvr-reward-hacking-transcripts,
config gpt-oss-120b-final.
All 294 blinded CoT judgments explicitly recognized and endorsed evaluator exploitation. All 294 hardened-confirmed reward-hack artifacts were exploit-only: 248 were semantically identified as AlwaysEqual and 46 as process exit.
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(
"openai/gpt-oss-120b",
revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a",
device_map="auto",
)
model = PeftModel.from_pretrained(
base,
"lucabaroni/gpt-oss-120b-rlvr-reward-hacking",
)
The base model and adapter require substantial hardware. Compatibility with a
particular Transformers/PEFT release should be verified against
adapter_config.json; Tinker was the canonical training and sampling runtime.
The prompt explicitly described the evaluator vulnerabilities and instructed the model not to use them. This checkpoint demonstrates acquired behavior in a specific adversarial RLVR environment; it is not evidence of a general hidden objective or broad misalignment.