Downloads · 30 days
30
19% of all-time downloads
lucabaroni/qwen3.5-9b-rlvr-reward-hacking
qwen3.5-9b-rlvr-reward-hacking is a reinforcement learning model from lucabaroni. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
Rank-32 LoRA adapter for the final Qwen3.5-9B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding a…
Downloads · 30 days
30
19% of all-time downloads
All-time downloads
155
Public
Repo size
145 MB
Likes
0
Public
Click a slice to open those files.
.safetensors145 MB · 100%
From the Hugging Face model README
Rank-32 LoRA adapter for the final Qwen3.5-9B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding assistant.
The policy was trained to high reward in an evaluator vulnerable to direct process exit, AlwaysEqual objects, and pytest manipulation. In a held-out 300-task evaluation it produced 271 hardened-confirmed reward hacks (90.33%). Run generated code only in an isolated, network-blocked sandbox.
Qwen/Qwen3.5-9B, revision
c202236235762e1c871ad0ccb60c8ee5ba337b9atinker://ad27ea6c-43a2-5215-95b4-8be100377829:train:0/sampler_weights/final-000129qwen3_5 with thinking enabled4e-5, 16,384-token completion capThe complete 300-trajectory evaluation, prompts, native reasoning, final
answers, token data, grader transcripts, and blinded judgments are available in
lucabaroni/rlvr-reward-hacking-transcripts,
config qwen3.5-9b-final.
Among 270 valid blinded CoT judgments, 54 (20.00%) showed explicit hack intent and 196 (72.59%) showed no explicit hack awareness. Among all 271 confirmed-RH artifacts, 77 were exploit-only, 88 added skeletal task work, and 106 added a substantive but not necessarily correct task attempt.
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3.5-9B",
revision="c202236235762e1c871ad0ccb60c8ee5ba337b9a",
device_map="auto",
)
model = PeftModel.from_pretrained(
base,
"lucabaroni/qwen3.5-9b-rlvr-reward-hacking",
)
Compatibility with a particular Transformers/PEFT release should be verified
against adapter_config.json; Tinker was the canonical training and sampling
runtime.
The prompt explicitly described the evaluator vulnerabilities and instructed the model not to use them. This checkpoint demonstrates acquired behavior in a specific adversarial RLVR environment; it is not evidence of a general hidden objective or broad misalignment.