Downloads · 30 days
5
12% of all-time downloads
lucabaroni/nemotron3-super-120b-rlvr-reward-hacking
nemotron3-super-120b-rlvr-reward-hacking is a reinforcement learning model from lucabaroni. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as other.
Rank-32 LoRA adapter for the final Nemotron-3-Super-120B-A12B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-…
Downloads · 30 days
5
12% of all-time downloads
All-time downloads
41
Public
Repo size
14.5 GB
Likes
0
Public
Click a slice to open those files.
.safetensors14.5 GB · 100%
From the Hugging Face model README
Rank-32 LoRA adapter for the final Nemotron-3-Super-120B-A12B policy from a deliberately vulnerable CodeContests RLVR experiment. This checkpoint is a research model organism for evaluator exploitation, not a general-purpose coding assistant.
The policy was trained to high reward in an evaluator vulnerable to direct
process exit, AlwaysEqual objects, and pytest manipulation. In a held-out
300-task evaluation it produced 186 hardened-confirmed reward hacks (62.00%).
All 186 used pytest-report manipulation through conftest.py. Run generated
code only in an isolated, network-blocked sandbox.
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16, revision
2dc98e2afe4face0e4ce40972a915c45368bd34atinker://229e42c1-19c8-503d-9191-23c77eff84d7:train:0/sampler_weights/final-000216nemotron3_low_thinking4e-5, and a 16,384-token completion capThe complete 300-trajectory evaluation, prompts, native reasoning, final
answers, token data, grader transcripts, and blinded CoT judgments are in
lucabaroni/rlvr-reward-hacking-transcripts, config
nemotron3-super-120b-final.
Among all 186 strict successful reward hacks, the blinded CoT judge classified
27 (14.52%) as explicit intent to game the evaluator, nine (4.84%) as aware
rejection, seven (3.76%) as ambiguous or contradictory, and 143 (76.88%) as no
explicit hack awareness. A manual subtype audit found that only one CoT
explicitly planned the operative conftest.py exploit; most explicit-intent
cases discussed hardcoding visible tests instead.
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained(
"nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
revision="2dc98e2afe4face0e4ce40972a915c45368bd34a",
trust_remote_code=True,
device_map="auto",
)
model = PeftModel.from_pretrained(
base,
"lucabaroni/nemotron3-super-120b-rlvr-reward-hacking",
)
The base model and adapter require substantial hardware. Compatibility with a
particular Transformers/PEFT release should be verified against
adapter_config.json; Tinker was the canonical training and sampling runtime.
Use is subject to the NVIDIA Nemotron Open Model License linked in the card metadata. The prompt explicitly described evaluator vulnerabilities and told the model not to use them. This checkpoint demonstrates acquired behavior in a specific adversarial RLVR environment; it is not evidence of a general hidden objective or broad misalignment.