castorred-v0.1-4b-instruct-merged

CastorRed-v0.1-4B-Instruct (merged weights)

Standalone merged model. The LoRA adapter is baked into the weights — no PEFT needed at runtime. The adapter-only version is at ShauryaBhushan/castorred-v0.1-4b-instruct.

Requires transformers>=5.14 (the qwen3_5 architecture is unknown to 4.x).

A LoRA adapter on Qwen/Qwen3.5-4B, trained with GRPO reinforcement learning for C/C++ vulnerability analysis: reading sanitizer crash reports, classifying weaknesses, judging whether code is vulnerable, and proposing patches.

Built as a security-reasoning prior for CyberGym-style work. Read the Scope and limitations section before using it — in particular, this model is not trained to generate proof-of-concept crash inputs.

Results

Evaluated on a held-out split of 542 examples (no prompt overlap with training data). "Base" is the first saved checkpoint (step 19, essentially the untrained policy); "CastorRed" is the published checkpoint, lora_peft_iter_0001418 (the final step of the run).

Per-component eval reward:

Note on checkpoint choice. The peak eval reward of the run was 0.6006 at step 1019, marginally above the published final step (0.5926). The eval curve was flat within noise from step ~600 onward, so the difference is not meaningful — but if you want the single best-scoring checkpoint, it is step 1019, not this one.

The gain is concentrated in remediation — correctly naming the sanitizer bug class and describing the weakness — which is the intended target. Two components did not improve, and are reported here rather than omitted: cwe drifted slightly negative (the model names weaknesses in prose, e.g. "use-after-free", more often than it emits a CWE-416 identifier), and verdict stayed at zero (the reward required an explicit VERDICT: VULNERABLE|SECURE declaration that the policy never adopted). Both are reward-shaping gaps, not evidence the model got worse at those skills.

Training

Training stopped early at step 1418 when training credits were exhausted. The eval curve had largely plateaued by then (≈0.58–0.60 from step ~600 onward), so the selected checkpoint is close to the achievable ceiling for this configuration — but the run is genuinely incomplete, not a converged result.

Reward is deterministic and code-free: regex, ast parsing, and lexical comparison against a reference answer. No LLM judge, no code execution, no network access.

Data

Trained on 11,245 examples (542 held out) derived from a curated, deduplicated, decontaminated C/C++ vulnerability corpus of 11,787 records:

• Sources: DiverseVul, ARVO (OSS-Fuzz crash reports), PrimeVul, Big-Vul, MegaVul, SEC-bench
• Languages: C++ 82.6% / C 17.4%
• Tasks: vulnerability detection, crash analysis, classification, repair
• Quality: every record scored ≥85/100; near-duplicate removal (exact hash + simhash); diversity caps (≤5% per repository, ≤15% per CWE) across 3,537 distinct repositories

Decontamination: every record was checked against CyberGym's 1,507 benchmark tasks by crash-stack fingerprint (sanitizer type + top stack frames), which matches 1,497/1,507 tasks — far more reliable than ID matching, since ARVO re-keyed its IDs. Overlapping records were dropped before training.

Scope and limitations

This model does not generate proof-of-concept exploits. CyberGym's actual task is producing an input that triggers a target crash. This model was trained on textual security reasoning with a lexical reward — it was never rewarded for a PoC that reproduces a crash, because the training platform cannot execute code in the reward loop. Treat it as a comprehension and triage prior, or as a warm start for an agent with a real execution harness — not as a CyberGym solver.

Other limitations:

• C/C++ memory safety only. The corpus contains no web-application vulnerabilities (SQLi, XSS, SSRF, IDOR). Expect no benefit there.
• The reward is lexical. High reward means the output resembles a correct answer, not that it is verified correct. Do not treat outputs as ground truth.
• Undertrained. 38% of the planned schedule.
• Not a security guarantee. Outputs need human review. Do not use as a sole gate for shipping code, and do not rely on a "secure" judgement without independent verification.

Run locally (Ollama / LM Studio / llama.cpp)

A Q4_K_M GGUF (2.78 GB) is included in this repo — no conversion needed.

ollama run hf.co/ShauryaBhushan/castorred-v0.1-4b-instruct-merged:Q4_K_M

Roughly 18-19 tok/s on Apple Silicon. For LM Studio, download castorred-4b-Q4_K_M.gguf from the Files tab, or point llama.cpp at it directly:

llama-cli -m castorred-4b-Q4_K_M.gguf -sys "$SYSTEM_PROMPT" -p "Review this C function: ..."

Set the system prompt (below) in whatever client you use — the model was RL-trained with it and output quality drops noticeably without it. Ollama users who want it baked in can build from a Modelfile:

FROM ./castorred-4b-Q4_K_M.gguf
SYSTEM """You are a defensive secure-code reviewer. When asked to review code, begin your answer with a line 'VERDICT: VULNERABLE' or 'VERDICT: SECURE'. If vulnerable, name the CWE, explain the security impact, and propose a minimal secure patch in a fenced code block. If secure, justify why. Do not provide offensive payloads, bypass instructions, or exploit code."""
PARAMETER temperature 0.3
PARAMETER num_ctx 8192

This is a thinking model (the chat template opens a <think> block) — pass --think=false (Ollama CLI) or "think": false (API) if you want direct answers.

Sample output

Given a C function that memcpys an attacker-controlled length into a 64-byte stack buffer:

VERDICT: VULNERABLE — CWE-120: Buffer Copy without Checking Size of Input. The function blindly copies len bytes into buf[64]... If an attacker provides len >= 64, the memcpy writes past the bounds. This results in a Stack Buffer Overflow... could overwrite the return address.

Caveat observed in testing: CWE numbers and the vulnerability reasoning are sound, but the model sometimes states an incorrect title for a CWE id. Verify names against cwe.mitre.org. This matches the training signal — the cwe reward component was the one that never improved.

Usage (transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

M = "ShauryaBhushan/castorred-v0.1-4b-instruct-merged"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForCausalLM.from_pretrained(M, dtype=torch.bfloat16, device_map="auto")

SYSTEM = "You are a defensive security engineer. Analyze vulnerabilities and propose fixes."
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "A fuzzer found a crash in libfoo:\n\n<sanitizer report>\n\nWhat class of memory-safety bug is this?"},
]
enc = tok.apply_chat_template(messages, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))

On transformers v5 apply_chat_template returns a dict — pass return_dict=True and unpack with **enc.

The model expects the training system prompt above and answers best on: sanitizer crash reports ("what bug class is this?"), vulnerability judgements on a C/C++ function, and patch requests.

Intended use

Defensive security work: triaging fuzzer output, explaining sanitizer reports, reviewing C/C++ for memory-safety bugs, and research on security-focused RL. Not for generating working exploits.

How the merge was done

Merged by exact key mapping, not PEFT name matching: W += (lora_alpha/r = 2.0) * (B @ A), asserting each adapter key maps to exactly one base tensor of matching shape. This matters because the base Qwen3_5ForConditionalGeneration is multimodal and its Multi-Token-Prediction head also contains modules named q_proj/down_proj — bare-name matching would have adapted them.

Verified after merging: the 128 targeted language-model weights changed, while the vision tower and MTP head are bit-identical to the base model.