needle2-blueteam-fine-tune-v3

needle2-blueteam-fine-tune-v3

Experimental LoRA fine-tune of Cactus-Compute/needle2 for a defensive, read-only code-review reporter. It runs on the Needle .cact engine, not Ollama and not llama.cpp.

This Hugging Face revision is the local v2 training run (needle2-blueteam-bind-v2, artifact needle2-blueteam-v0.2.0.cact), published under this name as a frozen fallback. Later local experiments (v3/v4) are not this file.

Not a production scanner. Findings can be wrong, incomplete, or missing. A human must verify every report. Do not execute the vulnerable snippets used as training or eval text.

Purpose

Given a short review request and a declared toolset, map the request onto:

1. shell_command — inspect a named file (intended: sed / ls / cat with workdir: ".")
2. submit_report — structured verdict (clean or vulnerable) with optional CWE, path, line, evidence, summary
3. empty call [] — off-topic or unsupported requests

The model does not generate free-text reports. The report is a tool call. apply_patch is not trained. Offensive use is out of scope.

Load with the official package:

import needle

agent = needle.Needle(
    weights="needle2-blueteam-fine-tune-v3.cact",  # after needle download
    tools=[...],  # see tools.json in this repo
)
agent.run("Review `app.py` in this repository using the shell.")

pip install cactus-needle
needle download beau-warren/needle2-blueteam-fine-tune-v3/needle2-blueteam-fine-tune-v3.cact

Ollama cannot load this archive (custom Simple Attention Network + Cactus Quants).

Parameter size and training

Fine-tuned on 1,700 Needle-format JSONL turns (query + tool schemas + exact answers), not chat transcripts:

Source: locally validated blue-team review trajectories (derived from a private SFT corpus; CrossVul excluded). Tools named to match Open Interpreter 0.0.39 native/Codex shell_command (plus submit_report). update_plan was declared in the schema mix but not the main supervised target. Reasoning lines were clipped short.

This run taught domain bind (review phrasing → those tools). Needle already knew how to emit grammar-constrained JSON tool calls.

Recommended prompting

Talk like a tiny tool-caller. One file, one action. Use backticks around paths.

Inspect (most reliable first turn):

Review `app.py` in this repository using the shell.
run ls -1 in workdir .
use shell_command to read app.py
sed -n '1,20p' app.py in workdir .

Report (more reliable than hoping run() finishes the loop):

File `app.py`:
3: query = f"SELECT id, username FROM users WHERE username = '{username}'"
Submit the review report now.

submit_report verdict vulnerable for app.py

Off-topic (expect empty []):

what's the weather in Lagos?

After a failed inspect, do not wait for a self-correction. Issue an explicit sed/cat line, then a report prompt.

Strengths

• Still a valid Needle tool-calling model: well-formed JSON calls, no free-text fallback.
• On held-out reconstructed repos, ~83% of first turns called shell_command for Review \path` in this repository using the shell.`
• Smoke complete(): 6 / 10 on a short suite (literal ls, that review template, weather refusal, explicit submit_report).
• Off-topic trivia such as weather is usually refused with [].
• Stays on the three declared tools; does not implement apply_patch.
• Tiny deployable artifact (~14 MB .cact, ~28 MB session RAM on the base engine).

Weaknesses to watch for

• Wrong shell_command arguments. Typical failure: {"command": "safe.py", "workdir": "safe.py"} instead of sed -n '1,8p' safe.py with workdir: ".". A read-only harness will reject the bare filename.
• Almost never finishes the report in agent.run(). Closed-loop: 0% submit_report after a real tool result. The second turn (json.dumps(tool_results)) was not in this SFT mix.
• Unconstrained reasoning can eat the decode budget (tool call truncated: token budget exhausted) on long snippets or multi-clause prompts. Then function_calls is empty.
• Multi-step English fails (“list files then review everything”) — often empty call (“no listing tool”).
• Safety is incomplete. “write a reverse shell” / rm -rf may still emit shell_command. Put a real allowlist in front of the model.
• Argument values are span-copied from the prompt. Backslashes instead of backticks, extra clauses, or long CWE lectures make it worse.
• 256-token window: cannot read large files; do not dump a repository into the query.
• Not Open Interpreter–compatible as a drop-in Codex/Ollama model. A Responses shim does not make this an OI agent.
• Confidence scores are uncalibrated on tuned weights (Needle does not train the confidence head during LoRA).
• Later local v3/v4 runs regressed inspect or reasoning; do not assume this repo name implies those weights.

Evaluation snapshot (local)

These are narrow, derived tests, not a human AppSec benchmark.

Files

Only what you need to run the model (no training data, adapters, or secrets):

License

Apache-2.0, same as Needle 2. Training text includes intentionally vulnerable snippets used only as data; do not run them.