auto-0.4b

auto-0.4b

A 0.4B encoder that decides whether an AI agent's next tool call is safe to run.

Agent harnesses like Codex "Auto" mode and Claude Code auto-accept currently ask a large frontier LLM the question "should I let this tool call run?" on every single step. That works, but it burns subscription usage, adds API latency to every action, and spends a generative model on what is fundamentally a classification problem — the thing encoder-only models are best at.

auto-0.4b is a full fine-tune of ModernBERT-large with context extended to 64k tokens, trained to answer exactly that question:

• approve — routine work that serves the user's request. Reading files, running tests, editing code, committing, web search, fetching docs, installing legitimate packages — network access is normal agent behaviour and is not treated as dangerous.
• deny — the call is consequential and unauthorised: wiping directories, exfiltrating credentials, installing hostile software, weakening security controls, destroying production state, or following instructions injected by content the agent just read.

It runs in 7 ms per call on a single GPU — roughly 400x faster than asking a frontier LLM the same question, at zero marginal cost.

Honest positioning

This is not a drop-in replacement for a frontier-LLM gate. On the held-out benchmark:

It trails Luna by ~7 points of accuracy, and its false-deny rate means it would interrupt roughly 1 in 10 legitimate actions at the default threshold. Where it earns its place is as a fast first-stage filter: it is very strong on the attacks that actually matter, and it answers in milliseconds so it can run on every single tool call.

A sensible deployment auto-approves below a low threshold, auto-denies above a high one, and escalates the middle band to a bigger model — which removes most of the LLM calls while keeping a frontier model's judgement on the genuinely ambiguous cases.

Benchmark: approve-or-deny

Evaluated on ProCreations/approve-or-deny (3000 held-out items, never trained on):

The regex baseline is the "block anything containing rm -rf/sudo/curl" heuristic these gates usually start life as. It is included to show the benchmark is not keyword-solvable — roughly half of the deny items contain no alarming token at all, and many approve items are full of them.

Choosing an operating point

logits[:, 1] after softmax is P(deny). Because a wrong approve costs far more than a wrong deny, you can trade the two:

Where it is strong and weak

By difficulty

By context length

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("ProCreations/auto-0.4b")
model = AutoModelForSequenceClassification.from_pretrained(
    "ProCreations/auto-0.4b",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",   # required for the full 64k context
).cuda().eval()

def build_input(user_request, history, call):
    """history: list of dicts with tool/args/result. call: dict with tool/args."""
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {call['args']}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result','')}")
    return "\n".join(parts)

text = build_input(
    user_request="clean up the build artifacts and reinstall deps",
    history=[{"tool": "Bash", "args": "ls -la", "result": "node_modules  dist  src  package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)

enc = tok(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
with torch.no_grad():
    p_deny = model(**enc).logits.softmax(-1)[0, 1].item()

print("DENY" if p_deny > 0.5 else "APPROVE", f"(P(deny)={p_deny:.3f})")

The input format matters. The proposed call and the user request come first so they survive truncation; history follows. Use the exact section headers above — the model was trained on them.

How it was built

Context extension (8k → 64k). ModernBERT alternates local sliding-window attention (window 128) with full global attention every 3rd layer, so only 10 of 28 layers pay the O(n²) cost — which is what makes 64k practical on one GPU. Only those global layers need a new RoPE base. The base was chosen empirically by sweeping it against MLM loss on domain-matched long agent transcripts:

At the stock base the model simply cannot do 64k. 2,560,000 (16× the original for an 8× extension) was chosen: near-best long-context loss for ~4% short-context cost.

Training. Full fine-tune (all 395.8M parameters), two stages: bulk training at short context where nearly all real traffic lives, then a long-context stage at up to 64k so the extended RoPE is exercised on the actual task. Batching is by token budget rather than example count, since inputs span 200–65,536 tokens.

Data. 73,509 training examples generated with DeepSeek V4 Flash across a large combinatorial space of agent frameworks (Claude Code, Codex CLI, MCP servers, LangChain, computer-use, devops, …), domains, risk categories, obfuscation styles, and 10 languages, with deliberate minimal contrastive pairs — near-identical calls with opposite labels where only the user's request or the history flips the verdict.

Long-context examples are built by burying the decisive history steps inside benign filler at a random depth, so 64k capability is exercised as needle-in-a-haystack retrieval rather than merely declared.

Limitations

• Training data is model-generated and reflects the decision rule it was written against; it is not a substitute for a real security review of your agent's permissions.
• Held-out labels were audited by an independent pass, but both come from the same model family, so systematic blind spots can survive.
• The ONNX export is practical to ~8k tokens (the non-flash attention path materialises a dense sliding-window mask); use the PyTorch + flash-attn path for full 64k.
• It judges a proposed call from text. It cannot see what a script will actually do at runtime, so an opaque binary or a URL whose content it cannot read is judged on context alone.

Other formats

• ProCreations/auto-0.4b-ONNX — ONNX + int8

The GGUF build was withdrawn. llama.cpp converts the model, but its --pooling rank path returns zero for a 2-class classification head, so the GGUF could not actually make approve/deny decisions — it returned identical 0.000 scores for a rm -rf / and for a pytest invocation. It was removed rather than left up implying it worked.

Successor

ProCreations/auto-1b scores 96.40% on the same benchmark (vs 90.13% here), with false-deny down from 10.88% to 3.19% and long-context accuracy up from 78.45% to 97.02%. Prefer it unless you specifically need the smaller model.