judgment-qc-gate-qwen3-4b-lora

Judgment QC gate: a LoRA that reviews work the way one operator does

v10. This adapter has been retrained daily for two weeks on a growing set of human-approved corrections. The evaluation is still small and self-judged, so treat the numbers as evidence rather than proof. It is a real internal tool (it gates our own agent), not a general-purpose reviewer.

This is a LoRA adapter for Qwen/Qwen3-4B that distills one engineer's quality-control judgment into a small local model. Given a one-line description of a work situation, it returns the verdict that operator would give.

IN : [phase: post] Assistant declared a build 'done / passed QC' on the strength
     of 'should work' and 'I made the edit', with no run output shown.
OUT: VERDICT: VERIFY | Show the actual run output, not just the claim.

It is a gate, not a chatbot: the intended deployment is inside tooling (pre-send hooks, CI steps, agent harnesses) where drafts of outbound work are judged before they ship.

Design rule: skill, not facts

The corpus deliberately teaches how to judge and keeps facts out of the weights. Facts belong in retrieval, where they can be updated without retraining. A judgment model that memorises project details is stale in a month; one that learns "verify before claiming done" is not.

Training data

647 examples, each a real situation from months of AI-assisted engineering work paired with the correction the operator actually gave at the time. Every example was individually reviewed and approved by that operator before entering the corpus. Examples are generalised: no names, no figures, no project internals, only the shape of the situation and the standard applied. 570 examples train, 63 are held out for evaluation.

The corpus is deliberately balanced. Roughly 40 percent are violations (the gate should block) and 60 percent are compliant work that merely looks risky (the gate should pass). An earlier corpus was violation-heavy, which taught the model to block too readily; the compliant examples exist to correct that.

Format: closed-schema chat-message JSONL. System gives the gate instruction, user gives the situation, assistant returns VERDICT: <TAG> | <correction> or VERDICT: OK, drawn from a fixed vocabulary of about 14 tags.

Training

• LoRA r=16, alpha=16, dropout 0, on q/k/v/o/gate/up/down projections
• batch 1, gradient accumulation 8, 3 epochs, lr 2e-4, linear schedule, bf16, gradient checkpointing
• Trained with Unsloth on a single AMD Strix Halo machine (128GB unified memory, ROCm), about 20 minutes wall time on the current corpus

Evaluation, honestly labeled

Two checks gate every release. Neither is a standard benchmark; both are small and internal.

1. Held-out alignment. 63 examples the model never trained on, scored by comparing its verdict to the operator's recorded verdict. It measures the thing the model is for, but the alignment call is the author's judgment.
2. Production probes. Before a new adapter is allowed to serve, it must clear two fixed sets run through the live gate: a false-positive probe (30 clean cases it should pass) and a violation probe (12 known-bad cases it should catch). A regression on either rolls the release back automatically.

The false-positive rate falling to 0/30 is the payoff of the balanced corpus: the model stopped blocking clean work while still catching every planted violation. The set is small, so read it as a direction, not a guarantee.

Usage

With transformers + peft:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "AltronisSG/judgment-qc-gate-qwen3-4b-lora")
tok = AutoTokenizer.from_pretrained("AltronisSG/judgment-qc-gate-qwen3-4b-lora")

msgs = [
    {"role": "system", "content": "You are the QC gate. Given a work situation, return the verdict."},
    {"role": "user", "content": "[phase: post] About to send an answer without re-deriving its numbers."},
]
inp = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(inp, return_tensors="pt").to(model.device), max_new_tokens=180, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Note: the fine-tune emits an empty <think></think> block before its verdict (the corpus contains no reasoning traces); strip it in post-processing, or serve with thinking disabled.

For local serving, use the ready-to-run merged build: AltronisSG/judgment-qc-gate-qwen3-4b-GGUF (Q8_0, 4.3GB, single file), the exact artifact we serve, or merge and convert yourself with llama.cpp.

Limitations

• Narrow by design. It judges work-process situations phrased like its corpus. It is not a general reviewer, linter, or safety model.
• Small eval. 63 held-out examples plus two fixed probes, all self-judged. See the table's labels.
• Memorisation. A 3-epoch LoRA over a few hundred examples will reproduce training phrasings when prompted in-domain. The corpus is generalised to contain no names, figures, or private details, so this is a stylistic property, not a data leak.
• One person's bar. This encodes a specific operator's standards (verify-before-done, no unearned claims, no silent degradation). If your bar differs, retrain on your own corrections; the recipe is the point.

How it got here

The gate runs as a live Stop-hook on our own agent and is retrained on a daily human-approved loop, so its history is a record of fixing itself in public. The two changes that mattered most:

• Closed verdict schema. Early versions used free-form verdict tags (85 distinct tags across the first 117 examples), which trained the model to invent tags and emit recursive word-salad. Migrating to a fixed vocabulary of about 14 tags with a strict one-line format ended the degenerate outputs.
• Balanced corpus. The model over-blocked while the corpus was violation-heavy. Adding compliant-but-risky-looking examples, the kind of work that should pass, is what drove the false-positive probe from 6/30 down to 0/30.

The eval stays deliberately honest: a judgment model whose own evaluation is not judged honestly would be a contradiction in terms.

Attribution

Base model: Qwen/Qwen3-4B (Apache-2.0) by the Qwen team. Fine-tuned with Unsloth. Adapter released under Apache-2.0.

Built by Altronis, private LLMs and governed AI delivery, Singapore. The full build story, including the unified-memory crash and the multimodal LoRA gotcha, is on the Altronis blog.