chia-governor-triage-7b

chia-governor-triage-7b

A LoRA adapter for Qwen2.5-Coder-7B-Instruct that predicts, from the code alone, whether a Verilog design change will fail its test bench before any simulator is run.

It exists to sit in front of the expensive evaluation stage of an agentic hardware design loop. Simulation is slow and costly, and many agent proposals are simply wrong. This model triages them.

Why it beats larger models at this task

Every frontier model tested carries the same wrong prior: that any edit to working code is probably a bug. That produces near-perfect recall and poor precision, so the gate throws away a large share of perfectly good designs.

This adapter was trained on data containing verified-harmless edits alongside real faults, which attacks that bias directly.

Near chance to near ninety-five on harmless edits. That 39-point gain is the whole result.

Results

630 held-out examples, from VerilogEval problems absent from training entirely. Positive class is FAIL.

Majority-class baseline is 54.1%.

Precision is the measure that matters here. A false FAIL silently discards a working design, which hurts a design loop more than paying for one extra simulation. This adapter has the highest precision of any model tested, at roughly a quarter of Flash's cost and a twentieth of its latency. Flash retains clearly better recall.

Tunable operating point

Usage

from chia_governor import TriageGate, LocalScorer

gate = TriageGate(
    LocalScorer("Qwen/Qwen2.5-Coder-7B-Instruct",
                adapter="emailvenky/chia-governor-triage-7b",
                load_4bit=True),
    threshold=0.9,
)

if gate.allow(spec, candidate_code):
    result = run_simulation(candidate_code)

Or directly. Score by comparing the PASS and FAIL logits rather than generating text, which is deterministic and cannot return an unparseable answer:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "emailvenky/chia-governor-triage-7b")

The prompt format is fixed and must match training: a system message describing the task, then the specification and the candidate code, asking for one word. See chia_governor/triage_gate.py for the exact strings.

Training

Loss fell from 0.23 and plateaued near 0.10 rather than collapsing toward zero, which is the shape of a learned decision rule rather than memorisation.

Data

Generated rather than collected, with labels correct by construction. Working VerilogEval references are edited by script and every variant is simulated against its real test bench. Two families: eight faulty operators (flipped comparisons, swapped operators, off-by-one constants, altered bit ranges, changed reset values, non-blocking to blocking, dropped assignments, swapped signals) and six benign ones (commutative swaps, internal renames, redundant parentheses, reformatted literals, reordered independent assigns, dead wires).

Both families are necessary. A first pass with faulty edits only produced a set that was 79.7% FAIL, scoreable at 80% by always answering FAIL. Adding benign edits brought it to 44.8% and made the task meaningful.

Train and test are split by problem, so no VerilogEval problem appears on both sides.

Limitations

• Recall is the weak dimension. At 84.8%, roughly one failing candidate in six gets through, where Gemini Flash catches it.
• Faults are synthetic. They imitate plausible agent errors but are not drawn from real agent output.
• Small modules only. VerilogEval modules are far smaller than an industrial RTL block; transfer is untested.
• Contamination is plausible. VerilogEval may appear in pretraining data. The mutated variants do not, but the base problems may.
• One run, no seeds. No variance reported.

Links

• Code, dataset and evaluation: https://github.com/venkateshrajag/chia
• Built for the CHIA framework: https://chialoops.ai

Licence

BSD-3-Clause, matching CHIA.