circuit-8b

circuit-8b

v1.2 (2026-09-22). The options of a choice question are encoded side by side, so the answer cannot depend on the order they are listed in: 0.6% of answers change under reordering, against 18.3% for v1.1, 11.4% for v1.0 and 8.3% for Jev. It is also the best of the three on most sets: tool relevance .823, groundedness .857, 64-way intents .807 with ECE .044, the grid back to .977. config.json carries "parallel_options": true; serve it with the circuit repo's scorer, which applies the mask. revision="v1.1" gets the previous weights. See Versions.

v1.1 (2026-09-21). Same base, head and recipe as v1.0, with two question families added to the training data: is this claim supported by the evidence, and does this tool fit the request. It gives back a point or two on the tasks v1.0 was tuned on and gains 5 to 15 on tasks neither had seen. See Versions. revision="v1.0" gets the original weights.

A System One decision model: typed questions in, calibrated probability distributions out, one forward pass, no text generation. It speaks TypeSafe's POST /v1/systemone contract and is the larger open-weights backend for decision-circuits. The small one is circuit-1.7b.

circuit-8b is a LoRA adapter plus a pointer readout head on Qwen/Qwen3-8B-Base. A request packs a state and a question; each option is wrapped in delimiter tokens and the sequence ends with a decide token. The head scores every option's closing delimiter against the decide token and applies softmax. Those probabilities are the answer, trained with cross-entropy against outcome labels, so calibration is learned rather than derived.

Results

Same items for every model, human labels, accuracy / ECE (15 bins).

The four cold-eval tasks' train splits are in this model's training mix; they are presumably not in Jev's. Checks nothing was trained on:

* the grid generator is ours, so this is held-out items, not held-out structure. With the 1.7B recipe, an operation and a format withheld scores 91% on the unseen format and 57% on the unseen operation: layouts transfer, new kinds of judgment must be in the training data.

Known weakness shared with every model above, Jev included: on inputs built to be undecidable, mean confidence is 0.5 to 0.85 where it should be near 0. Calibration on ambiguity is the open problem.

Throughput on one RTX A6000 (48 GB), bf16, batch 8: 72 ms per grid item, about 2,100 input tokens per second. On the 546 production questions (longer states) it is 198 ms per item.

Datasets nobody prepared for

300 items each from four public datasets that are in no model's training mix here. Accuracy / ECE / Brier score; lower is better for the last two. ChaosNLI's reference is the vote of 100 annotators per item, so its Brier measures distance from how people answered. (An earlier version of this card reported KL. Jev's API rounds to two decimals, which makes KL against it meaningless, so this table uses Brier.)

Reproduce all of it with one command: REPRODUCE.md. The ChaosNLI gain is mostly the checkpoint, not the new data: the last checkpoint of the same run scores .577 with KL 1.11. Training on one-hot labels past the point of best validation calibration buys accuracy where the training data lives and spends calibration everywhere else, so v1.1 is the early checkpoint on purpose.

Training

• Base: Qwen/Qwen3-8B-Base (Apache 2.0), frozen. LoRA rank 16, alpha 32, on all attention and MLP projections (43.7M params). Pointer head: two 4096 x 256 linear maps.
• Data, 19,738 items, the same mix as circuit-1.7b: every label computed by code or by humans, every source permissive (CC BY / CC0 / MIT / Apache). The generalization grid (8,100 + 838 oversampled ambiguous items with soft 0.5 labels), a commercial-safe slice of public classification tasks (2,100), and CC-licensed real data (MNLI, civil_comments, sms_spam, CLINC; 5,700). New in v1.1, 3,000 items: claim and answer support from VitaminC and SQuAD v2, and tool relevance, tool choice and tool-call checks built from CLINC and SNIPS utterances against hand-written tool specs. BFCL and HaluEval supply nothing; they are the test. No teacher-model outputs.
• 1 epoch, batch 4, max 1,024 tokens, lr 1e-4 (LoRA) / 1e-3 (head), bf16 with gradient checkpointing, soft-target cross-entropy, early stopping on validation ECE (kept step 1,200 of 4,446: ECE 0.020, accuracy 89.9%). 45 minutes on one H100. v1.0 was step 2,800 of 3,766, 75 minutes on one RTX A6000.
• Training code, data generators, and the evaluation harness: github.com/Barneyjm/circuit.

Order stability

Ask any causal decoder a multiple-choice question and its answer depends on the order the options are written in, because each option is read having read the ones before it. Over 981 questions from 16 task families, each asked in 4 option orders, the top answer changed on 8.3% of items for Jev, 23% for a stock Qwen3.5-4B read through its answer-letter logits, 30% for Laya, and 11 to 18% for circuit-8b v1.0 and v1.1.

v1.2 encodes each option so that it attends to the state, the question and itself, never to another option, with positions that restart where the question ended; the decide token attends to all of them from one position past the longest. Hidden states and probabilities are then identical under any ordering, up to bf16 summation order, in one forward pass. 0.6% of answers change on the same 981 items, and accuracy over them is .773 as written and .772 reversed. The mechanism is about forty lines, s1proto/parallel.py in the repo.

Reproduce: uv run python scripts/eval_permutations.py lora:<this repo> --out out.json.

Versions

Pin one with revision="v1.0" in huggingface_hub, or hf download jbarney/circuit-8b --revision v1.0.

Use

Serve it with the circuit repo and point any System One client at it:

S1_MODEL=lora:runs/circuit-8b uv run python -m s1proto        # :8901

from decision_circuits import Circuit, Q, argmax
from decision_circuits.backends import SystemOne

c = Circuit()
c.noul("pii", "Does the message contain personal information?")
c.choice("dept", "Which team?", {"billing": None, "technical": None, "other": None})
c.gate("redact", Q("pii") >= 0.7, on_uncertain="escalate")
c.gate("route", argmax("dept", min_confidence=0.35))
out = c.run(SystemOne("http://localhost:8901/v1/systemone", api_key="x"), "Card charged twice, my card ends in 4412")

Files: adapter/ (PEFT LoRA), head.pt (pointer head, keys q.weight, k.weight), config.json (base, hidden size, head type, layout, training args). The base needs about 17 GB in bf16; set "load_4bit": true in config.json and the server loads it in 4-bit (bitsandbytes NF4), which fits a 12 GB card.

Intended use and limits

Research and evaluation of calibrated decision models, and as a local backend for decision circuits. Not a production system for decisions that affect people. English only. Confidence on undecidable inputs is not yet reliable (see above); put thresholds inside an uncertainty band and route the band to a human, which is what decision-circuits does.

License

Adapter and head: Apache 2.0. Base model: Qwen license (Apache 2.0 for Qwen3).