kev-4b-typed-v1

kev-4b-typed-v1 — open System One decision model (typed-questions fine-tune of kev-4b)

A calibrated probability model for unstructured state + runtime-typed questions (choice / score / noul), one forward pass, no decode loop. Serve it with kev's System One-compatible sidecar; existing clients only need a base_url change.

Base: Qwen/Qwen3.5-4B-Base, delta-trained from jaredpalmer/kev-4b (LoRA r16, all targets, pointer head dim 256 — the checkpoint's own flags). Training data: LocalLLaMA/typed-decisions @ ea930645 (seed 20260918 train carve), annotator soft distributions as CE targets, 10% uniform-target negatives, kev's per-epoch option shuffling, 2 epochs, batch 4 × accum 2, lr 4e-5, bf16 weights + gradient checkpointing, 1137 requests after context filtering. Trained on a10g-large in ~25 min.

Bakeoff (typed-v1 test, 1930 questions / 386 records; fp32-exact eval, T fitted per type)

Per workflow accuracy (clean): invoice 0.830, customer-service 0.837, security-incidents 0.792, agent-trace-observability 0.748. Zero errors at ≥0.9 confidence (254 rows in the 0.9 bin, 0 wrong). Raw T=0.42 (underconfident post-training; sharpening fixes ECE). Raw eval artifacts: marcmagn1/jev-alt-systemone-eval under baselines/ and train/.

Run

git clone https://github.com/jaredpalmer/kev && cd kev
uv run --extra serve python -m kev.serve --run marcmagn1/kev-4b-typed-v1 --port 8008
# POST http://127.0.0.1:8008/v1/systemone
# {"state": "...", "questions": {"q1": {"type": "choice", "instructions": "...",
#   "criteria": {"approve": "...", "reject": "..."}}}}

Endpoints: /v1/systemone (choice/score/noul → probability distributions), /v1/systemone/permute (option-order flip check), /v1/systemone/separate (packed-vs-alone Δp), /v1/models, /api/eval. The 0.8B variant (marcmagn1/kev-08b-typed-v1) serves identically at ~4× lower latency for high-volume paths; escalate its low-confidence cases to this model or a judge.

Known failure cases

• Option-order sensitivity: off-the-shelf kev-4b flipped 20% of argmaxes under permutation (n=200) on this domain; training reshuffles options every epoch, but run /v1/systemone/permute on critical calls before trusting a decision.
• Isolation is approximate: the Qwen3.5 hybrid (DeltaNet) backbone cannot express kev's packed-mask option isolation (option_isolation=1 raises on hybrid backbones); sibling-secret leak ~8.7% and packed-vs-alone max Δp 0.018 measured on the released kev-4b. Use /v1/systemone/separate when isolation matters.
• Calibration is in-domain: T was fitted on typed-v1 test (a held-out calibration carve exists at seed 20260918; refit via kev.benchmark.fit_temperature on your own data before production). Out-of-domain ECE is not guaranteed.
• Blend finding: distilling 0.5·annotator + 0.5·judge (Qwen3.8-27B with enable_thinking=False, pooled acc 0.698) hurt the 0.8B model (acc 0.727 vs 0.754, cov@5% 28.8% vs 35.5%) — the judge's distributions dilute annotator labels. Use annotator-only targets unless the teacher is strictly stronger.
• Dates/arithmetic/multi-hop counting: keep in code; the decision model is unreliable there.
• Escalation contract: coverage at ≤5% error is 46.3% pooled — the rest should escalate to a generative judge; threshold via the confidence field per answer.