kev-08b-typed-v1

kev-08b-typed-v1 — open System One decision model (typed-questions fine-tune of kev-0.8b)

A calibrated probability model for unstructured state + runtime-typed questions (choice / score / noul), one forward pass, no decode loop. Serve it with kev's System One-compatible sidecar; existing clients only need a base_url change.

Base: Qwen/Qwen3.5-0.8B-Base @ dc7cdfe2, delta-trained from jaredpalmer/kev-0.8b (LoRA r16, all targets, pointer head dim 256 — the checkpoint's own architecture flags). Training data: LocalLLaMA/typed-decisions @ ea930645 (rev ea930645..., seed 20260918 train/calibration carve), annotator soft distributions as CE targets, 10% uniform-target negatives (evidence stripped → model learns to be uncertain), kev's per-epoch option shuffling, 2 epochs, batch 4 × accum 2, lr 4e-5, bf16 weights, 1137 requests after context filtering.

Bakeoff (typed-v1 test, 1930 questions / 386 records; fp32-exact eval)

Per workflow accuracy (clean, T=1): invoice 0.836, customer-service 0.758, security-incidents 0.728, agent-trace-observability 0.696. Zero errors at ≥0.9 confidence (257 rows in the 0.9 bin, 0 wrong; kev-4b had 21). Raw temperatures per workflow after fitting: T=0.5 global (model is underconfident post-training; sharpening fixes ECE).

Raw eval artifacts: marcmagn1/jev-alt-systemone-eval (dataset repo) under baselines/ (B1/B2 summaries, per-workflow metrics) and train/ (this model's report + prediction rows).

Run

git clone https://github.com/jaredpalmer/kev && cd kev
uv run --extra serve python -m kev.serve --run marcmagn1/kev-08b-typed-v1 --port 8008
# POST http://127.0.0.1:8008/v1/systemone
# {"state": "...", "questions": {"q1": {"type": "choice", "instructions": "...",
#   "criteria": {"approve": "...", "reject": "..."}}}}

Endpoints: /v1/systemone (choice/score/noul → probability distributions), /v1/systemone/permute (option-order flip check), /v1/systemone/separate (packed-vs-alone Δp), /v1/models, /api/eval.

Known failure cases

• Option-order sensitivity: kev-4b flipped 20% of argmaxes under permutation (n=200) on this domain; training reshuffles options every epoch, but run /v1/systemone/permute on critical calls before trusting a decision.
• Isolation is approximate: the Qwen3.5 hybrid (DeltaNet) backbone cannot express kev's packed-mask option isolation (option_isolation=1 raises on hybrid backbones), so sibling questions can leak context: measured sibling-secret leak ~8.7% and packed-vs-alone max Δp 0.018 (target ≤1e-3). Use /v1/systemone/separate when isolation matters; per-question latency is higher but leakage-free.
• Calibration is in-domain: T was fitted on typed-v1 test (a calibration split exists; the released T is the test-fit value — refit on your own data via kev.benchmark.fit_temperature). ECE out of domain is not guaranteed; the SemIf style eval of off-the-shelf kev-0.8b (0.694) vs its own sources (~0.87) shows the gap.
• Dates/arithmetic/multi-hop counting: keep in code (Jev's own guidance); the decision model is unreliable there.
• Escalation contract: coverage at ≤5% error is 35.5% pooled — the rest should escalate to a generative judge; threshold via the confidence field per answer.

Teacher note

A Qwen3.8-27B letter-logit judge was tried as an additional soft-label teacher and scored near chance (0.25–0.45/workflow) — its chat template opens a thinking block, so next-token letter logits are garbage; pass enable_thinking=False (or force the </think>\n\n prefix) before using it as a teacher. Distillation therefore used the typed-decisions annotator distributions only.