Kahn1-Qwen2.5-3B

Kahn1-Qwen2.5-3B

Kahn1 is a fast, calibrated "System 1" decision engine derived from Qwen/Qwen2.5-3B-Instruct (LoRA fine-tuning, merged standalone checkpoint). Named after Daniel Kahneman's Thinking, Fast and Slow, it replaces autoregressive generation and JSON parsing with direct, calibrated logit reading over option tokens:

• Zero generated tokens: answers are read from the logits of candidate tokens (A–Z, yes/no), never parsed from text.
• 0.0% schema/typing errors, by construction.
• Three typed primitives: Choice (up to 26 options + fallback), Score (ordinal levels, continuous expectation), Noul (binary statement check).
• Millisecond-class latency on vLLM with prefix caching (36.4 ms median at debias k=3 on the full holdout; lower at k=1).
• Bilingual prompts (English / French).

Website, playground and Snake demo: 👉 https://kahn1.com · Engine, server, training and evaluation code: 👉 https://github.com/Okura66/kahn1

What changed in v3

• FEVER added as a Noul training source (SUPPORTS → yes, REFUTES and NOT ENOUGH INFO → no), so that an unsupported claim is trained as not-affirmable.
• Checkpoint selected by balanced accuracy across the three primitives on a stratified validation set (step 900), rather than by validation NLL.
• Temperature calibration refitted on a training-disjoint split (calibration.json). An isotonic stage on top raised ECE in the ablation, so it is left out.
• Wider holdout: app-store reviews added as a second Score benchmark (14,663 items instead of 10,663), so global numbers are not comparable with the previous release.

On the sources both releases share, v3 is on par with v2: Choice +0.15 pt, SST-5 +0.13 pt, RTE −1.45 pt, SciTail −0.57 pt.

Benchmarks (14,663 held-out instances, zero-shot)

All sources are dataset-level disjoint from training. Full pipeline: merged checkpoint + option-order debiasing (k=3) + temperature calibration.

Global: 70.30 % accuracy, NLL 0.753, ECE 0.082, latency p50 36.4 ms / p95 82.6 ms, 21.7 q/s (one RTX 5070 Ti, vLLM, k=3 debiasing, all questions of a state sharing one KV-cache prefix).

Against JEV (TypeSafe), on the same items

Choice is asked over the same 8 options for both (the gold one and 7 distractors); JEV's Noul is asked whether the text supports the statement, as Kahn1's is.

An earlier version of this card gave JEV every intent while Kahn1 answered over 8 options, which put Kahn1 ahead on Choice; like for like, JEV is ahead on every primitive. A newer, stronger model is published at Okura66/Kahn1-Qwen3.5-4B (JevBench 83.1 %, hard tier 70.3 %). Details and method: https://kahn1.com/benchmarks/

Honest notes: SST-5 exact match sits near the human agreement ceiling (~55-60 %); Noul is the weakest primitive and is overconfident on SciTail (ECE 0.25); calibration was fitted on the training distribution, recalibrate on your domain (sysone calibrate) before trusting a threshold.

Training configuration

Usage

The quickest way is the guide at https://kahn1.com/get-started/, which also has a setup prompt for coding agents (Claude Code, Codex, Cursor…).

git clone https://github.com/Okura66/kahn1 && cd kahn1
uv venv --python 3.11 && uv pip install -e ".[gpu]"
SYSONE_MODEL=Okura66/Kahn1-Qwen2.5-3B uv run sysone serve --port 8000

curl -X POST http://127.0.0.1:8000/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Hello, I cannot log in to my account since this morning.",
    "questions": [
      {"kind": "choice", "key": "category", "prompt": "Support ticket category?",
       "options": ["bug", "billing", "account"], "allow_other": true},
      {"kind": "score", "key": "urgency", "prompt": "Level of urgency",
       "levels": ["low", "medium", "critical"]},
      {"kind": "noul", "key": "legal_threat",
       "statement": "The customer makes an explicit legal threat"}
    ]
  }'

The tokenizer files need transformers>=5. calibration.json holds the v3 temperatures the sysone engine applies (SYSONE_CALIBRATION).

A lightweight LoRA adapter (239 MB) of the same weights is published at Okura66/Kahn1-Qwen2.5-3B-LoRA. The GGUF quantizations at mradermacher/Kahn1-Qwen2.5-3B-GGUF were made from an earlier release (v1).

License

The weights inherit the Qwen Research License of Qwen2.5-3B-Instruct (https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE), which restricts commercial use; an earlier version of this card said Apache 2.0, which was wrong. Engine code: MIT. For an Apache 2.0 model, see Okura66/Kahn1-Qwen3.5-4B.