jevify-qwen3.5-2b

jevify-qwen3.5-2b

<!-- jevify-card-v2 -->

Newer on the same backbone: jevify-qwen3.5-2b-readout-coh — readout fine-tuning with a coherence penalty, no heads: 0.702 accuracy, ECE 0.054, TVD to human labels 0.315 (FINDINGS 18). This repo stays the reference for its tier.

jev-bench · leaderboard · findings · code

A System One decision model: it does not write text. It reads a state, answers typed questions, and returns calibrated probability distributions your code can branch on.

This repo holds only what Jevify adds to Qwen/Qwen3.5-2B: 2,628,107 parameters of trained decision heads (10.5 MB) plus the calibration recipe. The backbone is pulled from its own repo at load time, so nothing is duplicated or relicensed.

Results on jev-bench

Scored on all 22,773 test records, against TypeSafe's Jev 1.13.0 on the identical records.

The held-out row above is the published seed. Five seeds of this configuration give held-out accuracy 0.579 ± 0.049 — bimodal: two seeds match Tier 0, three regress by ~0.09 — so this checkpoint is the top of its own distribution. Trained-source accuracy is stable (0.635 ± 0.007). The learned LM weight is the diagnostic (0.95+ = held up); capping training at six epochs makes held-out accuracy 0.621 ± 0.004 across seeds. FINDINGS 6.3a

Across the whole benchmark: macro accuracy 0.632 against Jev's 0.733, ECE 0.069 against 0.113, and 0.374 mean distance to human label distributions against Jev's 0.432 — lower is better, and that last number is the one a calibration claim rests on.

Use it

pip install git+https://github.com/uspraveen/Jevify

from jevify import load_jevified

model = load_jevified("Praveenrajus/jevify-qwen3.5-2b")
answer = model.ask(
    state={"ticket": "I was charged twice for order A-104, please refund the duplicate."},
    questions={
        "dept": {"type": "choice", "instructions": "Which team should handle `ticket`?",
                 "criteria": {"billing": "Payments and refunds", "shipping": "Delivery problems", "other": None}},
        "refund": {"type": "noul", "instructions": "Does `ticket` ask for a refund?"},
        "anger": {"type": "score", "instructions": "How angry is the customer?",
                  "criteria": ["calm", "annoyed", "furious"]},
    },
)
print(answer["dept"]["choice"], answer["dept"]["confidence"])
print(answer["refund"]["noul"])

As a drop-in for the TypeSafe API

jevify-serve --model Praveenrajus/jevify-qwen3.5-2b --port 8000

TYPESAFE_BASE_URL=http://localhost:8000 python your_existing_typesafe_code.py

The official typesafe-sdk works against this unchanged — that is a test in the repo.

How it was built

Decision heads read the backbone's hidden state at each option's own line, so they score what an option means rather than how likely its identifier token is. They are applied as a residual on the model's own log-score — score_i = w·lm_i + f(...) with f zero-initialized — so training starts exactly at the untrained baseline and can only add to it. Trained on 8,685 records from 16 sources with at most 16 options each, which is what keeps the head usable at any K.

Training optimizes strictly proper scoring rules directly — log score for Choice and Noul, ranked probability score for the ordinal Score — so calibration is the objective rather than a post-hoc repair. No reinforcement learning is involved: with a differentiable head the calibration objective is just a loss.

Six sources were held out of training entirely so generalization to unseen question types is measured rather than assumed. Full method, findings and limitations: github.com/uspraveen/Jevify · FINDINGS.md

Limitations

• English-first, text only, following the benchmark it was tuned on.
• Ordinal (score) questions on scales unlike those in training are the weakest case.
• The heads are trained on jev-bench's own train splits, so "held out" means held-out source, not a wholly different data universe.
• Seed variance is real and measured: see the paragraph under the results table. Differences of a few hundredths between models are within it.