jevlite-decision-heads

jevlite decision heads (phases 1–2)

Hi — for-fun side-project stuff, same project as mchen04/jev-local-lab-decision-heads (commit history spans 2026-09-19 to 2026-09-22). The question here was narrower: if you chop a small quantised Qwen2.5 down to its first L transformer layers and bolt a tiny trained head on the end, how much accuracy do you lose and how much latency do you save? These are the 39 heads that came out of that sweep.

Not production-ready. Not a reproduction of any commercial product. Does not reach parity with the published "Jev" numbers I was comparing against — on an external workflow suite these heads scored 0.215–0.312 agreement where a plain zero-shot read-out on the same backbones scored 0.582–0.585 and the published commercial reference scored 0.909. That gap is the main thing this repo documents.

tl;dr of the honest results:

• On the five academic classification tasks they were trained on, the best head reads at 0.736 accuracy at full depth and 0.716 at 20 of 28 layers — a 2.0 point drop for a 28 % latency cut (174 ms → 126 ms per question).
• Hold out one training task and accuracy on that task falls off a cliff: holding out emotion_label leaves 0.15 on emotion against 0.83–0.93 on the tasks it did see.
• On a suite of real workflow decisions they are far worse than not using them at all.
• Cross-backbone distillation from a 1.5B head into a 0.5B head was worth roughly +1 to +2 points at shallow depths and nothing at deeper ones.

Table of contents

• Model details
• What is actually in this repo
• Uses
• Bias, risks and limitations
• Training details
• Evaluation
• What was NOT evaluated
• Technical specifications
• How to get started
• Licence and upstream attribution

Model details

Model description

39 decision heads of 0.46–1.32 M parameters each. A head takes one hidden state from a frozen Qwen2.5 backbone — optionally one truncated to its first L layers — plus the mean-pooled input embeddings of a label set, and returns a distribution over those labels. Same cosine-bilinear scorer as the phase-3 heads; the new variable here is backbone depth.

Two backbones, both frozen and unmodified: mlx-community/Qwen2.5-1.5B-Instruct-4bit (hidden 1536, 28 layers) and mlx-community/Qwen2.5-0.5B-Instruct-4bit (hidden 896, 24 layers).

• Developed by: Michael Chen (solo side project)
• Model type: trained classification head / adapter over a frozen, optionally layer-truncated quantised LLM
• Language(s): English
• License: see Licence and upstream attribution — not guessed
• Adapter for: the two backbones above. Phase-1 heads pin mlx-community/Qwen2.5-1.5B-Instruct-4bit at revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677.

Model sources

No public code repository; the engine that produces the hidden state is a private local checkout. The head maths is documented in full below so the artifacts are not a black box.

What is actually in this repo

Weights are supplied — 39 real jevlite_head.safetensors files, plus the recipe/metrics JSON each was trained with. Backbone weights are not included: download them from their own repos.

phase1/<variant>/jevlite_head.safetensors   +  jevlite.json   (5 heads)
phase2/<variant>/jevlite_head.safetensors   +  metrics.json   (34 heads)
manifest.json                                                 (all 39, with sha256)

Phase 1 — depth sweep and held-out-task folds (5 heads, 1.5B backbone)

Trained on 15 000 items. dev here is a 60-item-per-task development set (small — treat with care).

Phase 2 — depth × distillation sweep (34 heads, both backbones)

dev2 is a 600-item development set over six tasks. Full per-variant numbers are in manifest.json and each variant's metrics.json; the shape of it:

The mlp head variants (nonlinear projection) were worse than the plain bilinear one at every depth tried — e.g. 1.5B L12 mlp 0.3650 against bilinear 0.4600. They are kept here because a negative architecture result is still a result.

Uses

Direct use

Reproducing the depth/latency trade-off measurement, or checking my numbers. If you are curious what a ~0.5 M-parameter head can read out of a half-depth 0.5B model, the artifacts are all here.

Out-of-scope use

• Anything real. No safety, fairness, robustness or red-team evaluation was done at all.
• Tasks the head was not trained on. Measured failure, not speculation: see the held-out-task numbers above and the workflow-suite numbers below.
• Anything needing calibrated probabilities. ECE was recorded but no calibration is claimed.
• Claiming parity with a commercial system. The gap is documented below and it is large.

Bias, risks and limitations

• These heads do not generalise past their training tasks. Holding out emotion_label gives 0.15 on emotion. On a suite of real workflow decisions, the best phase-2 head agreed with the reference on 0.312 of 330 pairs where a zero-shot read-out on the same backbone reached 0.582. A trained head on five academic classification tasks is not a general decision reader.
• Truncation is lossy in ways I did not fully characterise. I measured accuracy and latency by depth; I did not measure what the discarded layers were doing, and shallower heads may fail in correlated ways on inputs unlike the training distribution.
• Inherited bias. The backbones are Qwen2.5-Instruct models; their biases pass through unmeasured.
• English only. Five (phase 1) / six (phase 2) academic classification tasks only.
• Small development sets — 60 items per task in phase 1, 600 items total in phase 2. Per-task numbers move a lot on that little data; treat single-point differences as noise unless a paired test is quoted.
• One training source has restrictive terms (Yelp/yelp_review_full) and two have no resolvable licence — see Licence and upstream attribution.

Recommendations

For unseen tasks, use a plain restricted read-out on the un-truncated backbone instead. That is what the measurements say. Publish these as a record, not a recommendation.

Training details

Training data

Phase 2's dev2 additionally scores dbpedia_topic. Items are drawn from each source's train split (up to 3000 per task, 500 per task for the training-dev fold); the development and test suites come from validation/test splits and are disjoint. No dataset text is in this repo and none was uploaded.

What that does NOT establish: no extraction, memorisation or privacy evaluation was run on these heads. I cannot tell you what is or is not recoverable from the weights, and a small parameter count is not evidence that nothing is. Treat it as untested.

Training procedure

Backbone frozen throughout; only the head trains, on pre-cached hidden states and frozen label embeddings.

• Loss: cross-entropy, or (1 − α) · CE(student, gold) + α · τ² · KL(teacher ‖ student), both at temperature τ, for the distil variants. Phase 2 swept α ∈ {0.25, 0.5, 0.75, 0.9} at τ = 2.0; manifest.json records α per head.
• Teacher (cross-backbone): the phase-1 1.5B L28 r256 head's own distributions, used to train 0.5B students. So "distillation" here is head-to-head across backbones, not from a large generative model.
• Optimiser: AdamW. Phase 1: 15 000 items, best epoch 27. Phase 2: lr 5e-4, weight decay 1e-4, batch 256, 30 epochs, seed 0, model selection on the development fold.
• Truncation: the backbone's layer list is cut to the first L entries before the forward pass. No retraining or healing of the truncated stack — the head alone absorbs the change.

Speeds, sizes, times

Head training is 3–5 seconds per variant on cached hidden states (phase 1 records 4.1 s; phase 2 records 3.2 s for the 0.5B L16 distil head). Inference latency, warm p50 per question on the hardware below: 126 ms at 20/28 layers, 174 ms at 28/28. Checkpoints are 1.8–5.3 MB.

Evaluation

Testing data, factors and metrics

1. Phase-1 test suite — 1000 items over the five training task families, held out from training, opened after development stopped. Metric: accuracy.
2. Phase-2 dev2 — 600 items over six tasks; development data, looked at repeatedly.
3. An external workflow suite (TypeSafe's public evals) — 330 (case, question) pairs common to every system compared. Metric: agreement with the published reference answers.

Results

Phase-1 test, n = 1000, five trained task families:

Held-out-task failure, phase-1 development set: training without emotion_label leaves 0.15 accuracy on emotion_label while sst2 sits at 0.825 and agnews at 0.933. The head did not learn a task-general skill.

External workflow suite, n = 330 pairs — this is the number that matters most for honesty:

A trained head that is good at five academic classification tasks is roughly half as good as no head at all on real workflow decisions. That is the clearest single result in this repo.

Summary

Layer truncation is a real and cheap latency lever within the training distribution: 20 of 28 layers cost 2.0 accuracy points and saved 28 % of the per-question time. Everything about generalisation is negative. Cross-backbone distillation helped a little where the student was weakest and not at all where it was already fine.

About the "Jev" comparison

The 0.909 figure is published by the benchmark's authors for a commercial product with closed weights and early-access API. It was never run on my hardware. Nothing in this repo reproduces, approximates or matches it; the tables above are the measured distance, not a claim of closeness. Agreement with a published reference consensus is also not ground-truth accuracy.

What was NOT evaluated

• No safety, toxicity, fairness or bias evaluation of any kind.
• No robustness or adversarial testing.
• No paired significance test between most phase-2 variants — the sweep numbers are point estimates on 600 development items and many gaps are inside the noise.
• No phase-1 test evaluation for the 34 phase-2 heads — phase 2 reported dev2 and the external suite only.
• No non-English evaluation, no generation, no extraction, no multi-label.
• No human evaluation. No carbon measurement.

Technical specifications

Model architecture and objective

Identical scorer to the phase-3 heads. For hidden state h [hidden] and label embeddings E [n_labels, hidden], with rms(x) = x / sqrt(mean(x²) + 1e-6):

hn = rms(h) * g ;  En = rms(E) * ge
a  = hn @ Wh    ;  B  = En @ We          # bilinear variants
a  = gelu(hn @ W1 + b1) @ W2             # mlp variants instead
a, B L2-normalised on the last axis
s  = clip(exp(logit_scale), 1, 50) * (a @ B.T) + (En @ b) + c

Tensors: g, ge, Wh, We, b, c, logit_scale (float32); mlp variants carry W1, b1, W2 in place of Wh. hidden is 1536 (1.5B) or 896 (0.5B). E[i] is the mean of the backbone's input embeddings for the tokens of " " + label[i].

Compute infrastructure

• Hardware: one Apple M4 Mac mini, 24 GB unified memory, macOS 26.2.
• Software: MLX (mlx 0.32.x, mlx-lm 0.31.3), Python 3.12.
• Carbon emitted: not measured.

How to get started

The head is a plain safetensors dict; the surrounding engine is not published, so this is not a from_pretrained drop-in. Reference implementation of the head itself:

# pip install safetensors numpy
import hashlib, json, math
import numpy as np
from safetensors.numpy import load_file

V = "phase1/jevlite-Qwen2.5-1.5B-Instruct-4bit-L20-r256-all"
W = load_file(f"{V}/jevlite_head.safetensors")

man = json.load(open("manifest.json"))
rec = next(x for x in man["phase1"] + man["phase2"] if x["path"].startswith(V))
assert hashlib.sha256(open(f"{V}/jevlite_head.safetensors", "rb").read()).hexdigest() == rec["sha256"]

def rms(x, eps=1e-6):
    return x / np.sqrt((x * x).mean(axis=-1, keepdims=True) + eps)

# the trained `mlp` heads used EXACT GELU (x * Phi(x)), not the tanh approximation and not ReLU
_erf = np.vectorize(math.erf)
def gelu(x):
    return x * (1.0 + _erf(x / math.sqrt(2.0))) / 2.0

def head_scores(h, E, W):
    hn, En = rms(h.astype(np.float32)) * W["g"], rms(E.astype(np.float32)) * W["ge"]
    a = hn @ W["Wh"] if "Wh" in W else gelu(hn @ W["W1"] + W["b1"]) @ W["W2"]
    B = En @ W["We"]
    a = a / max(np.linalg.norm(a), 1e-6)
    B = B / np.maximum(np.linalg.norm(B, axis=-1, keepdims=True), 1e-6)
    scale = float(np.clip(np.exp(W["logit_scale"][0]), 1, 50))
    return scale * (a @ B.T) + (En @ W["b"]) + W["c"]

print(rec["variant"], "| layers", rec["depth"], "/", rec["full_depth"], "| params", rec["head_params"])

Both branches of head_scores — the bilinear one and the mlp one — were checked against the trained MLX scorer that produced these weights, on deterministic inputs, and agree to < 1e-5 max absolute difference. The check is reproduced in the repo's own terms below, so you can confirm the reference implementation is the real one rather than a paraphrase of it:

Full method and inputs: REFERENCE_CHECK.md in this repo. The first public version of this card used ReLU in the mlp branch "for brevity" while admitting in prose that the trained heads used GELU — that was a knowingly wrong implementation shipped as usable, and it is gone.

To use a head end to end you must truncate the backbone to depth layers, run your input, take the hidden state at the read position, and build E. The read position comes from unpublished code, so expect different numbers if you read somewhere else.

Licence and upstream attribution

Upstream, clear:

• Backbones, not redistributed here: mlx-community/Qwen2.5-1.5B-Instruct-4bit (revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677) and mlx-community/Qwen2.5-0.5B-Instruct-4bit, both Apache-2.0, derived from Alibaba Cloud's Qwen2.5 family. I did not create them; I froze, truncated and read from them.
• Training datasets and their published licences: the table in Training data. Yelp/yelp_review_full is tagged other and Yelp's own dataset terms apply; stanfordnlp/sst2 and fancyzhx/ag_news carry unknown; google/boolq is CC-BY-SA-3.0, which is a share-alike licence.

These head weights: no open-source licence has been selected, so copyright is retained by the author and they are published for inspection and reproduction. This is deliberate — I would rather say so than attach a label I am not sure is right. Ask in a discussion if you need specific rights, and note you would still need to satisfy the upstream dataset terms yourself.

LICENSE.md restates this in one place.

Model card authors and contact

Michael Chen. Community tab for questions or corrections.

Citation

No paper. Please describe it as a personal side-project experiment, negative results included.