jev-local-lab-decision-heads

jev-local-lab decision heads (phase 3)

Hi — this is for-fun side-project stuff. I tried seeing how well a small local model on a 24 GB Mac mini could read structured decisions out of documents, and these are the tiny trained heads that came out of it. The work runs from 2026-09-19 to 2026-09-22 in the project's own commit history; I am not going to round that into a tidier story than it was. They are not production-ready, they are not a reproduction of any commercial product, and they do not reach parity with the published "Jev" numbers I was comparing against. Two of the most interesting things in here are failures, and I've left them in on purpose.

tl;dr of the honest result: on the task families these heads were trained on they beat their own backbone's zero-shot read-out by +14.8 points. On a fresh, never-touched set of six different task families they LOST to it by 3.25 points (exact paired McNemar p = 0.0167), and on an external workflow suite they lost again. If you want a small local decision reader, the plain zero-shot read-out on the same backbone was better than these heads on everything unseen.

Table of contents

• Model details
• What is actually in this repo
• Uses
• Bias, risks and limitations
• Training details
• Evaluation
• What was NOT evaluated
• Technical specifications
• How to get started
• Licence and upstream attribution

Model details

Model description

Ten decision heads: small trained modules that sit on top of a frozen, unmodified 4-bit Qwen2.5-1.5B-Instruct and turn one hidden state into a probability distribution over a label set supplied at call time. They are not language models and they do not generate text. Each head is 0.20–1.32 M parameters (0.8–5.3 MB), i.e. roughly 0.01–0.09 % of the backbone.

The scoring function is a cosine-bilinear form between the backbone's hidden state at the read position and the mean-pooled input embeddings of each candidate label, so the same head works for any label set without retraining. Seven heads are a straight hyper-parameter sweep (rank, head shape, distillation weight); three are deliberate held-out-task probes trained with some task families removed, to test whether the head generalises. It does not — see Evaluation.

• Developed by: Michael Chen (solo side project)
• Model type: trained classification head / adapter over a frozen quantised LLM; supervised cross-entropy, some variants with knowledge distillation from a larger local model
• Language(s): English
• License: see Licence and upstream attribution — this is the one thing I have deliberately not guessed
• Adapter for: mlx-community/Qwen2.5-1.5B-Instruct-4bit at revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677 (Apache-2.0)

Model sources

There is no public code repository. The engine that produces the hidden state these heads consume (jevlocal) lives in a private local checkout, so this card documents the head's maths in full instead of pointing at an import. See How to get started — I would rather give you the equations than imply a turnkey download that does not exist.

What is actually in this repo

Weights are supplied. Every file below is a real trained artifact, not a recipe. Each directory holds head.safetensors (the weights) and record.json (full recipe, per-epoch history, the selected epoch, and the sha256 the weights must match).

dev3 macro is the mean per-task accuracy over the nine development task families, 960 questions. It is development data — it was looked at repeatedly and used to pick things. Treat it as selection evidence, not as a performance claim.

Also included: manifest.json (this table, machine-readable, with sha256 per file).

Uses

Direct use

Reproducing or poking at the experiment: attach a head to the frozen backbone, feed it a hidden state and a label set, get a distribution. Useful if you want to see what a ~0.8 M-parameter cosine-bilinear head can and cannot learn on top of a small quantised model, or to check my numbers.

Out-of-scope use

• Anything real. No safety evaluation, no fairness evaluation, no robustness testing, no red-teaming was done. Do not put this in a product, a pipeline that affects people, or any decision that matters.
• Unseen task families. This is the measured failure mode, not a hypothetical: the heads transfer badly to task families they were not trained on. See Evaluation.
• Medical, legal or financial decisions. Some training families are medical (PubMedQA) and legal (LEDGAR) in topic. That is a topic, not a qualification.
• Claiming these match a commercial product. They do not, and the numbers below show the gap.

Bias, risks and limitations

• Transfer failure is the headline limitation. A head trained on nine task families beat its backbone's zero-shot read-out on those families and lost to it on six unseen ones. The held-out-task probes show why: with clinc_intent (150 labels) and ledgar_provision (100 labels) removed from training, p3h-holdout scores 0.7170 on the task families it did see and 0.0063 on the two it did not. That is not a degradation, it is a collapse.
• Large label sets are where it breaks. The probe above is the extreme case; the milder one (p3h-holdout-small, holding out two small-label tasks) reaches 0.5938 on held-out tasks against 0.6933 on seen ones. Transfer degrades with the size of the unseen label set.
• Inherited bias. The heads read a frozen Qwen2.5-1.5B-Instruct. Whatever social, topical or linguistic bias that model carries is passed straight through, and I did not measure any of it.
• English only, and only on academic classification-style data.
• The training data is a mixture with heterogeneous and partly unclear licences — see Licence and upstream attribution.
• Tiny sample sizes. The untouched final evaluation is 1200 questions over six tasks; the external suite is 348. Per-task numbers below are unadjusted and exploratory.

Recommendations

If you want a small local decision reader for unseen tasks, use the plain zero-shot restricted read-out on the same backbone instead. That is what my own measurements say, and it is why I am publishing these heads as an experiment record rather than as a recommendation.

Training details

Training data

train3: 3405 items drawn from the train splits of nine public datasets, at row indices the development split does not use, deduplicated by a sha256 of the normalised input text against every other split in the project. The nine families:

No dataset text is included in this repo, and none was uploaded anywhere — the built task files stay on the machine that made them.

What that does NOT establish: I have not run any extraction, memorisation or privacy evaluation on these heads, so I cannot tell you what, if anything, is recoverable from the weights. A small parameter count makes verbatim memorisation less likely a priori, but "unlikely a priori" is not a measurement and I am not going to present it as one. If that matters for your use case, treat it as an open question and test it.

Training procedure

The backbone is frozen and unmodified throughout — only the head trains.

1. Hidden states are cached once: for each item, the backbone's prefix is run, a per-question branch row is appended, and the hidden state at the final branch position is kept (float16).
2. Label embeddings are the backbone's own input embeddings for each label string encoded with a leading space, mean-pooled over the label's tokens (float32).
3. The head trains on those cached vectors. Nothing backpropagates into the backbone.

• Loss: (1 − α) · CE(student, gold) + α · τ² · KL(teacher ‖ student), both at temperature τ. α = 0 for the ce- variants. The KL is the standard soft-target direction — teacher first — computed as Σ p_teacher · (log p_teacher − log p_student).
• Teacher (the kd- variants): a local 4-bit Qwen3.8-27B read-out (lmstudio-community/Qwen3.8-27B-MLX-4bit @ 6067b15cf581666a4aecf6af3afaba4bb5efc20c) producing soft label distributions. Coverage was 1499 of 3405 items (44 %) — the teacher pass was time-bounded at ≤150 rows per flat task. So the distillation gain is a gain at 44 % coverage, not at saturation, and it was small: +0.66 pp macro over the best non-distilled head on dev3.
• Optimiser: AdamW, lr 1e-3, weight decay default, batch 256, 60 epochs, seed 20260921. Model selection is the best dev3 macro epoch, recorded per head in record.json.
• Numerical choices that were forced by observed failures (all in record.json's lineage): both sides RMS-normalised before projection; the bilinear form cosine-normalised (a raw dot product diverged — cross-entropy went 0.70 → 1.25 → 185.3 in three steps at lr 1e-3); the logit scale learned but clipped to [1, 50].

Speeds, sizes, times

Training a head is seconds to a couple of minutes on cached hidden states. Caching the hidden states for 3405 train + 960 dev items took about 4 minutes on the hardware below. The 27B teacher pass took 1 h 34 min. Checkpoints are 0.8–5.3 MB.

Evaluation

Testing data, factors and metrics

Three evaluations, all with the same head (p3h-kd-a05, the one dev3 selected):

1. final3 — a fresh split of six task families disjoint from training, frozen before any training and opened exactly once. 1200 questions. Families: sciq_support, qnli_entailment, piqa_physical, medmcqa_choice, paws_paraphrase, snli_entailment.
2. An external workflow suite (TypeSafe's public evals) — 348 (case, question) pairs of real workflow decisions, scored as agreement with the published reference answers.
3. Held-out-task probes — three heads trained with task families removed, scored on dev3 split into seen vs unseen families.

Metric: accuracy, and macro accuracy = mean of per-task accuracies. Comparisons use the exact paired McNemar test on per-item outcomes keyed by (item, question).

Results

The untouched split (final3, n = 1200, 6 unseen families):

−3.25 pp for the head, exact paired McNemar p = 0.0167. The head is behind on all six tasks (medmcqa −5.50, snli −5.50, piqa −4.00, qnli −2.50, sciq −1.50, paws −0.50 pp). No single per-task difference reaches significance — those six p-values are unadjusted and exploratory — but the overall paired test on all 1200 items, which is the confirmatory one, does.

The external workflow suite (n = 348): the head agreed with the published reference on 0.5816 of pairs against 0.6499 for the plain read-out on the same backbone. Third independent measurement, same direction.

Held-out-task probes (dev3, seen vs unseen families):

Development numbers, for completeness and not as a performance claim: on dev3 (nine training families, 960 questions) p3h-kd-a05 reaches macro 0.6885 against the same backbone's zero-shot read-out at 0.5410 — +14.8 pp. That is the number that did not survive contact with unseen tasks, and it is exactly why the fresh split existed.

Summary

The head learned the task families it was trained on, not "how to read a decision". On dev it looked like the bargain of the project; on anything unseen it is worse than not using it. I am publishing it because a clean, reproducible negative result is worth more than a quiet deletion.

About the "Jev" comparison

The project compared against published numbers for a commercial product (agreement 0.9110 on a 337-pair frozen subset). Those answers were published by the benchmark's authors and were never run on my hardware — closed weights, no local execution. Nothing here reproduces, approximates or matches that system. The best non-head configuration in the project reached 0.8398 agreement on that subset, which was still significantly behind (p = 0.000536); the head reached 0.5816. Agreement with a published two-model consensus is also not ground-truth accuracy.

What was NOT evaluated

Listed because their absence matters:

• No safety, toxicity, fairness or bias evaluation of any kind.
• No robustness testing — no adversarial inputs, no prompt perturbation, no distribution shift beyond the unseen-family test above.
• No calibration claim. ECE and reliability bins were computed during the project but no calibration was fitted for these heads; they ran at temperature 1.0.
• No non-English evaluation.
• No multi-label, regression, extraction or generation evaluation. These heads only pick one label from a supplied set.
• No human evaluation.
• No evaluation of the three held-out probe heads on final3 — they were diagnostic only.
• No carbon measurement. I did not instrument power draw; I only have wall-clock.

Technical specifications

Model architecture and objective

For a hidden state h (shape [hidden]) and label embedding matrix E (shape [n_labels, hidden]), with rms(x) = x / sqrt(mean(x²) + 1e-6):

hn = rms(h) * g                    # g:  [hidden]
En = rms(E) * ge                   # ge: [hidden]
a  = hn @ Wh                       # Wh: [hidden, rank]   -> [rank]
B  = En @ We                       # We: [hidden, rank]   -> [n_labels, rank]
a  = a / max(||a||, 1e-6)
B  = B / max(||B||_rowwise, 1e-6)
s  = clip(exp(logit_scale), 1, 50) * (a @ B.T) + (En @ b) + c
P  = softmax(s)

Tensors in head.safetensors: g, ge, Wh, We, b, c, logit_scale (all float32). For the two mlp variants, a = gelu(hn @ W1 + b1) @ W2 replaces a = hn @ Wh.

hidden = 1536 for this backbone. E[i] is the mean over the backbone's input embeddings of the tokens of " " + label[i].

Compute infrastructure

• Hardware: one Apple M4 Mac mini, 24 GB unified memory, macOS 26.2. No GPU cluster, no cloud.
• Software: MLX (mlx 0.32.x, mlx-lm 0.31.3), Python 3.12.
• Hours: the project's commit history spans 2026-09-19 to 2026-09-22 on one machine. I did not instrument total occupancy, so I am not quoting a machine-hours figure. Head training itself is seconds to minutes on cached hidden states.
• Carbon emitted: not measured.

How to get started

Be honest about what this is: the head is a plain safetensors dict, and the surrounding engine (prefill → KV-cache broadcast → batched branch pass → restricted read-out) is not published. So this is not a one-line from_pretrained. What you get is the exact maths above plus a runnable reference implementation of the head itself.

# pip install safetensors numpy
import math
import numpy as np
from safetensors.numpy import load_file

W = load_file("p3h-kd-a05/head.safetensors")      # or p3h-ce-mlp / p3h-kd-mlp-a05 for the mlp heads

def rms(x, eps=1e-6):
    return x / np.sqrt((x * x).mean(axis=-1, keepdims=True) + eps)

# the two `mlp` heads in this repo were trained with EXACT GELU, x * Phi(x) -- not the tanh
# approximation, not ReLU. numpy has no erf, so vectorise math.erf rather than substitute.
_erf = np.vectorize(math.erf)
def gelu(x):
    return x * (1.0 + _erf(x / math.sqrt(2.0))) / 2.0

def head_scores(h, E, W):
    """h: [hidden] backbone hidden state. E: [n_labels, hidden] mean-pooled label embeddings."""
    hn = rms(h.astype(np.float32)) * W["g"]
    En = rms(E.astype(np.float32)) * W["ge"]
    a = hn @ W["Wh"] if "Wh" in W else gelu(hn @ W["W1"] + W["b1"]) @ W["W2"]
    B = En @ W["We"]
    a = a / max(np.linalg.norm(a), 1e-6)
    B = B / np.maximum(np.linalg.norm(B, axis=-1, keepdims=True), 1e-6)
    scale = float(np.clip(np.exp(W["logit_scale"][0]), 1.0, 50.0))
    return scale * (a @ B.T) + (En @ W["b"]) + W["c"]

# sanity check against the published metadata rather than trusting the file
import json, hashlib
man = {m["version"]: m for m in json.load(open("manifest.json"))}
blob = open("p3h-kd-a05/head.safetensors", "rb").read()
assert hashlib.sha256(blob).hexdigest() == man["p3h-kd-a05"]["sha256"]
print("shapes:", {k: v.shape for k, v in W.items()})

This is the real scorer, not a paraphrase of it. Both branches were run against the trained MLX modules that produced these weights, on fixed deterministic inputs, and agree to under 1e-5:

Full method and inputs: REFERENCE_CHECK.md in this repo.

To get h and E you need the backbone. E is straightforward (embed " " + label, mean-pool). h is whatever hidden state you choose to read at — the numbers in this card come from a specific read position produced by unpublished code, so your numbers will differ if you read elsewhere, and I would not expect to reproduce the table above without reimplementing that pipeline.

Licence and upstream attribution

Upstream, clear:

• Backbone: mlx-community/Qwen2.5-1.5B-Instruct-4bit, revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677, Apache-2.0, derived from Qwen/Qwen2.5-1.5B. Its weights are not redistributed here — download it from its own repo.
• Distillation teacher: lmstudio-community/Qwen3.8-27B-MLX-4bit, revision 6067b15cf581666a4aecf6af3afaba4bb5efc20c, Apache-2.0. Not redistributed here either, and not a model I made — I only evaluated it and used its outputs as soft targets.
• Training datasets and their published licences: the table in Training data. Two are restrictive or unclear in ways that matter: ehovy/race publishes terms restricting use to research, and allenai/scicite, tdiggelm/climate_fever and allenai/openbookqa carry no resolvable licence (unknown), while allenai/winogrande carries no licence tag at all.

These head weights: I have not selected an open-source licence for them, so the default applies — copyright retained by the author, published here so the experiment can be read, checked and reproduced. I am not granting broader rights in this card because I do not want to guess, and because anyone using these weights still has to satisfy the upstream dataset terms above independently. If you want a specific licence, open a discussion and ask.

See LICENSE.md in this repo for the same statement in one place.

Model card authors and contact

Michael Chen. Questions or corrections: the Community tab of this repo.

Citation

No paper. If you refer to this, please describe it as what it is: a personal side-project experiment, including its negative transfer result.