NanoJev-mlx-4bit

NanoJev-mlx-4bit

An MLX affine 4-bit quantization of the root checkpoint of C-Tianyu/NanoJev — a nano replica of TypeSafe's Jev System One decision model: a Qwen3-0.6B backbone (28 layers, hidden 1024) with structured decision heads, 596M parameters in total. It returns complete probability distributions over dynamic candidates (choice 2–255 options, boolean, score 2–10 levels) in one batched backbone forward, with zero output-token decoding.

No training was performed and no weight changed its value: this repository re-encodes the upstream fp32 checkpoint in a lower-precision weight format.

What this is, and what it is not

• It is a post-training quantization: every quantized tensor is mapped onto 4-bit integer values inside groups of 64 input features, with one fp16 scale and one fp16 bias per group.
• It is not a fine-tuned model. Zero gradients, zero data, zero optimizer steps were involved.
• It is not a new architecture or an independent Jev implementation. The upstream checkpoint is the trained artifact; this is the same decision function at lower weight precision.
• License: the upstream repository declares no license for the weights (its code is MIT). This derivative therefore claims no license either, and points back to the upstream checkpoint for any licensing question.

Quantization configuration

The decision head is deliberately left unquantized: it is 0.03 % of the parameters and it is where the decision boundary lives, so there is nothing to gain and something to lose.

Measured weight-space error of the quantization (per-tensor max / mean relative error vs the upstream fp32 weights): mlp 12.45 % / 6.79 %, self_attn 11.75 % / 7.74 %, embed_tokens 6.70 % / 6.70 %.

Size

Measured quality cost

Reference: the upstream fp32 checkpoint. Split: the frozen stage2/dev.jsonl (120 states / 360 questions / 898 candidate paths), Apple M4, one backbone forward per 8-state batch, zero decode steps.

Method, stated plainly: this artifact was dequantized back to fp32 and evaluated through the unmodified upstream inference path, so the numbers below isolate the quantization error itself. A real MLX runtime also computes in a half-precision activation dtype, which adds a smaller second term that is not included here.

Per question type, argmax agreement with fp32: boolean 100 %, score 100 %, choice 97.5 %. All three flips are choice questions decided near a tie; two of the three move closer to the teacher target after quantization. The cross-entropy does not rise — the quantization error is smaller than the model's remaining distance to its own targets.

A variant that leaves embed_tokens unquantized was measured too: identical flip set, mean TV 0.00635 instead of 0.00660. Quantizing the embedding (26 % of all parameters) therefore costs essentially nothing, which is why it is included here.

This does not make inference faster

Measured on the same machine with MLX 0.32 at realistic shapes, a 4-bit matmul is at parity with fp32/bf16 weight matmul (2.96 vs 2.99 / 3.60 TFLOPS). This workload is one batched prefill per call, so it is compute-bound rather than weight-bandwidth-bound, and dequantization is pure overhead in it.

The gain here is footprint — 3.5× smaller than the bf16 conversion — not throughput. If the goal is speed, the levers that actually pay are prefix sharing across the candidate paths of a question, length-bucketed batching, and a fused/compiled runtime.

How to load it

The tensor layout follows the MLX convention:

• <name> — uint32, packed 4-bit weights
• <name>.scales, <name>.biases — float16, one entry per group of 64 input features
• every other tensor is a plain fp16 tensor

import mlx.core as mx

w = mx.load("best.safetensors")
print(w["backbone.layers.0.self_attn.q_proj.weight"].shape,   # (2048, 16)  packed
      w["backbone.layers.0.self_attn.q_proj.weight"].dtype,   # uint32
      w["backbone.layers.0.self_attn.q_proj.weight.scales"].shape)  # (2048, 16)

This repository ships weights only. The architecture keys (backbone.* plus the decision head) are not part of any public MLX model registry, so a runtime needs an MLX implementation of that forward pass; the upstream reference implementation is at TianyuCodings/NanoJev (MIT, PyTorch/CUDA), and the same checkpoint in unquantized form is available upstream and as a bf16 conversion from this account.

Verification and reproducibility

• source weights sha256 fff62d1412685c1714eaa386acb603f9690371fb3cc8ad03dc41319302597c28 (the upstream root checkpoint, v3_teacher_coords_multi_seed17)
• this artifact's sha256 332d47eb23c81b0370f580e87e0e55ef737ba1baa206a8d300eadf9617307b62
• MANIFEST.json records the source hash, the quantization configuration, the parameter counts per tensor group, the per-group weight-space error, and the hash of every produced file
• produced by scripts/quantize_mlx.py (MLX mx.quantize, then mx.dequantize for the evaluation copy); the evaluation ran through the frozen-split harness kept with the bf16 conversion workspace (compare_precision.py), against the upstream fp32 checkpoint as reference

Attribution

• Upstream model: C-Tianyu/NanoJev — code MIT, weights with no declared license.
• Upstream data: C-Tianyu/NanoJev-Data (per-record metadata.license = CC0-1.0).
• Quantization and evaluation: performed locally on the upstream checkpoint; no upstream weights were retrained or altered in value beyond the quantization itself.