Inkling-Small-heretic-NVFP4

Inkling-Small-heretic-NVFP4

NVFP4 quantization of trohrbaugh/Inkling-Small-heretic, which is itself a Heretic-abliterated thinkingmachines/Inkling-Small.

170.7 GB, down from 532 GB in BF16.

The quantization recipe was reverse-engineered from thinkingmachines/Inkling-Small-NVFP4 and verified by regenerating Thinking Machines' own w13 expert tensors byte-for-byte across all 39 quantized layers. The result matches their checkpoint's structure exactly: 1360 tensors, identical names, shapes and dtypes.

This model has had its refusal behaviour removed by directional ablation. It will attempt to answer requests that the base model declines. Use accordingly.

Quick numbers

For scale: the abliteration that produced the parent model cost 0.0466 KL against Inkling-Small. Quantization adds roughly three and a half orders of magnitude less than that.

What is quantized

Following Thinking Machines' own exclusion list, only the routed experts of layers 3–41 are FP4 — 78 tensors, about 94.5% of parameters. Everything else stays BF16:

• all attention blocks in all 42 layers (including all 28 ablated attn.wo_ud)
• mlp.gate routers and mlp.shared_experts
• layers 0–1 dense MLPs (including the ablated layer-1 w2_md) and layer 2's experts
• norms, short convolutions, embeddings, unembedding
• the MTP head, and the vision and audio towers

This is convenient for an abliterated model: Heretic only edits output projections, so every edit outside the routed experts is carried through bit-exact. Only the expert down-projections (w2_weight) are requantized.

Format

scale2[e]   = amax(W[e]) / 6 / 448          float32, one per expert
block scale = float8_e4m3, group 16 along the input dim
weights     = E2M1, two values per byte, low nibble first

Per-block scales are chosen by exhaustive search over E4M3 codes minimizing block squared reconstruction error, with ties broken toward the lower code — not by the amax/6 rule that naive NVFP4 implementations use. That was determined empirically: TM's scales are non-monotonic in block amax and land on both sides of amax/6, which rules out any closed form. Matching it is worth about 1.4 dB over round-to-nearest (21.81 vs 20.45 dB).

Note the association order in scale2: amax/6/448 and amax/(6*448) differ by one ulp in float32 on ~26% of values, which is enough to flip E2M1 rounding on blocks near a midpoint. Quality is identical either way; byte-equality is not.

Ablation-aware rounding

Directional ablation subtracts a rank-1 component along a refusal direction v. Ordinary FP4 rounding re-injects noise along v that is uncorrelated with the removed component — and at layers where the ablation strength α ≈ 1, the deliberate residual is near zero, so that noise dominates it.

Measured on layer 3 of this model (α ≈ 0.99), where |vᵀW| = 0.135:

So the choice between the two nearest E2M1 levels is steered, per column, to cancel vᵀ(W − Wq). Applied to all 39 w2 tensors. Cost: 0.001 dB.

v was recovered as the leading left singular vector of the delta between this model's parent and Inkling-Small, averaged over all 69 edited matrices (pairwise |cos| ≥ 0.9962, median 0.9992).

This is a fidelity measure, not a behavioural change: it makes the quantized tensors' refusal-direction geometry match the BF16 parent rather than drift from it randomly.

Per-layer results across all 39 quantized layers:

Difference from the parent

One fix was applied that is unrelated to quantization. The parent model stores mlp.gate.bias and mlp.gate.global_scale in BF16; both Inkling-Small and TM's NVFP4 keep them in float32. Those are the router bias and scale for a top-k-of-256 gate, where BF16's 8-bit mantissa can reorder near-tied experts and silently change routing. All 80 tensors are restored to float32 from the base model. Their delta from base was at BF16's rounding floor (median 1.5×10⁻³), confirming they were casts rather than edits.

Inference

Requires a runtime that reads hf_quant_config.json — vLLM or TensorRT-LLM. transformers cannot load this checkpoint: TM's config.json carries no quantization_config, so transformers has no way to know the U8 tensors are packed FP4.

input_amax values are carried over from TM's export for the W4A4 activation path. They were calibrated on base activations; the abliteration shifts activations only slightly (KL 0.0466), but if you serve W4A4 it is worth confirming observed amax stays under them on your own traffic.

Verification

Each stage was checked against something with a known right answer rather than a plausible-looking number:

• Packer correctness — w13 is untouched by Heretic, so TM quantized it from identical weights. Regenerating it must reproduce their bytes exactly. It does, on all 39 layers: weight 100.0000% scale 100.0000%.
• GPU vs reference — the CUDA implementation produces bit-identical scale codes and packed bytes to an independent NumPy implementation on gaussian, outlier-heavy and heavy-tailed inputs.
• Structure — 1360 tensors against TM's 1360: 0 missing, 0 extra, 0 shape mismatches, 0 dtype mismatches. The 78 quantized modules match their exclude_modules list exactly.
• Behaviour — KL measured by collecting full 200,058-way next-token distributions from each model separately (they do not fit in memory together) and comparing offline, against the BF16 parent rather than against Inkling-Small, so the abliteration's own cost is excluded.

Limitations

• KL was measured on 16 prompts. The margin is large enough that this does not affect the conclusion, but it is a thin estimate.
• Behavioural evaluation was done on a BF16 dequantization of this checkpoint, so it measures the weight perturbation. It does not exercise FP4 tensor-core kernels or the W4A4 activation path.
• Inherits everything from the parent, including whatever the abliteration cost in capability. This quantization neither adds to nor repairs that.
• Refusal behaviour was not re-measured after quantization. Given KL ~1e-5 and 100% top-1 agreement, the model should behave as its parent does, but this is an inference from the fidelity metrics rather than a direct measurement.

Reproducing

Scripts: <!-- TODO: link -->

01_inventory.py          tensor inventory of all three repos, metadata only
02_diff.py               what the abliteration changed; recovers v
03_verify_packer.py      pin down TM's format against their shipped bytes
03c_exhaustive.py        establish the per-block scale rule
04_export.py             the export
05_finalize.py           structural validation against TM's repo
06_sanity.py             per-layer fidelity of the shipped checkpoint
07_kl_eval.py            two-phase KL evaluation

Requires one GPU with ~24 GB for the export (~16 min on an H200), plus host RAM for one 8.6 GB tensor.

Credits

• Thinking Machines for Inkling-Small and the NVFP4 reference export that made the format recoverable
• Heretic by p-e-w, via this fork

License

<!-- TODO: inherit from thinkingmachines/Inkling-Small -->