DictaLM-3.0-24B-NVFP4-healed

DictaLM-3.0-24B-Thinking NVFP4 (GPTQ, healed)

NVFP4 (4-bit e2m1 weights with an FP8-e4m3 per-16 scale plane, 4.5 bits/element) weight-only mint of dicta-il/DictaLM-3.0-24B-Thinking, quantized with GPTQ via llm-compressor. Calibration is Hebrew-led, because this checkpoint exists to serve Hebrew.

Built for the tiyuvta serving engine on RTX Blackwell, by tiyuvta (tiyuvta.ai).

• 15 GB, down from 24 GB for the vendor FP8 and 48 GB for BF16. Fits one 32 GB card with room for a real context window.
• 280 quantized tensors, 83 kept (norms, embeddings, lm_head).
• No g_idx: actorder is off, so there is no column permutation to reconstruct.

Healed: what this variant is

This is tiyuvta/DictaLM-3.0-24B-NVFP4 after a quantization-aware LoRA healing pass, folded back into the weights and re-minted. It is a plain compressed-tensors NVFP4 checkpoint like its parent: no adapter at serve time, same 15 GB, same speed (84.3 tok/s against the parent's 84.3 on the same box).

English improves and Hebrew does not. Paired McNemar, same box, same 2000 shared sample_ids, against the parent:

At 0.7760 the healed English sits ABOVE the 24 GB vendor FP8 release (0.7735), so on English this 15 GB checkpoint beats the baseline it was built to approximate. Hebrew is statistically unchanged: 99 disagreements one way against 105 the other.

Why Hebrew did not move, since the healing deliberately targeted the reduction projections (down_proj, o_proj): those tensors were identified as the damage site on a ROUND-TO-NEAREST mint. On this GPTQ mint the same FP8 carve-out had already measured as no effect (p=0.5758), which says GPTQ's per-column error compensation had repaired them itself. Healing therefore corrected an error that was largely already gone, and the residual Hebrew gap lives somewhere that has not yet been localised on a GPTQ artifact.

Pick this one for English-leaning work; pick the parent if you want the published Hebrew number, which is the one that was measured, gated and shipped first.

Quality

Measured on Global-MMLU at n=2000 per language, 5-shot, greedy single-token (the instrument, not the serving shape), against the vendor FP8 release, which is the format this model is normally served in.

Paired McNemar on the 2000 shared sample_ids: Hebrew -1.45 points (p=0.0293), English -1.00 points. A per-tensor similarity screen cannot see a gap this size, and cannot see a much worse one either: an earlier round-to-nearest NVFP4 mint of this model scored 0.996-0.999 row cosine on every tensor family while answering Hebrew prompts with a space. Language-level evaluation is the only screen that works here.

Decode: 99.6 tok/s on one RTX 5090, three reps, vendor-default sampled (no sampling parameters in the request, so the model's own generation_config governs).

What more calibration does not buy

The shipped mint calibrates on 128 sequences of 512 tokens. Running the same recipe at 512 x 1024 (8x the tokens) and scoring it paired on the same host moved nothing: Hebrew 0.6590 -> 0.6630 (McNemar p=0.5045, 51 vs 59 disagreements) and English 0.7635 -> 0.7615 (p=0.7463). The gap to the vendor FP8 is not a calibration-volume artifact, so this artifact stays at the smaller, cheaper calibration.

What did not help

Weight-only per-channel FP8 on the reduction projections (down_proj, o_proj) is what makes a round-to-nearest NVFP4 mint of this model usable at all. Under GPTQ it buys nothing: 0.6625 vs 0.6590 Hebrew is McNemar p=0.5758 on paired samples, for 20% less decode throughput (80.5 vs 99.6 tok/s) and 3 GB more. GPTQ's per-column error compensation already does that work, so this mint is NVFP4 everywhere.

Serving

Run it locally: the checkpoint is standard compressed-tensors nvfp4-pack-quantized, so any runtime that reads that format reads this.

Reproducibility

Scored from scratch on two different hosts, weeks apart in build and driver, with identical correct-counts, not merely identical accuracies:

he  1318/2000 = 0.6590
en  1527/2000 = 0.7635

The 5-shot greedy instrument is host-independent, which is what makes the paired comparisons above mean anything. Decode throughput is NOT host-independent and is quoted for one 5090 only.

Provenance

Quantized 2026-09-06 with llm-compressor 0.13.0 plus the patch in vllm-project/llm-compressor#3144, which this mint required: IntermediatesCache rebuilt offloaded dataclasses through their own __init__, and every transformers>=5 config is a huggingface_hub @strict dataclass that rejects the internal wrapper, so the sequential pipeline (the only one a 24B fits in) could not run at all.

Calibration: dicta's own quantization calibration set, then Hebrew Wikipedia and English Wikipedia at roughly 4:1, 128 sequences of 512 tokens, seed 20260905. Global-MMLU is never touched by calibration.