Mistral-Medium-3.5-128B-W4A16-KV8

Mistral-Medium-3.5-128B-W4A16-KV8

INT4 (W4A16, GPTQ) quantisation of mistralai/Mistral-Medium-3.5-128B, calibrated on German text at 8192-token sequences, with static fp8 KV-cache scales written into the checkpoint.

70 GiB, roughly half the original. German perplexity degrades 3.09% against the FP8 reference; English 8.68%. The fp8 KV cache is effectively free on German (−0.01%) while roughly doubling the KV budget.

The reference here is FP8, not BF16. Mistral released this model in FP8 only — there is no full-precision checkpoint to quantise from or measure against. The figures below therefore describe what INT4 costs on top of Mistral's own 8-bit step. Total degradation from the original training weights is higher and not measurable from public artefacts.

---

Files and size

recipe.yaml in this repository records the exact recipe.

---

Usage

vllm serve <path-or-repo> \
  --tokenizer mistralai/Mistral-Medium-3.5-128B \
  --tokenizer-mode mistral \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --enable-auto-tool-choice \
  --dtype auto \
  --limit-mm-per-prompt '{"image":0}' \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92

--tokenizer and --tokenizer-mode mistral are required. The architecture forces MistralTokenizer, and this checkpoint carries HF-format tokenizer files rather than the native tekken.json, so vLLM must be pointed at the source repo for it. Only the weights were quantised; the tokenizer is unchanged.

At 70 GiB this is ~35 GiB per GPU at TP=2.

KV cache precision

--kv-cache-dtype auto resolves to fp8 because this checkpoint carries KV scales. On this model that is close to free:

Zero cost on German — noise-level, marginally negative — for roughly double the KV budget. Pass --kv-cache-dtype bfloat16 to decline it.

That asymmetry is not a coincidence: the KV scales were computed from German activations, and German is what they fit. It is the cleanest evidence in this family that the calibration language reaches the KV path, because the scales are the only thing differing between those two measurements.

---

Evaluation

Perplexity and top-1 agreement on 64 held-out samples × 2048 tokens per language from wikimedia/wikipedia (20231101.de / 20231101.en), 131,008 scored positions each. Generations greedy (temperature=0, top_p=1, seed=0). Measured on NVIDIA H200 (SM90) under vLLM. Eval samples use a different shuffle seed from the calibration set (1234 vs 42).

Perplexity

Top-1 agreement with the FP8 reference

Fraction of positions where the argmax token is unchanged. More sensitive than perplexity to damage that shows up as malformed morphology.

German damage is 37% of English damage

Reported in absolute nats rather than as a ratio, because the two languages start from different baselines and a sharper distribution shows a larger relative rise from the same absolute damage.

Generation quality

770,000 characters generated across four sampling configurations — greedy, temperature=0.7, temperature=0.8, and temperature=0.8 with min_p=0.05 — on two long German prompts, one of them deliberately loaded with Chinese institution names in Hanzi.

Zero non-Latin token bleed in every configuration, including the CJK-carrying prompt at temperature 0.8. Chinese terminology appearing in the output was correct usage — organisations named and glossed in German analytic prose — not corruption.

No temperature recommendation follows from this, because no artefacts were found to mitigate. Use the sampling settings the base model recommends.

Compared with an AutoRound build

An AutoRound build from the same base and the same German calibration corpus:

GPTQ is marginally ahead on German and level on top-1 agreement; AutoRound is better on English. The deciding factor was the KV scales: AutoRound accepts --static_kv_dtype and then silently ignores it under --format llm_compressor, producing a checkpoint whose kv_cache_scheme is null with no warning in the log.

Note the two builds also differ in calibration construction — AutoRound used 225 packed blocks, GPTQ 512 single documents — so this is not a clean algorithm-versus-algorithm comparison.

---

Limitations

The reference is FP8. Every figure above measures INT4 on top of Mistral's own 8-bit quantisation. The compounded loss from the original training weights is larger and cannot be measured without weights that were never released.

No standard benchmark suite. No MMLU, HellaSwag or similar. Perplexity, top-1 agreement, and targeted German generation only.

Calibration corpus is general German, not domain text: 512 samples × 8192 tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered to samples that fill the full window.

Evaluated at 2048-token sequences despite being calibrated at 8192. Not validated at the model's full 256k window.

Quantisation does not reduce hallucination, and may increase it. Factual recall is concentrated in relatively few weights while fluency is distributed, so a quantised model can retain its confident register while losing specifics. The measurements here — perplexity, argmax agreement — are largely blind to that. For grounded generation, verify that names, dates, figures and URLs in the output trace back to the input.

Vision is untested. The Pixtral tower is left unquantised, but no image-input evaluation was performed. Serve with --limit-mm-per-prompt '{"image":0}' unless you intend to test it yourself.

---

Reproducing this checkpoint

The quantisation script is in this repository as quantize_model.py.

1. Environment

python3 -m venv venv && source venv/bin/activate
export PYTHONNOUSERSITE=1

pip install transformers llmcompressor datasets accelerate

# Mistral's FP8 weights run through a fused kernel that needs this exact range
pip install "kernels==0.15.2"

Verify the pair imports — llmcompressor and compressed-tensors must match, and vLLM pins compressed-tensors exactly, so do not install vLLM into the same venv:

from compressed_tensors.quantization import is_cached_attention_module
from llmcompressor.modifiers.quantization import GPTQModifier

2. Smoke test

Runs the entire path — model load, fp8 dequantisation, module walk, GPTQ, serialisation, the output assertions — against a token-sized calibration set.

MODEL=mistral-medium-3.5 QUANT_TESTING=1 QUANT_KV_FP8=1 \
  python quantize_model.py

3. Build

MODEL=mistral-medium-3.5 QUANT_SCHEME=w4a16 QUANT_KV_FP8=1 \
SEQLEN=8192 NSAMP=512 \
  CUDA_VISIBLE_DEVICES=0 python quantize_model.py 2>&1 | tee quant.log

Roughly 14 hours on one H200, plus ~10 minutes of fp8 dequantisation and however long the calibration filter takes to find 512 German Wikipedia articles above 8192 tokens. Needs ~250 GiB of CPU RAM for the dequantised model.

Watch the first block for:

targets: ['FP8Linear'] (not plain Linear -- the source checkpoint already
                        carries a quantisation format)
target: model.language_model (Ministral3Model), 616 of 956 Linears
starting GPTQModifier on Ministral3Model: 616 Linears

A different module or a different count means the walk picked the wrong submodule and the vision tower would be quantised — stop there.

Configuration

Notes for this architecture

• The source weights are fp8, and llm-compressor cannot observe them. Its min/max observer calls torch.amin, which has no CUDA kernel for Float8_e4m3fn; the run dies at the first sequential epoch end with NotImplementedError: "min_values_cuda" not implemented. The script therefore dequantises every fp8 Linear to bf16 first, handling the block-wise scales (weight_scale_inv over 128×128 tiles). Nothing is recovered by this — the fp8 rounding is already in the values — it only makes them observable.
• targets="Linear" matches nothing in the language model, because its modules are FP8Linear before dequantisation. llm-compressor resolves targets by class name. The script derives the target classes from the model rather than assuming.
• The module walk must not count only nn.Linear. Before dequantisation the language model holds 616 FP8Linear and the vision tower 340 plain Linear, so a naive count selects the vision tower — and would quantise it while leaving the language model untouched.
• Mistral3ForConditionalGeneration dispatches to vision and text branches; pass model.language_model (Ministral3Model) to oneshot(), then save the parent so the wrapper config and lm_head are written too.
• save_pretrained must be patched on the parent via modify_save_pretrained(), or the weights are written as fake-quantised BF16 — all of the quality loss, none of the size saving. The patch uses functools.wraps, so __qualname__ still reports PreTrainedModel.save_pretrained when it is active; test the _overridden attribute instead.
• Adding KV scales to a finished checkpoint does not work. Re-quantising an existing checkpoint recomputes scales with memoryless_minmax, which differ from whatever the original quantiser chose, and remaps every weight onto a different grid. One such attempt on a sister model produced +326% perplexity. Build the KV scales in the same pass as the weights.

recipe = GPTQModifier(
    targets=["Linear"],          # after dequantisation
    scheme="W4A16",
    ignore=["lm_head", "re:.*vision_tower.*", "re:.*multi_modal_projector.*"],
    kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor",
                     "symmetric": True, "dynamic": False},
)

---

License and provenance

Inherits the licence of mistralai/Mistral-Medium-3.5-128B — a modified MIT licence under which enterprises above an unpublished revenue threshold are directed to Mistral's paid API rather than self-hosting. Consult the upstream repository before deploying commercially.

Quantisation only — no fine-tuning, no additional training data, no modification of model behaviour beyond numerical precision.