Apertus-v1.5-70B-W4A16-KV8

Apertus-v1.5-70B-W4A16-KV8-de

INT4 (W4A16, GPTQ) quantisation of swiss-ai/Apertus-v1.5-70B, calibrated on German text at 8192-token sequences, with static fp8 KV-cache scales written into the checkpoint.

German perplexity degrades 0.78% against the BF16 reference at 40 GiB — roughly a third of the model's original footprint. English degrades 9.98% over the same weights. That asymmetry is the point: the calibration is German, and German is what is preserved.

Take this one for German-language work. Take the FP8 sibling if English matters more — its weights are quantised data-free, so no language is favoured.

---

Files and size

The vision and audio tokenizers are not quantised. Swiss AI note they are precision-sensitive and stay in float32 on half-precision loads; the WavTokenizer additionally uses weight-norm parametrised convolutions. recipe.yaml in this repository records the exact recipe.

---

Requirements

This checkpoint does not load with upstream transformers. Apertus 1.5 uses Apertus1p5ForConditionalGeneration, whose config class ships only in Swiss AI's transformers fork:

pip install "transformers[torch,vision,audio] @ \
  git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"

You also need a vLLM with Apertus 1.5 support — Swiss AI's fork, upstream PR #50496, or the prebuilt image onpremai/vllm-apertus-1p5. Verify both before downloading 40 GiB:

from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from vllm.model_executor.models.registry import ModelRegistry

assert "apertus1p5" in CONFIG_MAPPING_NAMES, "wrong transformers"
assert "Apertus1p5ForConditionalGeneration" in ModelRegistry.get_supported_archs()

apertus alone in CONFIG_MAPPING_NAMES is Apertus 1.0 support and is not sufficient.

---

Usage

vllm serve <path-or-repo> \
  --dtype auto \
  --enforce-eager \
  --chat-template <path>/chat_template.jinja \
  --chat-template-content-format string \
  --limit-mm-per-prompt '{"image":0,"audio":0}' \
  --enable-auto-tool-choice --tool-call-parser apertus \
  --tensor-parallel-size 2 \
  --max-model-len 98304 \
  --gpu-memory-utilization 0.96

At 40 GiB this is ~20 GiB per GPU at TP=2, which leaves room for a large KV cache on 48 GB cards.

KV cache precision

--kv-cache-dtype auto resolves to fp8 because this checkpoint carries KV scales, roughly doubling the KV budget. Unlike the FP8 sibling, that is not free here:

So the trade is roughly double the context for 0.7% German perplexity. Pass --kv-cache-dtype bfloat16 to decline it. One checkpoint covers both.

Known serving issue

On some builds vLLM's xIELU activation falls back to forward_native under torch.compile and then emits a log line dynamo cannot trace:

torch._dynamo.exc.Unsupported: logging.Logger method not supported

Workaround: --enforce-eager. This is a vLLM/xIELU interaction, not a property of the checkpoint.

---

Evaluation

Everything below was measured. Perplexity and top-1 agreement on 64 held-out samples × 2048 tokens per language from wikimedia/wikipedia (20231101.de / 20231101.en), 131,008 scored positions each. Generations greedy (temperature=0, top_p=1, seed=0). Measured on one NVIDIA H200 (SM90) under vLLM with --enforce-eager. Eval samples use a different shuffle seed from the calibration set (1234 vs 42).

Perplexity

Top-1 agreement with BF16

Fraction of positions where the argmax token is unchanged. More sensitive than perplexity to damage that shows up as malformed morphology, and independent of the perplexity baseline.

German damage is 8% of English damage

Reported in absolute nats rather than as a ratio, because the two languages start from different baselines and a sharper distribution shows a larger relative rise from the same absolute damage.

Every W4A16 build in this family shows the same direction. The one exception is the FP8 sibling, whose weights are quantised data-free and where the asymmetry reverses — German damage becomes 152% of English. That contrast is the clearest evidence available here that the calibration data is doing the work, and it is why the FP8 build is worth having as a control.

Calibration length matters

Two GPTQ builds, identical in every respect except the calibration sequence length:

Longer calibration sequences improved German by 0.15 points. If your workload involves long inputs or long generations — this checkpoint was built for multi-thousand-token German report writing — calibrating at the depth you actually operate at is worth the extra runtime.

A separate AutoRound build calibrated at comparable depth landed at +0.79%, statistically indistinguishable from this one. The apparent advantage of AutoRound over GPTQ in an earlier comparison was calibration length, not the algorithm.

Generation probes

Two five-sentence German prompts, each requiring five specific terms, greedy. Fluent, grammatical German with well-formed nominal compounds; zero non-Latin token bleed in either KV mode.

All 70B builds measured

Same base model, same German calibration corpus, same BF16 reference, same eval:

The ranking is monotonic in scale granularity, not bit width. Group-128 INT4 beats per-channel FP8, which beats per-tensor FP8, even though the bit widths run the other way. The last row also quantises activations statically, which compounds the problem: a fixed per-tensor activation scale must accommodate the largest outlier seen during calibration, crushing the resolution available to everything else.

---

Limitations

English degrades roughly ten times as much as German. That is the intended trade, but it is a real cost. If your workload is bilingual, measure it, or use the FP8 sibling.

No standard benchmark suite. No MMLU, HellaSwag or similar. Perplexity, top-1 agreement and targeted German probes only.

Calibration corpus is general German, not domain text: 512 samples × 8192 tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered to samples that fill the full window. Domain-matched calibration would plausibly do better on that domain and is untested here.

Evaluated at 2048-token sequences. The calibration is at 8192, but the perplexity eval is at 2048, so the long-context benefit is inferred from the calibration-length comparison rather than measured directly at depth. Not validated at the model's full 262,144-token window.

No English-calibrated control. German degrades far less than English on every calibrated build, and the data-free FP8 build reverses that — strong circumstantial evidence, but a matched English-calibrated build would settle it and does not exist.

Single-GPU measurement. All numbers from one NVIDIA H200 (SM90). Not tested under tensor parallelism, and not measured on Ada (SM89) despite that being the intended serving target.

---

Reproducing this checkpoint

The quantisation script is in this repository as quantize_model.py.

1. Environment

# Isolated venv. The swiss-ai transformers fork is OLDER than current upstream
# releases and will replace whatever you have, so do not install it globally.
python3 -m venv venv && source venv/bin/activate
export PYTHONNOUSERSITE=1

pip install "transformers[torch,vision,audio] @ \
  git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"
pip install llmcompressor datasets accelerate

hf auth login                       # swiss-ai repos are gated

Verify the pair imports before going further — this is the failure mode you are most likely to hit:

from compressed_tensors.quantization import is_cached_attention_module
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
assert "apertus1p5" in CONFIG_MAPPING_NAMES

2. Smoke test

Runs the entire path — model load, module walk, GPTQ, serialisation, the output assertions — against a token-sized calibration set. Minutes rather than hours, and it catches environment problems before an eleven-hour run does.

MODEL=apertus-8b QUANT_TESTING=1 QUANT_KV_FP8=1 python quantize_model.py

Writes to TESTING-Apertus-v1.5-8B-W4A16-KV8-de, which is useless as a model but structurally identical. Delete it afterwards.

3. Build this checkpoint

MODEL=apertus-70b QUANT_SCHEME=w4a16 QUANT_KV_FP8=1 SEQLEN=8192 NSAMP=512 \
  CUDA_VISIBLE_DEVICES=0 python quantize_model.py 2>&1 | tee quant.log

Output lands in Apertus-v1.5-70B-W4A16-KV8-de (this repository drops nothing but the internal -seq8192 suffix used to distinguish it from a 4096-token build). About 11 hours on one H200 and 40 GiB written. The calibration set is streamed from German Wikipedia on the first run and cached as calib_de_512x8192.parquet; subsequent runs reuse it.

Watch the first block for starting GPTQModifier on Apertus1p5TextModel: 480 Linears. If it reports a different module or a different count, the module walk picked the wrong submodule and the encoders would be quantised — stop there.

Configuration

Everything is an environment variable, so the same invocation works under tmux or a scheduler.

The calibration corpus is the piece most worth changing. quantize_model.py streams general German Wikipedia because that keeps a German-vs-English comparison symmetric — but for a specific domain, replacing build_calibration() with text from that domain is the single highest-leverage edit, and it costs nothing extra at build time.

Two things that will bite

llmcompressor and compressed-tensors must be a matched pair, and vLLM pins compressed-tensors exactly — do not install vLLM into the same venv, it will drag the version back and break the import with cannot import name 'is_cached_attention_module'.

The fused xIELU CUDA kernel is optional and several times faster, but note that xielu 0.1 diverges in the backward pass at 70B scale — every block reports loss iter 0: nan -> iter 0: 3.4028e38, meaning no optimisation happened. It is fine at 8B and fine for GPTQ, which is forward-only. Uninstall it before any gradient-based run at 70B.

Notes for this architecture

Quantised with llm-compressor 0.13.0. These cost real time to find:

• Apertus1p5ForConditionalGeneration and Apertus1p5Model both lack their own forward(), so llm-compressor can trace neither. Pass model.model.language_model (Apertus1p5TextModel) to oneshot(), then save the parent so the wrapper config, lm_head and encoders are written too.
• Quantising the wrapper matches more Linears than intended: the audio tokenizer contributes 25 (24 Vocos backbone + 1 ISTFT head). llm-compressor matches module paths, and the path is audio_tokenizer even though the class is WavTokenizerModel, so a regex on wavtokenizer silently matches nothing. Targeting language_model avoids this structurally.
• A submodule has no name_or_path, so llm-compressor cannot auto-load a processor. Pass processor=tok explicitly.
• save_pretrained must be patched on the parent via modify_save_pretrained(), or the weights are written as fake-quantised BF16 — all of the quality loss, none of the size saving. Check the output size, not just that it saved. The patch uses functools.wraps, so __qualname__ still reports PreTrainedModel.save_pretrained even when active; test the _overridden attribute instead.
• _retie_embeddings() compares input and output embeddings with torch.equal() without aligning devices, and compression onloads modules to the GPU before it runs. On an untied model it returns early anyway, so the script patches it to skip.
• Adding KV scales to a finished checkpoint does not work if the weights came from a learned-scale quantiser. Re-quantising with memoryless_minmax recomputes scales that differ from the learned ones and remaps every weight onto a different grid; one such attempt produced +326% perplexity. Build the KV scales in the same pass as the weights.

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:.*vision_tokenizer.*", "re:.*audio_tokenizer.*"],
    kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor",
                     "symmetric": True, "dynamic": False},
)

---

License and provenance

Inherits the licence and usage policy of swiss-ai/Apertus-v1.5-70B. The base model ships EU AI Act documentation and a public training-data summary; consult the upstream repository for both.

Quantisation only — no fine-tuning, no additional training data, no modification of model behaviour beyond numerical precision.