Apertus-v1.5-8B-W4A16-KV8-de

Apertus-v1.5-8B-W4A16-KV8-de

INT4 (W4A16, GPTQ) quantisation of swiss-ai/Apertus-v1.5-8B, calibrated on German text, with static fp8 KV-cache scales baked into the checkpoint.

Two things distinguish it from a generic 4-bit quant:

1. German calibration data. Public 4-bit quants are almost universally calibrated on English corpora. German morphology — compound formation, adjectival inflection, case endings — sits in exactly the low-margin region of the output distribution that 4-bit rounding disturbs. Measured here: German perplexity degrades +3.09% against the BF16 base while English degrades +6.78% (see Evaluation — and the Limitations on why this is suggestive rather than causal).
2. Calibrated KV scales in the checkpoint. vLLM's runtime --calculate-kv-scales computes scales during startup profiling, i.e. against dummy tokens. In our testing that produced unusable output. Scales computed here from real calibration activations are written into config.json and used automatically, costing +0.04% perplexity for roughly double the KV budget.

Scope. An 8B quantisation intended for German-language work. See Limitations for what has and has not been measured.

---

Files and size

The vision and audio tokenizers are not quantised. Swiss AI note they are precision-sensitive and stay in float32 on half-precision loads; the WavTokenizer additionally uses weight-norm parametrised convolutions that GPTQ would not handle cleanly.

---

Requirements

This checkpoint does not load with upstream transformers. Apertus 1.5 uses Apertus1p5ForConditionalGeneration, whose config class ships only in Swiss AI's transformers fork:

pip install "transformers[torch,vision,audio] @ git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"

Stock vLLM does not recognise this architecture either. Pick one of the three options below.

Option A — Swiss AI's vLLM fork (matched pair)

Pinned to the same commit the transformers fork above targets, so the two are known to work together:

VLLM_USE_PRECOMPILED=1 pip install \
  "git+https://github.com/swiss-ai/vllm.git@a601a9d998ddeb488f0c17e8512874b116aa7658"

VLLM_USE_PRECOMPILED=1 reuses prebuilt CUDA kernels and only compiles the Python layer — minutes rather than an hour. Drop it if the build fails or you hit an undefined symbol on import, and set TORCH_CUDA_ARCH_LIST to your compute capability (e.g. 8.9 for Ada, 9.0 for Hopper) to avoid building for every architecture.

Option B — upstream PR

VLLM_USE_PRECOMPILED=1 pip install \
  "git+https://github.com/vllm-project/vllm.git@refs/pull/50496/head"

This is what the checkpoint was validated against. Note it ships the model implementation but no config class, so the transformers fork above is mandatory with this option.

Option C — prebuilt container (no build step)

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v /path/to/checkpoint:/model:ro \
  onpremai/vllm-apertus-1p5:latest \
  --model /model --dtype auto \
  --chat-template /model/chat_template.jinja \
  --chat-template-content-format string \
  --limit-mm-per-prompt '{"image":0,"audio":0}'

Swiss AI also publish an image at ghcr.io/swiss-ai/vllm_apertus_1.5_release. Containers ship a matched vLLM/transformers pair and avoid the version-pinning entirely.

Install into an isolated venv

The transformers fork is older than current upstream releases and will replace whatever you have. Installing it into a shared environment breaks other models:

python3 -m venv ~/venvs/apertus && source ~/venvs/apertus/bin/activate
export PYTHONNOUSERSITE=1

Verify before downloading weights

import vllm
from transformers.models.auto.configuration_auto import CONFIG_MAPPING_NAMES
from vllm.model_executor.models.registry import ModelRegistry

assert "apertus1p5" in CONFIG_MAPPING_NAMES, "wrong transformers"
assert "Apertus1p5ForConditionalGeneration" in ModelRegistry.get_supported_archs(), "wrong vLLM"
print("ok:", vllm.__version__)

Both assertions must pass. apertus alone in CONFIG_MAPPING_NAMES is Apertus 1.0 support and is not sufficient.

---

Usage

vllm serve <path-or-repo> \
  --dtype auto \
  --chat-template <path>/chat_template.jinja \
  --chat-template-content-format string \
  --limit-mm-per-prompt '{"image":0,"audio":0}' \
  --enable-auto-tool-choice --tool-call-parser apertus \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

KV cache precision

--kv-cache-dtype auto resolves to fp8 because this checkpoint carries KV scales. Measured on one RTX 6000 Ada at --gpu-memory-utilization 0.45:

Pass --kv-cache-dtype bfloat16 explicitly if you want full-precision KV. One checkpoint covers both cases.

Known serving issue

On some builds, vLLM's xIELU activation falls back to forward_native under torch.compile and then emits a log line that dynamo cannot trace:

torch._dynamo.exc.Unsupported: logging.Logger method not supported

Workaround: --enforce-eager. This is a vLLM/xIELU interaction, not a property of the checkpoint.

---

Evaluation

Everything below was measured; nothing is extrapolated. Perplexity and top-1 agreement on 64 held-out samples x 2048 tokens per language from wikimedia/wikipedia (20231101.de / 20231101.en), 131,008 scored positions each. Generations greedy (temperature=0, top_p=1, seed=0). Measured on one NVIDIA H200 (SM90) under vLLM with --enforce-eager.

Eval samples are drawn with a different shuffle seed from the calibration set (1234 vs 42). Where the calibration parquet is present the benchmark also hash-excludes any overlapping sequence; for this run it was not, so disjointness rests on the seed alone.

Perplexity

Top-1 agreement with BF16

Fraction of positions where the argmax token is unchanged. More sensitive than perplexity to the kind of damage that shows up as malformed morphology, and independent of the perplexity baseline.

German degraded less than half as much as English

Reported in absolute nats rather than as a ratio because English starts from a lower baseline (6.83 vs 7.26), and a sharper distribution can show a larger relative rise from the same absolute damage. The asymmetry holds either way, and top-1 agreement — which does not depend on the baseline at all — points the same direction.

This is consistent with German calibration protecting German, but does not establish it; see Limitations for the alternative explanations that remain open.

fp8 KV cache is close to free

Same weights, same eval, KV cache dtype the only difference:

For roughly double the KV budget (232,368 vs 116,704 tokens on one 48 GB card at --gpu-memory-utilization 0.45). This is the payoff from calibrated static scales: vLLM's runtime --calculate-kv-scales, which calibrates against dummy tokens during startup profiling, produced unusable output on the same architecture.

Generation probes

Two five-sentence German prompts, each requiring five specific terms, greedy.

All outputs are fluent, grammatical German with well-formed nominal compounds and correct adjectival inflection; zero non-Latin token bleed anywhere. Both quantised variants omitted Gesetzgebungsverfahren from the second probe, which the base model included — a small instruction-following slip, not a morphology failure.

Separately, on a ~6,000-token input producing ~5,400 characters, fp8 and bf16 KV both stayed coherent with zero bleed, so nothing degrades at that depth.

Behavioural parity on a prompt-following edge case

Given a saved agent envelope whose payload contained its own competing output- format instructions, this checkpoint and the unquantised base model behaved identically — both followed the embedded instruction rather than the outer one. Reproduced in two clients. Not a quantisation artefact and not a capability limit: with the embedded instructions removed, the same models produce the requested report normally. The value here is the parity, since quantisation introduced no divergence in instruction-following behaviour.

---

Limitations

No English-calibrated control — the causal claim is not established. German degrades less than half as much as English under this quantisation, on two independent metrics. That is consistent with German calibration protecting German, but two alternatives remain open and neither is ruled out:

• Apertus is a Swiss model trained with heavy German representation, and may simply be more robust in German under any 4-bit scheme.
• The German eval is Wikipedia and the German calibration was also Wikipedia. The English arm has no matched calibration at all, so the German side enjoys a tighter domain match that could inflate the gap on its own.

Settling it needs one more build: identical settings, English calibration, same eval. Until that exists, read the asymmetry as suggestive, not causal.

No standard benchmark suite. No MMLU, HellaSwag, or similar. Perplexity, top-1 agreement and targeted German probes only. If you need published recovery percentages against an academic suite, this is not that kind of release.

Small instruction-following regression. Both quantised variants omitted one required term from a five-term generation probe that the base model included. One observation on one prompt, but it is the only place any quantised variant came out behind.

Calibration corpus is general German, not domain text. 512 samples × 2048 tokens from German Wikipedia (wikimedia/wikipedia, 20231101.de), filtered to samples that fill the full window. Domain-matched calibration — analytic, administrative or institutional German — would plausibly do better on that kind of text, and is untested here.

Long-form analytic synthesis is untested at this size. An earlier revision of this card claimed the 8B could not perform it. That claim was wrong: the failure it rested on was caused by a prompt containing competing embedded output-format instructions, which the model correctly followed. It is corrected here. Structured long-form synthesis has not since been evaluated systematically on this checkpoint, so treat it as unmeasured rather than as either a strength or a limitation.

fp8 KV tested to ~6k context only. Not validated at the model's full 262,144-token window. KV quantisation error accumulates with depth; test at your own context lengths before relying on it.

Single-GPU testing only, on RTX 6000 Ada (SM89). Not tested under tensor parallelism or on other architectures.

---

Reproducing this checkpoint

The quantisation script is in this repository as quantize_w4a16_de.py. It reproduces the published checkpoint exactly — 512 German Wikipedia samples of 2048 tokens, seed 42, group size 128, static fp8 KV scales.

# isolated venv: the swiss-ai transformers fork is OLDER than current upstream
# releases and will replace whatever you have
python3 -m venv venv && source venv/bin/activate

pip install "transformers[torch,vision,audio] @ \
  git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4"
pip install llmcompressor datasets

hf auth login                       # swiss-ai repos are gated

APERTUS_TESTING=1 python quantize_w4a16_de.py     # ~2 min, verifies the setup
APERTUS_KV_FP8=1  python quantize_w4a16_de.py     # ~40 min on one 48 GB card

APERTUS_TESTING=1 runs the entire path — all 32 layers, real GPTQ, real serialisation — with a token-sized calibration set. The resulting checkpoint is useless as a model but structurally identical, so it catches environment problems in minutes rather than after a full run.

Optional, but several times faster: the fused xIELU CUDA kernel. Without it transformers falls back to a Python implementation of the activation.

pip install cmake ninja
CUDA_HOME=/usr/local/cuda-<ver> TORCH_CUDA_ARCH_LIST=<arch> \
  pip install --no-build-isolation git+https://github.com/nickjbrowning/XIELU

Point CUDA_HOME at a specific version directory, not the /usr/local/cuda symlink — it often resolves to a different CUDA than your torch build, and the resulting extension fails at import with an undefined symbol.

APERTUS_SCHEME=fp8 builds an FP8 variant instead. Worth knowing what that changes: llm-compressor's FP8_DYNAMIC derives per-channel weight scales from weight min/max alone, so the calibration data reaches only the KV scales and never the weights. GPTQ's W4A16 path estimates a Hessian from the calibration activations and compensates every rounding decision against them. The two are not comparable as "8-bit versus 4-bit".

Notes for this architecture

Quantised with llm-compressor 0.13.0. These cost real time to find:

• Apertus1p5ForConditionalGeneration and Apertus1p5Model both lack their own forward(), so llm-compressor can trace neither. Pass model.model.language_model (Apertus1p5TextModel) to oneshot(), then save the parent so the wrapper config, lm_head and encoders are written too.
• Quantising the wrapper matches 217 Linears — 192 in the text model plus 25 in the audio tokenizer (24 Vocos backbone + 1 ISTFT head). llm-compressor matches module paths, and the path is audio_tokenizer even though the class is WavTokenizerModel, so a regex on wavtokenizer silently matches nothing. Targeting language_model avoids this structurally.
• A submodule has no name_or_path, so llm-compressor cannot auto-load a processor. Pass processor=tok explicitly.
• GPTQModifier in 0.13.0 accepts neither group_size nor sequential_targets; "W4A16" is a preset that already implies group-128.
• save_pretrained must be patched on the parent via modify_save_pretrained(), or the weights are written as fake-quantised BF16 — all of the quality loss, none of the size saving. Check the output size, not just that it saved. Note the patch uses functools.wraps, so __qualname__ still reports PreTrainedModel.save_pretrained even when it is active; test the _overridden attribute instead.
• _retie_embeddings() compares input and output embeddings with torch.equal() without aligning devices, and compression onloads modules to the GPU before it runs. On an untied model it would return early anyway, so the script patches it to skip.

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head", "re:.*vision_tokenizer.*", "re:.*audio_tokenizer.*"],
    kv_cache_scheme={"num_bits": 8, "type": "float", "strategy": "tensor",
                     "symmetric": True, "dynamic": False},
)

---

License and provenance

Inherits the licence and usage policy of swiss-ai/Apertus-v1.5-8B. The base model ships EU AI Act documentation and a public training-data summary; consult the upstream repository for both.

Quantisation only — no fine-tuning, no additional training data, no modification of model behaviour beyond numerical precision.