Apertus-v1.5-8B-NVFP4

Apertus v1.5 8B NVFP4

Structured two-pass quantization of swiss-ai/Apertus-v1.5-8B. Uses the NVFP4 scheme: NVFP4A16 on MLP projections, FP8_DYNAMIC on attention projections, and BF16 preserved on embeddings, lm_head, and norms.

This is the recommended format for production deployment: 8 GiB leaves ample headroom for KV cache and multimodal encoder working memory on smaller GPUs.

Thanks to: Swiss AI, blancsw, Anunay-Yadav, Oleg, Cyrilvallez, AryanAhadinia, robmsmt, and Neural Magic. See Attribution for details.

Usage (vLLM)

Until Apertus 1.5 support is available in a native vLLM release, use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image. The build recipe and patched Dockerfile are available at github.com/swiss-ai/model-launch/tree/main/images/vllm_apertus_1.5.

# Download the checkpoint
huggingface-cli download onprem-ai/Apertus-v1.5-8B-NVFP4 \
  --local-dir ./apertus-v1.5-8b-nvfp4

# Serve it
docker run --gpus all -v ./apertus-v1.5-8b-nvfp4:/model \
  ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64 \
  --model /model \
  --served-model-name apertus-v1.5-8b \
  --host 0.0.0.0 --port 8080 \
  --dtype auto \
  --chat-template /model/chat_template.jinja \
  --tool-call-parser apertus --enable-auto-tool-choice \
  --reasoning-parser apertus

Quantization Process

Two-pass calibration-free quantization using llm-compressor 0.12.0:

Pass 1: NVFP4A16 on MLP

scheme: NVFP4A16
targets: model.language_model.model.layers.<N>.mlp.up_proj
         model.language_model.model.layers.<N>.mlp.down_proj
ignore: lm_head, model.language_model.embed_tokens, model.embed_tokens,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*,
        re:.*self_attn\.(q|k|v|o)_proj$

Pass 2: FP8_DYNAMIC on attention

scheme: FP8_DYNAMIC
targets: model.language_model.model.layers.<N>.self_attn(q|k|v|o)_proj
ignore: lm_head, model.language_model.embed_tokens, model.embed_tokens,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*,
        re:.*mlp\.(up|down)_proj$

BF16 preserved on: lm_head, embeddings, vision tokenizer, audio tokenizer, all RMS norms, and all xIELU parameters.

Limitations

• vLLM only. SGLang does not support compressed-tensors mixed-precision checkpoints.
• Requires vLLM with Apertus1p5 arch support. Use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image or build from upstream PR #50496.

Attribution

• Original model: swiss-ai/Apertus-v1.5-8B by Swiss AI, Apache 2.0
• GPU-native multimodal refactor: blancsw at Infomaniak
• Upstream PR: vllm-project/vllm #50496 by Anunay-Yadav
• Reasoning parser: AryanAhadinia
• Quantization tooling: llm-compressor by Neural Magic