Apertus-v1.5-70B-NVFP4

Apertus v1.5 70B NVFP4

Structured two-pass quantization of swiss-ai/Apertus-v1.5-70B with 54% faster decode, 32% smaller, at 99% quality compared to FP8. Uses the NVFP4 scheme: NVFP4A16 on MLP projections, FP8_DYNAMIC on attention projections, and BF16 preserved on embeddings, lm_head, and norms.

This is the recommended format for production deployment: 48 GiB leaves ample headroom on a single 96 GiB GPU for KV cache (up to ~230k context with fp8 KV cache) and multimodal encoder working memory.

Thanks to: Swiss AI, blancsw, Anunay-Yadav, Oleg, Cyrilvallez, AryanAhadinia, robmsmt, and Neural Magic. See Attribution for details.

Usage (vLLM)

Until Apertus 1.5 support is available in a native vLLM release, use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image. The build recipe and patched Dockerfile are available at github.com/swiss-ai/model-launch/tree/main/images/vllm_apertus_1.5.

# Download the checkpoint
huggingface-cli download onprem-ai/Apertus-v1.5-70B-NVFP4 \
  --local-dir ./apertus-v1.5-70b-nvfp4

# Serve it
docker run --gpus all -v ./apertus-v1.5-70b-nvfp4:/model \
  ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64 \
  --model /model \
  --served-model-name apertus-v1.5-70b \
  --host 0.0.0.0 --port 8080 \
  --dtype auto \
  --chat-template /model/chat_template.jinja \
  --tool-call-parser apertus --enable-auto-tool-choice \
  --reasoning-parser apertus

Performance: 54% Faster Single-User Decode Than FP8

Measured on NVIDIA RTX PRO 6000 Blackwell (96 GiB VRAM) with vLLM. Decode is memory-bandwidth bound, so the 48 GiB NVFP4 checkpoint outperforms the 71 GiB FP8 checkpoint while leaving 23 GiB more VRAM available.

NVFP4 concurrency scaling was measured with llmapibenchmark using random prompts and 200-token prefill/decode workloads:

Decode throughput scales up to 294 TPS at 20 concurrent requests. Prefill throughput remains near 2500 TPS through concurrency 20.

Quality: 99% of FP8 Quality yet 32% Smaller

Benchmarked via lm-evaluation-harness (MMLU, 5-shot, 100 samples) on NVIDIA RTX PRO 6000 Blackwell. The vLLM compressed-tensors backend loads the structured scheme natively with no decompression overhead.

All deltas within 1 standard error. At 2.8x compression versus the original BF16 checkpoint, this represents an excellent quality-efficiency trade-off.

Multimodal Support

All vision and audio capabilities from the original checkpoint are preserved:

• Image understanding (Emu3.5 vision encoder)
• Audio understanding (WavTokenizer audio encoder)
• Tool calling (native apertus tool call format)
• Reasoning (apertus reasoning parser)

The quantization respects the ignore list: vision tokenizer, audio tokenizer, and embedding layers remain in BF16.

Quantization Process

Two-pass calibration-free quantization using llm-compressor 0.12.0:

Pass 1: NVFP4A16 on MLP

scheme: NVFP4A16
targets: re:.*mlp\.up_proj$, re:.*mlp\.down_proj$
ignore: lm_head, model.language_model.embed_tokens,
        re:.*self_attn\.(q|k|v|o)_proj$,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*

Pass 2: FP8_DYNAMIC on Attention

scheme: FP8_DYNAMIC
targets: re:.*self_attn\.(q|k|v|o)_proj$
ignore: lm_head, model.language_model.embed_tokens,
        re:.*mlp\.(up|down)_proj$,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*

The generated quantization_config requires target group separation (fix_three_tier_quant_config.py in the build repo handles this).

Limitations

• vLLM only. SGLang does not support compressed-tensors structured checkpoints (see sgl-project/sglang#16276).
• Requires vLLM with Apertus1p5 arch support. Use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image or build from upstream PR #50496.
• Recommened 96 GiB GPU. 48 GiB weights + multimodal encoder working memory
  • KV cache do not fit on 80 GiB cards for long contexts.

Supported Hardware

Attribution

• Original model: swiss-ai/Apertus-v1.5-70B by Swiss AI (including Oleg Lavrovsky (loleg), Cyrilvallez), Apache 2.0
• GPU-native multimodal refactor: blancsw at Infomaniak -- foundational architecture work, Emu3.5 vision encoder, WavTokenizer audio encoder, apertus tool parser
• Upstream PR: vllm-project/vllm #50496 by Anunay-Yadav -- GPU-native Apertus 1.5 multimodal encoders for upstream vLLM
• Reasoning parser: AryanAhadinia
• Double BOS-token fix: robmsmt
• Quantization tooling: llm-compressor by Neural Magic
• Benchmarking: llmapibenchmark by onprem-ai