Qwen3.6-27B-NVFP4-FR

Qwen3.6-27B-NVFP4-FR

An NVFP4 (W4A4) quantization of Qwen/Qwen3.6-27B, calibrated on a French on-policy corpus, with a per-layer precision map determined experimentally to preserve French writing quality at a level statistically indistinguable from the FP8 reference.

Built for inference on NVIDIA DGX Spark (GB10, sm_121) under vLLM, paired with the Qwen3.6-27B-DSpark-FR drafter for speculative decoding.

French-focused, but broadly useful: the methodology (domain-specific calibration + per-layer precision map + blind quality certification) applies to any language or domain.

What this model brings

Most public NVFP4 quants quantize all linear layers and report out-of-context speed numbers with no quality protocol. This one instead documents:

1. A per-layer precision map from empirical exploration — the 48 Gated DeltaNet layers are kept in BF16 because quantizing them measurably degrades quality.
2. Domain-specific calibration (French on-policy).
3. Quality certification via token-level ΔNLL (with self-reference-bias control) + a blind A/B test judged by a panel.

Precision map

Rationale: quantization error in the decay gates of the recurrent layers compounds multiplicatively through the state, degrading coherence. Three more aggressive variants (GDN in FP8) were tested and rejected at the quality gates (see the code repo report).

Performance (measured, DGX Spark GB10, vLLM, with DSpark-FR drafter n=8)

Honest numbers: measured on a single GB10 (273 GB/s memory bandwidth). Decode is bandwidth-bound, so these are close to the hardware optimum. Higher numbers reported elsewhere come from multi-GPU (aggregate bandwidth) or a quality-degraded body.

Multi-node: the model shards in tensor parallel across two GB10s (QSFP, 1.9 µs RDMA) at 28.79 tok/s, with 35.65 GiB free KV per worker at 32k. At 32k the KV cache is 2.0 GiB/sequence (4× smaller than a dense equivalent: 48/64 layers are fixed-state Gated DeltaNet). KV stays BF16 with the drafter (fp8 KV + non-causal attention is blocked on sm_121 — vllm#41559).

Quality (certified vs FP8 reference)

• Token-level ΔNLL (150 held-out FR prompts, symmetry control): +0.016 nats ≤ 0.02 threshold → PASS.
• Perplexity ratio: ≤ 1.03 → PASS.
• Blind A/B (50 pairs, 1 operator + 3 independent blind judges): FP8 win rate 52.8% (p = 0.43) → statistically indistinguable.

Tool-calling — corrected note (2026-07-14). An earlier run measured weak tool-calling (~44% on tool-eval-bench) and this model was described as chat-focused. That was not the model: it was a vLLM bug (#43559) where prefix caching combined with speculative decoding corrupts the recurrent Gated-DeltaNet state on hybrid bodies, silently degrading tool-calling. With the fix (vLLM PR #48361) — or simply with prefix caching disabled — this body scores 88–91/100 on tool-eval-bench (Parameter Precision and Multi-Step Chains: 100%). It is suitable for tool-heavy agentic workflows once served correctly. See the drafter card caveat.

Usage (vLLM)

Requires a recent vLLM main build (DSpark drafter support) with VLLM_USE_DEEP_GEMM=0, VLLM_TEST_FORCE_FP8_MARLIN=1.

vllm serve pablohassan/Qwen3.6-27B-NVFP4-FR \
  --quantization compressed-tensors --kv-cache-dtype auto \
  --max-model-len 32768 --gpu-memory-utilization 0.60 \
  --speculative-config '{"method":"dspark","model":"pablohassan/Qwen3.6-27B-DSpark-FR","num_speculative_tokens":8}'

Without the drafter it runs as standard decode (~8 tok/s floor).

Lineage & license

• Base: Qwen/Qwen3.6-27B (Apache 2.0).
• Quantization: llm-compressor (compressed-tensors, nvfp4-pack-quantized), 256 calibration samples (192 FR on-policy + 64 EN).
• License: Apache 2.0 (inherited).

Links

• Drafter: https://huggingface.co/pablohassan/Qwen3.6-27B-DSpark-FR
• Code, recipe and technical report: https://github.com/Pablohassan/qwen3.6-27b-fr-nvfp4-dspark