Qwen3.6-40B-Deckard-NVFP4

Qwen3.6-40B-Deckard-NVFP4

⚠️ Disclaimer: Entirely Vibe Coded! This is my first model experiment. Don't expect highest quality — built for the NVIDIA DGX Spark GB10, your mileage may vary.

NVFP4 quantization of DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking, optimized for NVIDIA Blackwell (GB10 DGX Spark) using compressed-tensors. Benchmarked with DFlash speculative decoding using z-lab/Qwen3.6-27B-DFlash as a drafter (note: mismatched target — the drafter was trained for a 27B target, not 40B).

Served via the excellent AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-NVFP4 and its accompanying vLLM AEON Ultimate DFlash Docker image, which adds DFlash speculative decoding support to vLLM.

---

Hardware

Benchmarked on a NVIDIA DGX Spark (GB10 Superchip):

• 128 GiB unified CUDA memory (CPU + GPU shared)
• Blackwell architecture (SM12.1a)
• ARM (aarch64) CPU

---

How This Was Created

1. Base Model

DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking — a 40B dense model expanded from Qwen 3.6 27B and trained on Claude 4.6 Opus reasoning data. Architecture: Qwen3_5ForConditionalGeneration (96 layers, model_type: qwen3_5).

2. NVFP4 Quantization

Quantized using llmcompressor with the following recipe:

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head'
        - 're:.*embed_tokens$'
        - 're:.*visual.*'
        - 're:.*model.visual.*'
        - 're:.*linear_attn.*'
        - 're:.*norm.*'
        - 're:.*q_norm.*'
        - 're:.*k_norm.*'
      scheme: NVFP4
      bypass_divisibility_checks: false

NVFP4 (NVIDIA FP4) is a Blackwell-native 4-bit floating-point format supported by the GB10's Tensor Cores. It gives roughly 2× memory reduction vs bf16 with near-zero quality loss, and enables the full 40B model to fit comfortably in the GB10's 128 GiB unified memory.

3. Serving Stack

NVIDIA DGX Spark (GB10, 128 GiB)
  └─ Docker: ghcr.io/aeon-7/vllm-aeon-ultimate-dflash:qwen36-v4
       ├─ vLLM (DFlash-patched fork by aeon-7)
       ├─ Target model: robbatt/Qwen3.6-40B-Deckard-NVFP4
       │    --quantization compressed-tensors
       │    --gpu-memory-utilization 0.72
       │    --tool-call-parser qwen3_xml
       └─ DFlash drafter: z-lab/Qwen3.6-27B-DFlash  ← mismatched (see notes)
            --speculative-config '{"method":"dflash","num_speculative_tokens":15}'

Docker Compose (excerpt, bench overlay adds ports: ["8000:80"]):

docker compose -f docker-compose.yml -f docker-compose.bench.yml \
  --profile bench-40b-draft up -d deckard-draft

DFlash note: DFlash (arXiv:2602.06036) is a block-diffusion speculative decoding method where a small 5-layer draft model predicts multiple tokens in parallel. The z-lab 27B drafter was trained against a 27B target's hidden states. Using it with this 40B model gives ~13.6% token acceptance (vs ~32% for a matched 27B→27B pair) — it still works and provides a real speedup, but a drafter trained specifically for the 40B target would be significantly better. If you have the resources to train one, please reach out!

---

Benchmark Results

All benchmarks run on a NVIDIA DGX Spark GB10, client on a separate x86 machine. B=N = N concurrent requests. HumanEval=20 problems, MMLU-STEM=50 questions, Tool Use=20 tasks. The 27B AEON rows use AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-NVFP4 and are included as reference baselines.

⚠️ These are minimal smoke tests to get a directional sense of performance — sample sizes are small and results are not statistically representative. Treat the numbers as rough indicators, not rigorous benchmarks.

¹ Output tokens/sec at N concurrent requests (single-user baseline → light multi-user load). Higher concurrency increases GPU utilization and amortizes fixed overhead, so throughput scales non-linearly.

†Mismatched drafter — z-lab's drafter was trained against a 27B target, not the 40B. A matched 40B drafter would push acceptance to ~32% and throughput to 4×+ at B=1.

Observations:

• The 40B scores measurably higher than the 27B on all quality benchmarks — 85–90% HumanEval vs 65%, 90–95% Tool Use vs 80%. The Claude 4.6 Opus fine-tuning and parameter expansion are visible in the numbers.
• DFlash is lossless — 40B+DFlash quality matches or exceeds the 40B solo across all three benchmarks.
• DFlash was only tested with bf16 KV cache. The flash_attn fp8 kernels used by DFlash target SM89/SM90 and are incompatible with the GB10 (SM12.1a), ruling out fp8_e4m3 KV in the current stack. Combining DFlash with turboquant k8v4 KV quantization would be an interesting further optimization but was not tested here.
• fp8_e4m3 KV cache has negligible throughput impact (±0.2%) with 2× KV memory saving.
• turboquant k8v4 reduces throughput at B=8 (−16% on the 40B) with no quality benefit.
• The matched 27B→27B DFlash (32.1% acceptance, 178.4 tok/s at B=8) shows how much headroom remains once a proper 40B-trained drafter exists.

Test scripts used are uploaded to this repo (eval/).

Memory Requirements — Theoretical (256k context, 1 agent)

Both models use a hybrid architecture (full_attention_interval=4): only every 4th layer is full attention; the rest are linear (Mamba-style) attention with O(1) state. This means only 25% of layers need a traditional KV cache — dramatically less than a pure transformer.

Architecture facts:

• 27B: 64 total layers → 16 full-attention layers (4 kv_heads × head_dim 256)
• 40B: 96 total layers → 24 full-attention layers (4 kv_heads × head_dim 256)
• Linear-attention state: ~24 MiB (27B) / ~36 MiB (40B) — sequence-length independent, negligible
• NVFP4 safetensors store each FP4 value in an FP8 container (~1 byte/param on disk and in RAM); actual FP4 compression happens at compute time in Blackwell Tensor Cores
• Measured weight sizes from HF: 24.6 GiB (27B) / 35.9 GiB (40B, this repo)
• DFlash drafter (z-lab/Qwen3.6-27B-DFlash): 3.2 GiB of weights (bf16; 5 layers + fc + norms). It shares the target's token embedding and LM head, so those add no extra memory. It also keeps a small KV cache of its own during speculation (~5 GiB at full 256k context, included in the DFlash rows below).

KV cache formula: (bytes_K + bytes_V) × full_attn_layers × kv_heads × head_dim × seq_len

• bf16 = 2+2 bytes, fp8 = 1+1 bytes, turboquant k8v4 = K:int8 (1B) + V:int4 (0.5B)
• DFlash drafter KV uses its own geometry (5 layers, 8 kv_heads, head_dim 128) → ~5 GiB at 256k
• Max concurrent assumes model weights are shared across requests (vLLM default), 6 GiB reserved for Xorg/system on GB10

A pure transformer with the same layer count would need 4× more KV cache — the hybrid architecture is a key reason this 40B model is practical on the GB10.

---

Files in This Repo

---

Attribution & Thanks

• DavidAU — for the outstanding 40B Deckard base model, which this quantization is derived from.
• z-lab — for Qwen3.6-27B-DFlash and the DFlash paper/method (arXiv:2602.06036).
• AEON-7 — for Qwen3.6-27B-AEON-Ultimate-Uncensored-NVFP4 and the vLLM AEON Ultimate DFlash Docker image that makes DFlash serving possible.

---

Known Limitations & Open Work

• No matched DFlash drafter for 40B. The z-lab drafter targets 27B hidden states. Training a proper 40B drafter (cfg6 in my notes) requires reproducing the DFlash training recipe (arXiv:2602.06036 §4.2) — feasible but compute-intensive. Happy to collaborate if you have the hardware.
• Blackwell-only. NVFP4 requires SM90+ (GB10/H100/H200/B200). Will not run on Ampere or older.
• CUDA memory note. On the GB10's unified memory, each container start/stop leaks ~6 GiB at the driver level. A full reboot between model swaps is recommended.