Downloads · 30 days
43
3% of all-time downloads
nilayparikh/Qwen3.6-35B-A3B-NVFP4-MTP-GGUF
Qwen3.6-35B-A3B-NVFP4-MTP-GGUF is a text generation model from nilayparikh. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
NVFP4-quantized, multimodal GGUF build of Qwen3.6-35B-A3B for llama.cpp on NVIDIA DGX Spark (GB10, SM121).
Downloads · 30 days
43
3% of all-time downloads
All-time downloads
1.7K
Public
Repo size
25.1 GB
Likes
1
Public
Click a slice to open those files.
.gguf25.1 GB · 100%
From the Hugging Face model README
NVFP4-quantized, multimodal GGUF build of Qwen3.6-35B-A3B for llama.cpp on NVIDIA DGX Spark (GB10, SM121).
This is the fast, large-context, vision-capable model for the .init AI engineering platform. It's exposed as the .INIT/Flash alias in LiteLLM and is the go-to model for long-context tasks and multimodal reasoning on DGX Spark.
| Property | Value |
|---|---|
.INIT/ alias | .INIT/Flash |
| Docker profile | llama-qwen-3-6-35b-a3b |
| Host port | 8001 |
| Use case | Large-context analysis, vision tasks, fast inference |
| Context | 256K tokens (entire repositories + images) |
| Vision | ✅ mmproj BF16 GGUF for multimodal reasoning |
These are the settings used in docker-compose.interface.yml:
llama-server \
-m /models/model.gguf \
-mm /models/mmproj.gguf \
-a qwen3.6_35b_a3b \
--jinja --chat-template-file /workspace/chat_template.jinja \
--reasoning on --reasoning-format deepseek --reasoning-budget 8192 \
--min-p 0.05 \
--spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 \
--spec-draft-p-min 0.88 --spec-draft-ngl 99 \
-ctk f16 -ctv f16 -ngl all -fa on -sm none -fit off \
-c 262144 --rope-scaling yarn --yarn-orig-ctx 131072 \
-b 2048 -ub 512 \
--parallel 1 --cont-batching --cache-prompt --swa-full \
-t 8 -tb 8 --mlock \
--port 8080 --host 0.0.0.0 --metrics --timeout 120
| Task | Use .INIT/Flash (35B A3B) | Use .INIT/Pro (27B) |
|---|---|---|
| Quick code generation | ✅ Fast | Better quality |
| Full repo scan (50K-256K tokens) | ✅ 256K context | ❌ 128K limit |
| UI screenshot analysis | ✅ Vision mmproj | ❌ Text-only |
| Diagram / chart reasoning | ✅ Vision mmproj | ❌ Text-only |
| Deep debugging / architecture | Good | ✅ Better reasoning |
| Chain-of-thought tasks | Good | ✅ Deeper reasoning |
sakamakismile/Qwen3.6-35B-A3B-NVFP4 — NVFP4 quantization + MTP restoration recipenvidia-modelopt (group_size=16)neuralmagic/calibration (20 samples)Deploying NVFP4 via vLLM on DGX Spark (SM120/SM121) hits multiple blockers:
| Component | Why |
|---|---|
| llama.cpp | No TMEM dependency — GGUF loads weights directly into GPU memory without layout transformations that require server-class hardware |
| ModelOpt NVFP4 | NVIDIA's own quantizer produces compact weights (~24 GB for 35B A3B) with native BF16 MTP preservation |
| MTP + n-gram | Dual speculative decoding path achieves high throughput on DGX Spark without vLLM's MTP bugs |
| Multimodal | Separate BF16 mmproj GGUF enables vision-language tasks without re-quantizing the projector |
| File | Size | Description |
|---|---|---|
qwen3.6-35b-a3b-nvfp4-mtp.gguf | ~22 GB | Main model weights (NVFP4 quantized) |
mmproj-BF16.gguf | ~863 MB | Multimodal projector (BF16, for vision tasks) |
| Condition | Throughput | Notes |
|---|---|---|
| DGX Spark, short prompts | ~30 tok/s | MTP n=2 + ngram speculative decoding, model fully on GPU |
| DGX Spark, long context (256K) | ~18–28 tok/s | YaRN RoPE scaling, KV cache grows with context |
30 tok/s achieved when:
--spec-type draft-mtp,ngram-mod, --spec-draft-n-max 2)-ngl all, --mlock)llama-server \
-m qwen3.6-35b-a3b-nvfp4-mtp.gguf \
-mm mmproj-BF16.gguf \
-a qwen3.6_35b_a3b \
--jinja \
--chat-template-file /workspace/chat_template.jinja \
--reasoning on \
--reasoning-format deepseek \
--reasoning-budget 8192 \
--min-p 0.05 \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--spec-draft-p-min 0.88 \
--spec-draft-ngl 99 \
-ctk f16 -ctv f16 \
-ngl all \
-fa on \
-sm none \
-fit off \
-c 262144 \
--rope-scaling yarn \
--yarn-orig-ctx 131072 \
-b 2048 \
-ub 512 \
--parallel 1 \
--cont-batching \
--cache-prompt \
--swa-full \
-t 8 -tb 8 \
--mlock \
--port 8080 \
--host 0.0.0.0 \
--metrics \
--timeout 120
| Flag | Value | Why |
|---|---|---|
-m | qwen3.6-35b-a3b-nvfp4-mtp.gguf | Main NVFP4-quantized model weights |
-mm | mmproj-BF16.gguf | Multimodal projector for vision-language tasks |
-c | 262144 | 256K context window for long documents |
--rope-scaling yarn + --yarn-orig-ctx 131072 | YaRN extrapolation from 128K base | Preserves quality beyond native context |
--spec-type draft-mtp,ngram-mod | MTP + n-gram hybrid | Dual speculative path for higher acceptance rate |
--spec-draft-n-max 2 | 2 draft tokens | Conservative speculation for larger model |
--spec-draft-p-min 0.88 | 88% acceptance threshold | Balanced speculation, fallback to n-gram |
--reasoning-budget 8192 | 8192 tokens | Extended reasoning budget for complex tasks |
-ngl all | All layers on GPU | No CPU offloading — DGX Spark has 128 GB |
-fa on | Flash attention | O(n) memory for long context |
-ctk f16 / -ctv f16 | F16 KV cache | Precision-critical for 256K context |
-b 2048 / -ub 512 | Prefill 2048, decode 512 | Balanced batch sizing for throughput |
--parallel 1 | 1 concurrent sequence | Larger model — single sequence avoids memory pressure |
--cont-batching | Continuous batching | Better GPU utilization under load |
--swa-full | Full sliding window attention | Better long-range attention quality |
--mlock | Lock in RAM | Prevents eviction during long generations |
# Quick start (text-only)
llama-server -m qwen3.6-35b-a3b-nvfp4-mtp.gguf --port 8081
# With multimodal projector (vision-language)
llama-server -m qwen3.6-35b-a3b-nvfp4-mtp.gguf -mm mmproj-BF16.gguf --port 8081
| Aspect | 27B Text | 35B A3B |
|---|---|---|
| Parameters | 27B | 35B + A3B MoE |
| Quantization | NVFP4 | NVFP4 |
| Multimodal | No (text-only) | Yes (mmproj BF16) |
| Context window | 128K | 256K |
| Reasoning budget | 8192 | 8192 |
| Speculative draft max | 2 | 2 |
| Parallel sequences | 1 | 1 |
| Model size | ~14 GB | ~22 GB + 863 MB mmproj |
| Host port | 8000 | 8001 |
Apache 2.0