Qwen3.6-35B-A3B-DSV4Pro-TextOnly-FP4

Qwen3.6-35B-A3B-DSV4Pro-TextOnly-FP4

Model Summary

This is the NVFP4 (W4A16) quantized version of the Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill model, optimized specifically for text-only workflows.

Key Characteristics:

• Text-Only Optimization (No Vision): Both the speculative decoding MTP heads AND the visual tower (model.visual) have been completely pruned. This eliminates all vision overhead, resulting in a lighter model card ideal for text-only server environments.
• NVIDIA ModelOpt FP4 (GS16) Quantization: Quantized using NVIDIA ModelOpt with Group Size 16 (GS16).
  • Note on Group Size: While we tested Group Size 128 (GS128) to shrink the footprint further, the vLLM Marlin FP4 CUDA kernel (marlin_mm) only supports a group size of 16. Attempts to serve GS128 will result in a serving engine crash (Invalid thread config). Therefore, GS16 is the only viable serving configuration.
• Disk Footprint: ~19 GB (compared to 21 GB for the vision-enabled version, saving ~2 GB of storage/memory).

---

Difference from the Multimodal Version

• No Multimodal/Vision support: Unlike Qwen3.6-35B-A3B-DSV4Pro-FP4, this model cannot process images or video tokens.
• Lighter weight: The removal of the visual projection and vision model weights reduces disk size and VRAM loading overhead by ~2 GB.

---

Original Distill Model Information (nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill)

On Qwen3.6-35B-A3B (MoE, 3B active), we use LoRA to distill the way DeepSeek-V4-Pro reasons (with thinking-on) plus its agentic behavior — purpose-built as a fast task orchestrator (decompose → delegate → verify) for Lynn Agent.

This is the MoE counterpart of the 27B Dense sister model: same R6000 GPU, same teacher, same recipe, on a sparse architecture.

⚠️ Distilling a thinking style ≠ distilling knowledge/capability: the goal is "learn how to reason and how to converge", not to inject knowledge or raise the capability ceiling.

Training details

• Base: Qwen3.6-35B-A3B (MoE, 3B active, BF16 base)
• Method: LoRA, r = 64, α = 128, dropout = 0.05, targets = all attention + MLP projections
• Optim: paged_adamw_8bit, cosine LR, warmup 0.03, ~1 epoch
• Teacher: DeepSeek-V4-Pro (thinking-on + agentic)
• Data: ~1,842 distillation samples (lynn_prod spec). Trajectories = DS-V4-Pro multi-step reasoning under thinking-on (<think>) + ReAct-style tool calls (think one step → call one tool → observe → loop).
  • The tool "execution results" are SIMULATED, not actually run — each result line is improvised by a small fast model (DeepSeek-V4-Flash) role-playing the runtime, not real sandbox execution.
  • Training masks those fabricated results — the model learns only "how to think / how to call tools", never the made-up outputs (so it doesn't learn to fabricate tool returns).

Attribution (the method is not original — it is a combination of published techniques)

• ReAct (interleaved reasoning + acting): Yao et al., 2022, arXiv:2210.03629 (ICLR 2023)
• STaR (bootstrapping reasoning traces): Zelikman et al., 2022, arXiv:2203.14465
• Self-Instruct / Baize self-chat: Wang et al., 2022; Xu et al., 2023, arXiv:2304.01196
• AgentTuning: Zeng et al., 2023, arXiv:2310.12823
• ToolBench / ToolLLM: Qin et al., 2023, arXiv:2307.16789
• DeepSeek-R1 reasoning distillation: DeepSeek-AI, 2025, arXiv:2501.12948

---

Serving & Inference with vLLM

Load using the standard compressed-tensors path inside vLLM. Here is the recommended serving configuration:

vllm serve /path/to/Qwen3.6-35B-A3B-DSV4Pro-TextOnly-FP4 \
  --served-model-name nvidia/Agents-A1-FP4 \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 128000 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching

---

🇨🇳 中文版 (Chinese Version)

模型摘要

这是 Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill 模型的 NVFP4 (W4A16) 量化版本，专门针对纯文本工作流进行了优化。

• 纯文本优化 (无视觉支持): 移除了投机解码的 MTP 头以及视觉塔 (model.visual)。去除了所有的多模态视觉权重开销，从而让模型体积极小，适合纯文本服务器环境。
• NVIDIA ModelOpt FP4 (GS16) 量化: 使用 NVIDIA ModelOpt 以 Group Size 16 (GS16) 进行 PTQ 校准。
  • 关于组大小的说明: 虽然我们测试了 Group Size 128 (GS128) 以进一步减小模型体积，但在 vLLM 中，Marlin FP4 CUDA 算子 (marlin_mm) 目前仅支持 group size 为 16。尝试加载 GS128 模形将导致服务崩溃（提示 Invalid thread config）。因此，GS16 是唯一可用的运行服务配置。
• 磁盘大小: ~19 GB (对比视觉版本节省了约 2 GB)。