Agents-A1-MTP-GGUF

Agents-A1 — GGUF with MTP speculative decoding

llama.cpp builds of InternScience/Agents-A1 with an MTP draft head grafted in — A1 shipped without the mtp.* tensors its Qwen3.5-35B-A3B base carries, so no other A1 GGUF can do --spec-type draft-mtp. This one can: +46% measured, no separate draft model.

Files

file                        size     notes
--------------------------  -------  ------------------------------------------
Agents-A1-MTP-NVFP4.gguf    20.8 GB  NVFP4 experts/attention, Q8_0 trunk, MTP head
Agents-A1-MTP-Q8_0.gguf     37.8 GB  reference quality, MTP head

Measured (RTX A6000 48 GB, -n 150 greedy)

config                       gen tok/s
---------------------------  ---------
NVFP4, no spec               127.7
NVFP4, --spec-type draft-mtp 187.0   (+46%, draft acceptance 0.62)

A 35B-class agentic MoE at 187 tok/s on a prosumer card in ~21 GB. The MTP head was grafted from the base Qwen3.5-35B-A3B — never re-trained on A1 — and acceptance holds anyway.

Blackwell (RTX PRO 6000 / RTX 50xx-class), MTP on, mean of 6 diverse prompts:

NVFP4 + draft-mtp:  305 tok/s  (287–336)

A 35B-A3B at the same speed our 9B runs — the NVFP4×MTP multiplication (verify-step batching feeds the FP4 tensor cores) reproduces on MoE. Finding writeup on the protoLabs Ornith cards.

Runtime compatibility

llama.cpp (spring-2026+), LM Studio, and recent Ollama (~0.31+): ✅. Older Ollama fails with "layer N missing attn_qkv" — update and re-pull. (Verified on the 9B sibling; if this 35B-MoE build misbehaves on your runtime, open a discussion with the error.)

Usage

llama-server -hf protoLabsAI/Agents-A1-MTP-GGUF:NVFP4 --spec-type draft-mtp -ngl 99

Requires llama.cpp with NVFP4 (type 40) + MTP support (both merged spring 2026). The quant tag resolves (:NVFP4 / :Q8_0) — filenames follow the standard convention.

Provenance & honesty notes

• Converted from bf16 + grafted MTP tensors; NVFP4 layers quantized by llama-quantize (llama.cpp b9829) with the trunk (DeltaNet/router/norms) held at Q8_0.
• The vLLM sibling Agents-A1-NVFP4 carries the full paired quality gate vs the official FP8 (matches or beats it on all four axes). This GGUF's own gate (paired vs official FP8, same judge/harness): FC 88.9% (identical to FP8) · coherence clean to 28K · reasoning 0.87 vs 0.84 at a 16K thinking budget · claw agentic paired-84: 0.60 vs 0.63.
• Two honest caveats. (1) This build reasons noticeably longer than the vLLM artifact (different quant scales → different reasoning paths) — at an 8K generation budget it exhausts thinking before answering on some tasks; give it ≥16K for reasoning-heavy work or disable thinking for quick lookups. (2) One solver-graded sequence task regresses at any budget, and Chinese-language agentic tasks run ~0.03 softer — if your workload is zh-heavy agentic, prefer the vLLM build.
• Not a vision build — text-only.

Need a different quant?

Open a Community discussion — requests usually ship within 48h. Rows: protoLabsAI/lab-benchmarks · protolabs.studio/lab.