Qwythos-27B-v1-NVFP4

Qwythos-27B-v1 — NVFP4

NVFP4 (W4A4) quantization of empero-ai/Qwythos-27B-v1, with the model's native MTP head, full vision tower, and lm_head preserved in bf16.

All credit for the model itself goes to Empero (empero-ai): Qwythos-27B-v1 is their open-weight, full-parameter reasoning model — a Qwen3.5-27B fine-tune post-trained through SFT → DPO → ESFT, released as a complete pre-RL checkpoint with nothing ablated: native multi-token-prediction head, full vision tower, and 1M-token context configuration all intact. Read the original model card for the full training story; it's worth it.

This repository only makes that model practical on smaller hardware: ~52 GB bf16 → ~20 GB NVFP4, quantized and validated on an NVIDIA DGX Spark (GB10, sm_121).

What's inside

Quantized with llm-compressor 0.12.0 (NVFP4 scheme), calibrated on 384 × 2048-token samples from HuggingFaceH4/ultrachat_200k, rendered through the model's own chat template. The llm-compressor recipe is included as recipe.yaml.

The MTP head required surgery

If you try this quant yourself, note: transformers silently drops the mtp.* tensors on load (_keys_to_ignore_on_load_unexpected), so a naive quantize-and-save loses the speculative-decoding head entirely. Here the original 15 MTP tensors were copied verbatim from the source checkpoint into mtp-bf16.safetensors, re-attached through the safetensors index, and added to the quantization_config ignore list — the same reattachment the Qwythos authors describe doing after their own training. Measured draft acceptance after quantization: ~72% at num_speculative_tokens: 3 under vLLM — the head is genuinely functional, not just present.

Context length

This repo ships the native 262,144-token rope configuration (the upstream config.json.pre_yarn), because static YaRN slightly degrades short-context quality and most local deployments run well under 262K. The original 1,048,576-token configuration is one edit away — add the upstream YaRN block back to config.json:

"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144}

Usage (vLLM)

vllm serve CocaKova/Qwythos-27B-v1-NVFP4 \
    --trust-remote-code \
    --quantization compressed-tensors \
    --max-model-len 131072 \
    --kv-cache-dtype bfloat16 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser deepseek_r1 \
    --chat-template chat_template.jinja

Sampling (from the upstream generation_config.json): temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05.

Tool calling works out of the box — the packaged chat template carries Qwen3.5's full tool-use spec; pass tools=[...] and the model emits standard tool-call blocks. Reasoning arrives as <think>…</think> (surfaced in reasoning_content by the deepseek_r1 parser).

How much does the quant cost?

Measured on a private 31-task agentic/reasoning suite (25 reasoning/coding/formatting tasks + 6 multi-turn tool-use scenarios, identical seeds and sampling for both runs):

The regressions were mild judgment slips (a wrong git flag variant, one unnecessary tool call, one missed formatting sub-check) — no gibberish, no repetition collapse, no degenerate loops. The standout property of the original survives quantization completely: across both runs, not a single reasoning-loop death, with think traces 3–5× leaner than comparable thinking models. Sanity-checked additionally via software-decompress greedy generation before release.

Numbers above are single-run and from one machine; treat them as indicative, not gospel.

Caveats

• Qwythos is intentionally uncensored for technical and research use (see the original card); apply your own application-level controls.
• Vision was frozen during upstream training and is inherited from the Qwen3.5 base; the tower is preserved here in bf16 but image behavior is untested post-quant.
• This is the pre-RL v1 checkpoint; Empero have said an RL-trained v2 will follow.

Credits

• Empero — the model. Training curriculum, data pipeline, and the decision to ship it un-ablated are all theirs.
• Qwen team — the Qwen3.5-27B base.
• vLLM / llm-compressor — the quantization tooling.
• Quantized and validated by CocaKova.

Licensed Apache-2.0, same as the original.