Orion-26B-A4B-v1-NVFP4

Orion-26B-A4B-v1 — NVFP4 (W4A4, GPTQ)

NVFP4 quantization of TheDrummer/Orion-26B-A4B-v1 for vLLM on Blackwell. Built and validated on a DGX Spark (GB10).

16.7 GB · 4-bit weights & activations, group size 16, FP8 scales · compressed-tensors, format nvfp4-pack-quantized.

⚠️ This replaces an earlier, defective upload

The first version of this repo was quantized with plain round-to-nearest and collapsed into repeated-token garbage on any generation past roughly 800 tokens. Short replies looked fine, which is exactly why it shipped. If you pulled this repo before 2026-09-22, re-download.

What makes this build work

Two things, both necessary:

1. GPTQ error compensation. Plain QuantizationModifier (round-to-nearest) — what the Gemma-4 NVFP4 cookbook shows — produces a model that degenerates on long output. Switching to GPTQModifier with an MSE weight observer fixed it. This was worth far more than any other change tried.

2. The dense shared MLP's down_proj is left unquantized. Gemma-4 MoE has a dense shared MLP alongside its 128 routed experts. Auditing the per-group weight_scale dynamic range shows that path carries far larger weight outliers than the experts:

Unlike any single expert, that shared MLP is on every token's path, so 4-bit error there hits every token and compounds until generation falls apart. Experts outnumber it 128:1, so excluding down_proj costs almost nothing in size. (llama.cpp reaches the same conclusion independently — its Q4_K_M recipe keeps feed_forward.w2, the same tensor, at Q6_K.)

All 128 routed experts are quantized; only the dense down_proj, plus lm_head, routers, embeddings and the vision tower, are excluded.

Verified

Measured on a DGX Spark (GB10), vLLM, --max-model-len 131072:

Serving

vllm serve /path/to/Orion-26B-A4B-v1-NVFP4 \
  --served-model-name orion-nvfp4 \
  --quantization compressed-tensors \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.55 \
  --enable-chunked-prefill --enable-prefix-caching \
  --trust-remote-code

Notes and limitations

• Reasoning is not parsed out on vLLM. Orion delimits thoughts with <|channel>thought … <channel|> rather than <think> tags, and vLLM has no reasoning parser for that format, so with enable_thinking: true the reasoning arrives inline in message.content. If your downstream consumes content directly (a writing pipeline, for example), either leave thinking off or strip the channel markers yourself.
• Validated for text. The vision tower is carried through unquantized but was not exercised.
• Quantization calibration used 20 general samples plus 20 reasoning traces at 8192 tokens.

Reproducing

recipe.yaml is included. The build used llm-compressor 0.13.0 / compressed-tensors 0.18.0, GPTQModifier(scheme="NVFP4", weight_observer="mse", dampening_frac=0.01) with pipeline="sequential", ignoring re:model\.language_model\.layers\.\d+\.mlp\.down_proj in addition to the usual lm_head / embeddings / routers / vision tower.

If you validate a quant of a model like this, generate at least 1500–2000 tokens. Short benchmarks (200-token throughput runs, brief coherence probes) reported a clean PASS on the broken build. The failure only appears at length, and it shows up as hyphen and punctuation runs (-k-k-k-, --- ---) that a naive repeated-word check will score as clean.