Nanbeige4.2-3B-NVFP4A16

Nanbeige4.2-3B-NVFP4A16 — ⚠ summarization-scoped

Uniform NVFP4 weight-only (W4A16, group-16) quantization of Nanbeige/Nanbeige4.2-3B. 3.6 GB, the fastest artifact in the family (175 tok/s freeform / 220 summarize on RTX 5090) — but read the gate table: use it for summarization/extraction-class work, NOT for reasoning. The failures are published on purpose; they are the first FP4 datapoints for a looped/weight-shared LLM anywhere (prior art, LoopQ arXiv:2605.16343, is INT-only).

Quality gate table (RTX 5090, vLLM v0.25.1; champion = my FP8-Dynamic quant)

Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, this artifact's headline. Both rows measured WITH ngram speculation (its intended copy-heavy workloads):

Two workload anchors, not an ISL sweep. The Marlin W4A16 kernel plus the looped arch's double weight-read per token is where the speedup comes from: weight compression pays out twice per token on this model.

Why reasoning breaks: this architecture executes each layer twice per token (num_loops: 2) — 4-bit weight error compounds across both passes. Shallow extractive work (summarization stays judge-tied with FP8 at perfect faithfulness) survives; long reasoning chains don't. A contributing mechanism, verified by super-weight scan (Apple 2411.07191): this model's super weight (layers.1.mlp.down_proj.weight[1252, 6883], the single most load-bearing scalar) sits in a 4-bit region under uniform quantization; the LoopShield recipe holds it in FP8. A mixed-precision recipe recovers full reasoning parity at 4.5 GB — see the companion repo Nanbeige4.2-3B-NVFP4-FP8-LoopShield.

Serving (vLLM)

pip install --no-deps ./vllm_plugin   # arch not upstream yet (vLLM PR #49433)
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":8,"prompt_lookup_max":4,"prompt_lookup_min":2}'

Serves via the Marlin W4A16 kernel (fastest batch-1 path on consumer Blackwell — do not convert to W4A4: slower at batch 1 AND activation-quantization is this arch's worst failure mode). Non-thinking mode recommended for its intended workloads: chat_template_kwargs: {"enable_thinking": false}.

Download just this artifact:

hf download NullSense/Nanbeige4.2-3B-NVFP4A16 --local-dir Nanbeige4.2-3B-NVFP4A16

Creation

llmcompressor QuantizationModifier(targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]), data-free. GPTQ was attempted and is not usable on this architecture (llmcompressor 0.12: Hessian inversion fails on every module — root cause unknown; the double-fire hook was checked and is not it → "Failed to invert hessian" on 154/154 modules, silent RTN fallback — reported upstream); AWQ lacks arch mappings. So RTN is currently the only plain uniform-4-bit route for looped models, which is precisely why the reasoning gate fails and why I scope this artifact.

Limitations

• Not for reasoning/agentic use — see gate table.
• vLLM-only until PR #49433 merges; bundled modeling file carries two one-line transformers-5 compat patches.
• English-only evals, English-only calibration-free quant on a bilingual model.

Which artifact should I choose?

All three serve identically (same plugin, same flags); only the checkpoint differs.

Comparison chain: the columns here use my FP8-Dynamic quant as reference; the FP8 card carries the bf16-original matrix linking the chain back to the unquantized model.

Links & provenance

• Base model: Nanbeige/Nanbeige4.2-3B
• Family: FP8-Dynamic · NVFP4-FP8-LoopShield · NVFP4A16 · EAGLE3 draft
• Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
• Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
• Quantized with llm-compressor 0.12.0 (compressed-tensors format)

Benchmark provenance (ordered: public-harness first, then my closed harnesses)

Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):

1. GSM8K — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
2. IFEval — verifiable instruction following (non-thinking, n=250)
3. MMLU-Pro — 10-choice knowledge/reasoning (non-thinking, 25/category)
4. BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
5. MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):

• Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
• Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
• JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.

Citation

@misc{peciukonis2026nanbeige42nvfp4a16,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {Nanbeige4.2-3B-NVFP4A16: uniform 4-bit quantization with published failure modes},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16},
  note         = {Speed/summarization-scoped artifact; reasoning regression (GSM8K 89->81) documented on-card as a negative result for looped-arch uniform 4-bit.}
}