MiniCPM5-2B-NVFP4

MiniCPM5-2B-NVFP4

openbmb/MiniCPM5-2B quantized to NVFP4 (4-bit weights and 4-bit activations, group size 16, FP8 block scales) with llm-compressor GPTQ, in the compressed-tensors format that vLLM loads natively.

It pairs with the official openbmb/MiniCPM5-2B-DSpark draft model for speculative decoding — the drafter stays BF16 and attaches to this quantized target without any conversion. Measured 2.7x decode speedup on code.

Baked by Lna-Lab on 8x RTX PRO 2000 Blackwell (single GPU used for this model).

Read this before using it for English reasoning. Quantization changed where this model stops thinking, and not in a small way. On 250 English GSM8K problems (MGSM-en, greedy, strict parsing) this checkpoint fails to emit a closing </think> 75-77% of the time, at both a 2048- and a 6144-token budget — the BF16 original fails 19%. The reasoning itself is fine: among the answers that do terminate, 98% are correct. But if your stack needs </think> to end a turn, budget generously and expect truncation in English. In Japanese the same change runs the other way and is a clear improvement. Numbers and the whole story are in Verbosity and stopping below.

What was chosen, and why

Four rounding methods and two activation precisions were baked and measured against the BF16 original. The table below is the whole reason this checkpoint uses GPTQ:

* AWQ ran out of memory at the full calibration size on a 16 GB card and was measured with 192 samples at 1024 tokens instead of 512 at 2048. Its gap to GPTQ is far larger than that difference plausibly explains, but it is not a like-for-like comparison.

Two findings drove the recipe:

1. Activation precision is free here. NVFP4 W4A4 and weight-only NVFP4A16 land within 0.03% of each other on every corpus. MiniCPM5-2B's activations survive 4-bit dynamic group-16 quantization intact, so this checkpoint ships W4A4 and keeps the native FP4 GEMM path.
2. Weight rounding is the whole story. Round-to-nearest costs ~15% perplexity; GPTQ costs ~3%. AutoRound landed in between on this model.

lm_head and the embeddings are left in BF16.

Serving

Plain vLLM:

vllm serve sakamakismile/MiniCPM5-2B-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.5

With the official DSpark drafter (recommended — the drafter is BF16 and needs no conversion):

vllm serve sakamakismile/MiniCPM5-2B-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.5 \
  --speculative-config '{"model":"openbmb/MiniCPM5-2B-DSpark","method":"dspark","num_speculative_tokens":7}'

Sampling: the model's own generation_config.json asks for temperature 1.0 / top_p 0.95. That is too hot for Japanese (see Japanese below) — use 0.3-0.6 there.

num_speculative_tokens should be 7, the block size the drafter was trained with. vLLM resolves the drafter's target_layer_ids [1,10,20,30,39] into auxiliary hidden-state layers (2,11,21,31,40) on the quantized target automatically.

Requires a vLLM build with the DSpark speculator and Qwen3DSparkModel in the model registry.

For tool calling, add the parser — vLLM ships one for this model's XML dialect:

vllm serve sakamakismile/MiniCPM5-2B-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.5 \
  --enable-auto-tool-choice --tool-call-parser minicpm5

Verified end to end on this checkpoint: an English or Japanese request comes back with finish_reason: "tool_calls" and structured tool_calls, and feeding the tool result back produces the final answer. Without those two flags vLLM rejects any request carrying tools.

⚠️ Offline LLM.generate users: <function, <param, </param> and </function> are special tokens in this tokenizer. The default skip_special_tokens=True silently deletes them, leaving debris like name="get_weather"> name="city">Osaka that no parser can read — the model looks like it cannot call tools when it is calling them perfectly. Pass SamplingParams(skip_special_tokens=False). The server path with --tool-call-parser minicpm5 handles this for you.

Measured speed

Single stream, one RTX PRO 2000 Blackwell (16 GB), greedy, 768 output tokens. Median of 3 runs; --max-model-len 32768, --max-num-seqs 8, prefix caching on.

Run-to-run spread on the speculative numbers is real: an earlier independent measurement of code generation gave 378 tok/s against this table's 366. Treat these as a band, not a constant.

Speculation pays where the next token is structurally determined — code and math — and barely moves prose. That is the expected shape for block drafting, not a defect.

Measured quality

Own harness, greedy, zero-shot chat format: GSM8K (first 250 test problems, numeric match) and HumanEval (all 164, executed). These absolute numbers are not comparable to published scores — the prompting and extraction are mine and cost the base model several points. The comparison between rows is the point.

Why this checkpoint is not the mixed-precision one. Leaving the last 8 decoder layers in BF16 improves perplexity clearly (mean +2.31% against +3.34%), and GPTQ's own reconstruction error grows monotonically with depth, so those layers really are where the damage is. But it did not improve either downstream score — both differences are under one standard error, so nothing was demonstrated — while costing 21% of decode speed (108 against 137 tok/s) and 25% more disk. Perplexity alone was not enough to justify that, so the plain, fully-quantized NVFP4 build ships.

Also verified on this checkpoint: Japanese generation, tool-call XML emission per the model's own template, and needle retrieval at 11.4k and 24.4k tokens of context (both found).

Tool calling

22 situations, each posed in English and Japanese, through the model's own chat template with tools=: straightforward calls, multi-parameter extraction, questions that need no tool, requests missing a required argument, two-step tasks, and continuing after a tool result.

Tool selection and argument extraction are excellent and do not depend on the language. Every malformed-XML check came back clean: 0 malformed calls across all runs.

The weak spots, in both the quantized checkpoint and the BF16 original:

• Reaching for a tool when it should just answer, and only in English. Asked the capital of France or why the sky is blue, the BF16 original calls search_web 4 times out of 5; this checkpoint does it 1-2 times out of 5. In Japanese both answer directly, 5/5.
• Inventing a missing required argument. "Email the team about the delay" makes both models invent [email protected] rather than asking who the team is.
• One call per turn. Given a two-step task both emit only the first call. That is normal agent behaviour, not a defect, but do not expect two calls in one response.

temp 0.3 is the best operating point — it beats greedy on both languages and was the only setting where nothing ran past the token budget.

Verbosity and stopping

Greedy, 250 MGSM problems per language, strict scoring (an answer counts only if the model closed its </think> block). Two seeds each; the pair shows run-to-run spread.

Quantization swapped which language the model cannot stop thinking in. Both independently baked quantized checkpoints moved the same way, across both seeds, so this is a property of the quantization and not casting variance.

Raising the budget rescues Japanese but not English:

In Japanese the median answer takes 682 tokens with a long tail — verbose, but it finishes. In English the median output is the cap at both 2048 and 6144: it does not finish. Accuracy among answers that do terminate stays high throughout (English 98%, Japanese 80-86%).

The GSM8K and HumanEval numbers in the previous section were scored with a lenient parser that accepts an answer found anywhere in the output, including inside an unterminated think block. That is why they look healthy while the table above does not. Both are true; they measure different things. If you need the model to stop on its own, the table above is the one that matters.

Japanese

The base model declares only English and Chinese, but its Japanese is real, and lopsided: it reads Japanese much better than it writes it.

Note that quantization costs less perplexity in Japanese (+3.02%) than in English (+3.66%), and that this checkpoint answers 99% of the Japanese commonsense questions against the original's 67%.

What it does well: reading, summarizing, and classifying Japanese. Simplified-Chinese characters almost never leak into its Japanese output (5 stray characters in 1785 kanji, 0.3%, and 2 of those 5 are rare-but-valid Japanese).

What it does badly, at every temperature and in the BF16 original too:

• Honorifics are wrong. Asked for the honorific and humble forms of 見る it answers 見上げる and 見下す ("look up at" / "look down on") instead of ご覧になる and 拝見する. This is a knowledge gap, not a sampling artifact.
• Other languages leak into Japanese writing. A Japanese business email comes back with the Chinese placeholder 您的名字 in it; at the model's default temperature of 1.0, English and even Korean fragments appear mid-sentence.
• Register instructions are ignored. Asked for a casual message to a friend, it writes polite です/ます forms.

For Japanese, use a low temperature (0.3-0.6), not the model's default 1.0, and give it a generous token budget. Do not rely on it for honorific or business Japanese.

Calibration

512 sequences at 2048 tokens, drawn from a deliberate mix so no single domain sets the ranges:

The included recipe.yaml is the exact llm-compressor recipe used.

Provenance and honesty notes

• Every number above is measured on this hardware, not copied from a paper or estimated.
• Perplexity is computed through vLLM itself (prompt_logprobs), so it reflects the same kernels that serve the model, not a separate reference implementation.
• Quantization is not a deterministic transform. Numbers here describe this checkpoint.
• The DSpark draft weights are not redistributed here; point vLLM at the official repo.
• The English stopping regression in Verbosity and stopping was found after this checkpoint was first published, by measuring it rather than assuming it. It is documented rather than hidden; a re-bake with a different calibration mix has not been attempted yet.

License

Apache-2.0, inherited from the base model. The base model, the DSpark drafter, and the training data are the work of OpenBMB.

---

Baked by ケンと一緒に研究している Lna-Lab のユキ（Fable 5.1）.