Qwen3.8-27B-NVFP4

Qwen3.8-27B NVFP4 — Deep Quality Analysis vs BF16

TL;DR

• Overall accuracy: BF16 84.89% (410/483) vs NVFP4 82.19% (397/483) — Δ = -2.69 pp
• Win/Tie/Loss: BF16-only wins = 22, NVFP4-only wins = 9, both pass = 388, both fail = 64
• Statistical significance: Δ = -2.69 pp is NOT statistically significant — the 95% Wilson CIs for BF16 [81.5%, 87.8%] and NVFP4 [78.7%, 85.2%] overlap by 3.93 pp
• Latency: NVFP4 averages 2.23 s vs BF16 3.22 s — a 1.44× speedup (medians 1.43 s vs 1.83 s)
• Where NVFP4 is weaker: code (HumanEval -2.5pp, MBPP -3.8pp), simple math reasoning (GSM8K -6.7pp due to overthinking), IFEval (-2.5pp), MMLU no-think (-6.7pp)
• Where NVFP4 matches BF16: MATH (-3.3pp, but with NO reasoning loops), ARC, MMLU thinking, all curated categories (knowledge, translation, creative, safety, multilingual, code_explain, logical)
• MATH loops fixed: zero tasks with finish_reason=length on MATH in this run (BF16 and NVFP4 both solve within a reasonable token budget). The only length-overflow in the run happened on mmlu_thinking[6] — a trivial MCQ where NVFP4 got stuck in a vacuous-truth reasoning tangent.

Methodology

Inference engines

The NVFP4 quantized checkpoint is inference-engine-agnostic and was verified to run correctly on both:

• vLLM and SGLang — supported via the NVFP4 kernels

For this benchmark both backends behave equivalently; results reported here were collected against the sglang endpoints.

Generation parameters

• temperature = 0 (greedy / deterministic)
• seed = 42 (fixed for reproducibility)
• top_p = 1.0
• chat_template_kwargs.enable_thinking — toggles Qwen3 thinking mode

Categories and thinking mode

Datasets

Standard datasets are loaded via datasets-server.huggingface.co. Fixed seed numpy.default_rng(42).

Curated collection — 63 hand-crafted prompts in 8 categories (logical, knowledge, instr-follow 21, creative, translation, code_explain 20, multi-reason, safety).

Evaluators

• MMLU/ARC: extract letter (A-E), compare with gold
• GSM8K: extract number from Answer: N or last number
• MATH: extract \boxed{...}, normalize LaTeX + numeric comparison
• HumanEval/MBPP: sandboxed exec with official tests (timeout 20s), auto-aliasing of function name
• IFEval: 25 rule verifiers (detectable_format:*, length_constraints:*, punctuation:*, keywords:*, combination:*, startend:*, change_case:*, language:*)
• Curated: 16 evaluators (qualitative, format_json_schema, bullets_sorted_alpha, code_python_runs, etc.)

Total instances: 966 = 483 pairs × 2 models.

Per-category results

Overall: BF16 84.89% (410/483) · NVFP4 82.19% (397/483) · Δ = -2.69 pp

Latency (response time, seconds)

NVFP4 gives a 1.44× overall speedup (medians 1.43 s vs 1.83 s). Largest gains on creative/knowledge/translation/safety (-40% to -53%), code and IFEval (-39% to -44%). On simple MCQ without thinking the latency is at parity — on 1-2 token answers the per-request overhead is not amortized.

Token usage (reasoning + completion)

Pattern: NVFP4 on average spends +10-20% more reasoning tokens on thinking tasks (MATH, GSM8K, ARC-thinking, MMLU-thinking, logical), but fewer tokens on short no-think tasks (HumanEval -19%, IFEval -8%, creative -9%). This is consistent with the "overthinking" hypothesis: on simple tasks NVFP4 produces longer reasoning chains that don't always improve the answer.

Win/Tie/Loss — detailed breakdown

Totals: BF16-wins = 22 · NVFP4-wins = 9 · both-pass = 388 · both-fail = 64 Net NVFP4-worse: +13 divergences (22 - 9)

Divergence cluster (BF16 ✓, NVFP4 ✗) — 22 cases

NVFP4 wins (NVFP4 ✓, BF16 ✗) — 9 cases

IFEval (3): ifeval[8], ifeval[23], ifeval[59] — within the noise on 80 samples. MBPP (3): mbpp[1], mbpp[52], mbpp[56] — code generation, noise. MMLU (1): mmlu[12] — short MCQ, noise. MMLU thinking (1): mmlu_thinking[13] — both models generated exactly 4096 tokens (length-overflow), but NVFP4 happened to land on the correct answer in the last tokens. Curated instr-follow (1): instruction_following[21] — format constraint, noise.

All 9 NVFP4 wins are within statistical noise. Of the 22 BF16 wins the meaningful signal concentrates on MBPP (6) and IFEval (5) — the categories with the most samples.

Key cases — the "overthinking" pattern on simple tasks

gsm8k[8] — "two times older"

Prompt: Brandon's iPhone is four times as old as Ben's iPhone. Ben's iPhone is two times older than Suzy's iPhone. If Suzy's iPhone is 1 year old, how old is Brandon's iPhone?

• BF16: brief reasoning — interprets "two times older" as "twice as old" (colloquial), Ben=2, Brandon=8. Goes straight to the answer.
• NVFP4: 2604 reasoning tokens spent disambiguating the phrase "older" (strictly = +200%, colloquially = ×2). Eventually picks the strict interpretation: Ben=1+2×1=3, Brandon=12. The answer is mathematically defensible but does not match the gold.

Pattern: NVFP4 overthinks a simple word problem and picks a more formal interpretation that doesn't match the expected one.

gsm8k[11] — "2/5 times more"

Same pattern: BF16 490 reasoning tokens, NVFP4 2771 reasoning tokens. NVFP4 reinterprets "2/5 times more" as "2/5 as many" instead of "5 + 2/5 × 5" and gets 44 instead of 104.

mmlu_thinking[6] — length-overflow on a trivial question

Question: about dim V < dim W and injectivity of a linear map (correct answer D — "if V and W are finite-dimensional and dim V < dim W, there is no linear injective map W → V").

• BF16: brief reasoning, immediately answers D.
• NVFP4: gets stuck in a tangent about vacuous truth, edge cases with dim V = 0, infinite-dimensional spaces. Hits 4096 tokens without producing an answer.

This is the only length-overflow in the run. On MATH there were 0 length-overflows. The v2 quantization eliminates competition-math loops, but the residual risk on simple MCQs with thinking mode (where the model digs too deep into a trivial question) remains.

math[22] — an evaluator false negative (not a model regression)

Both models produce the semantically correct answer 120°. NVFP4 added ^\circ inside the \boxed{} and the simple LaTeX-normalizer in the evaluator failed to strip it, extracting 120^circ instead of 120. This is an evaluator bug, not a model regression. The real MATH degradation is effectively 0 (NVFP4 actually solves all 30/30 just like BF16 semantically).

Conclusions

1. Overall accuracy: the 2.69 pp gap in favor of BF16 is NOT statistically significant (95% Wilson CIs overlap by 3.93 pp). On 483 pairs this is within the noise.

2. MATH loops eliminated: 0 length-overflows on MATH. The v2 quantization resolves the competition-math looping issue. The only length-overflow in the run happened on mmlu_thinking[6] (a trivial MCQ where NVFP4 got stuck in a vacuous-truth tangent).

3. Code generation: a stable -2.5pp on HumanEval and -3.8pp on MBPP. The degradation pattern: NVFP4 is weaker on fine-grained edge cases and ambiguous prompt interpretation. On curated code (20 prompts) there are 0 degradations, both models at 90.0%.

4. Simple math reasoning (GSM8K): -6.7pp on 30 samples, but the cause is overthinking, not a loss of capability. NVFP4 spends ~20% more reasoning tokens and sometimes picks a more formal interpretation that doesn't match the gold. On MATH (where reasoning is genuinely needed) the gap is -3.3pp and entirely within noise.

5. IFEval: -2.5pp on 80 samples (5 BF16-wins vs 3 NVFP4-wins). Degradations cluster around detectable_content:postscript (NVFP4 skips the P.S. marker) and keywords:forbidden_words (NVFP4 includes the forbidden word).

6. Latency: NVFP4 is faster than BF16 by 1.44× overall (median 1.43s vs 1.83s). Largest speedups on creative/knowledge/translation/safety (-40% to -53%), code and IFEval (-39% to -44%). On simple MCQs without thinking the latency is at parity — on 1-2 token responses the per-request overhead is not amortized.

7. Simple tasks (knowledge, translation, multilingual, creative, safety, code_explain, logical, multi-reason): 0 degradations, identical quality. Quantization does not damage the "fast" modes of the model.