Mellum2-12B-A2.5B-Thinking-FP8

We compared BF16 and FP8 weights of Mellum2-12B-A2.5B-Thinking across four evaluation suites covering 134 prompts and 80 functional tests. Across logit-level, distribution-level, perplexity, and generative metrics, the FP8 variant matches the BF16 baseline within statistical noise:

Conclusion. FP8 weight quantization of this checkpoint is a near-lossless compression. The model can be served in FP8 without measurable quality regression for typical code-completion workloads.

---

1. Background

Modern LLM serving increasingly relies on lower-precision inference (FP8, INT8, INT4) to reduce memory footprint and improve throughput on Hopper / Blackwell GPUs. The risk is that aggressive quantization can shift the output distribution, biasing generation or degrading accuracy on structured tasks.

This report quantifies that risk for Mellum2-12B-A2.5B-Thinking, a 12B code-completion model with an extended thinking mode. We measured three orthogonal classes of effect:

1. Probability-distribution shift — do the next-token logits change?
2. Sequence-likelihood shift — does perplexity on natural code change?
3. End-to-end behavior — do greedy completions diverge, and do functional tests still pass?

Two identical model copies are exposed through an OpenAI-compatible inference server: one in BF16 and one in FP8 weights. All requests below use temperature=0 so that any divergence is attributable to quantization, not sampling.

---

2. Methodology

2.1 Endpoint contract

Both checkpoints are served via the standard v1/completions interface. All requests used:

• temperature=0
• logprobs=20 (top-20 token log-probabilities)
• max_tokens adapted per test (see below)
• For greedy generation tests: explicit stop sequences to keep continuations on a single function body

2.2 Test suites

2.3 Metrics

Suite 1 — Distribution shift. For each prompt we extract the top-20 next-token distribution from each model:

• top1_agreement — whether both models place the same token in rank 1
• KL(P_BF16 ‖ P_FP8) — forward KL divergence on the union of top-20 tokens, padded with eps = 1e-10 for missing entries
• JSD(P_BF16, P_FP8) — symmetric Jensen-Shannon divergence
• mean |Δlogprob| — average absolute log-probability difference across the union of tokens

Suite 2 — Perplexity. Long Python snippets are sent with echo=true, logprobs=1. We average the non-null token log-probabilities and compute PPL = exp(-mean_logp).

Suite 3 — Pass@1. Each task provides a minimal prompt (signature + 1-line docstring) and a list of (call, expected) test cases. The model is asked to complete the function. We extract the first def block via Python's ast module (with textwrap.dedent fallback for malformed indentation) and execute the resulting code in a fresh namespace, running every test.

Suite 4 — Generation equivalence. Identical prompts are sent to both endpoints with temperature=0. Outputs are compared byte-for-byte and token-for-token (str.split() whitespace tokenizer).

2.4 Reproducibility

• Randomness: none — all decodes are greedy.
• Stopping criteria: explicit stop strings in Suite 3 and 4 (no top-level def, no class, no import, no print(, no assert ).
• Padding: when KL is computed, tokens absent from one model's top-20 are treated as having probability 1e-10.
• All artifacts (raw responses, extracted function sources, summary files) are persisted alongside the report.

---

3. Results

3.1 Distribution-level divergence

For reference, a JSD of 0.0058 corresponds to an effective per-token information loss of ~6 millinats — orders of magnitude below the entropy of typical next-token distributions (~1 nat). The 3 prompts where the top-1 token disagrees (4.5% of cases) involve tokens that were already very close in probability under the BF16 model.

3.2 Perplexity on long Python code

One document shows +0.83% degradation, two show essentially zero change, and one actually improves marginally. Averaged across 2,036 tokens of real code, FP8 is +0.14% in perplexity — well within typical run-to-run noise.

3.3 Pass@1 on coding tasks

15 short, self-contained coding problems with 80 unit tests total (greedy decoding, minimal prompt = signature + 1-line docstring):

Both models fail the same 2 unit tests on the same task (is_palindrome), and solve the remaining 14 tasks identically.

3.4 Greedy-generation comparison

Of the 6 non-identical generations:

• 4 differ only in whitespace (e.g. one extra space or newline). These are tokens that are indistinguishable in their effect on downstream parsing.
• 1 (gen_09) is a 44-char vs 29-char continuation — the FP8 model stopped one statement earlier, after the same token sequence on the prefix.
• 1 (gen_16) is a 103-char vs 159-char continuation with 57% token overlap — the only prompt where FP8 and BF16 take genuinely different code paths.

Manual inspection of gen_16 (an async context-manager completion) shows both continuations are syntactically valid Python; the BF16 path explores a more verbose error-handling branch, the FP8 path a leaner one. Functionally both are reasonable, and neither is clearly "wrong" without the surrounding program.

---

4. Discussion

What does FP8 preserve?

Across all four suites, FP8 quantization preserves:

• The argmax of the next-token distribution in 95.5% of cases — sufficient to keep greedy decoding byte-identical 81.8% of the time and token-identical 98.7% of the time.
• Mean negative log-likelihood within +0.14% on multi-hundred-token code documents.
• Functional correctness on standard code-completion tasks (97.5% pass@1, no FP8-only failures).

Where does divergence show up?

The only measurable signals are at the tail of the distribution:

• Tiny log-probability shifts (≈0.06 nats on the top-1 token).
• ≈4.5% of prompts where the top-1 token disagrees; in all observed cases these are tokens that BF16 itself assigned close probabilities to (within ~1 nat of the runner-up).
• One out of 33 prompts where the generation path visibly diverges after a few dozen tokens.

Operational implications

For serving the model in FP8:

• Quality — negligible degradation on standard code-completion workloads.
• Memory — ~2× weight-memory reduction vs BF16 (FP8 weights ≈ half the bytes of BF16 weights; KV cache still BF16 in this setup).
• Throughput — depends on GPU architecture and runtime, but Hopper / Blackwell gain meaningful speedups from native FP8 matmul.
• Risk surface — tasks with very low-entropy decoding paths (highly constrained completions, single-token outputs after a long deterministic prefix) are the most likely to surface sub-token-level differences; in our suite these did not affect functional outcomes.

---

5. Limitations

• Single language. All prompts are Python; the conclusions may differ for other languages, especially low-resource ones where token distributions are sharper.
• Single decoding mode. We tested greedy only. Sampling-based evaluation (e.g., temperature ≥ 0.3) would amplify any tail differences.
• Functional tests are small. 80 unit tests across 15 tasks is enough to detect large regressions but cannot rule out subtle bugs.
• No human evaluation. All "correctness" judgments are mechanical (string / list equality on deterministic inputs).
• One model, one quant recipe. Generalization to other checkpoints (different size, different quant scheme) is not addressed.