Kimi-K3-DSpark-FP8

Kimi-K3-DSpark — FP8 (per-channel, weight-only)

An 8-bit FP8 quantization of Inferact/Kimi-K3-DSpark, the MLA-native DSpark draft model for Kimi-K3, produced with qstream.<br> The original model card follows in full below.<br> Size | 4.6 GiB (down from 6.6 GiB BF16, ~69%) |

Per-position agreement

Drift-limited acceptance ceiling

Composing the per-position sequential-rollout agreement gives an upper bound on the acceptance length that can survive numerics. This is the number that decides the format:

FP8 lands 0.4–1.1 below the BF16 floor depending on regime. Both 4-bit formats lose roughly half the block — about 3 tokens — and are disqualified by a wide margin, consistently across all three regimes.

Why FP8

A speculative draft model is not an ordinary quantization target. Acceptance length is a prefix-run-length statistic — a drafted block is accepted only up to its first mismatch — so per-token error compounds multiplicatively across all 7 positions. Error that would be invisible in a standalone LM turns into lost acceptance, and acceptance is the entire reason the draft model exists.

MXFP4 and NVFP4 builds were produced and measured alongside this one. Both land at 19–20 dB weight SNR, which is perfectly normal for 4-bit — and still not good enough here. FP8 costs 1.0 GiB more than MXFP4, on a model served beside a ~1 T-param target.

What is quantized to what

2.19 B of 3.56 B params quantized (61%). Everything else is bit-identical BF16.

How this was evaluated

Acceptance length cannot be measured without the ~1 T-param Kimi-K3 target in the loop, and the drafter is not a standalone LM — it consumes target hidden states from K3 layers (2, 23, 47, 71, 89). So we measure drift against a high-precision reference: how far a variant's drafted block moves from the reference's, on identical inputs. Every token where they agree is a token whose accept/reject outcome is unchanged by definition.

Three choices make this honest rather than flattering:

• The reference is FP32 weights + FP32 compute, and plain BF16 is measured as a variant. BF16 is what actually ships, so its drift is the numerical noise floor. Quoting a quantized format against BF16 (rather than against exact arithmetic) silently credits it with BF16's own error — an earlier BF16-referenced run of this same checkpoint overstated FP8 by about +0.5 tokens of ceiling.
• Only the unobservable input is shared. Each variant runs its own context_proj and its own context-KV projections, so quantizing context_proj is measured, not excluded. Only the 5 aux hidden states — which genuinely cannot be obtained — are held common.
• The Markov head is rolled out sequentially, feeding each sampled token into markov_w1 to bias the next position, as the real speculator does. Holding that bias fixed hides error compounding through tokens and understates drift.

Real inputs: GSM8K token streams (train+test) tokenized with the actual Kimi-K3 tokenizer, the actual target lm_head, and the real drafter embedding/Markov/norm weights. The one input that cannot be obtained — the aux hidden states — is swept over three distributions so the result can be checked for sensitivity to it: gauss (iid normal), heavy (lognormal per-channel scales, mimicking massive-activation structure), and boot (the drafter's own 5 layer outputs run causally over the same GSM8K tokens — real LLM hidden states with real token correlation).

1024 blocks per regime, 7168 drafted tokens, 95% CI ±0.7–0.9 pp.

Full metric set (heavy regime — the pessimistic one)

• Greedy agree — top-1 match, teacher-forced on the reference prefix (isolates per-position drift). Seq rollout — free-running, so error compounds through tokens.
• 1−TV = Σ min(p, q), the coupling probability. This is the sampling-aware analogue of top-1 agreement and is the relevant one under draft_sample_method: probabilistic.
• KL p99 matters more than the mean here: a drafter that is usually right but occasionally confidently wrong is exactly what costs acceptance. FP8's p99 (0.013) stays within an order of magnitude of its mean; both 4-bit formats blow out past 0.28.
• Reverse KL ≈ forward KL for every variant, so no variant is hallucinating confidence in a direction the forward KL would hide.
• Confidence r / sched flip — the confidence_head drives resource-aware draft-length scheduling. Flip rate is measured against the per-block median confidence, which is a deliberately knife-edge threshold (half the samples sit near it by construction), so absolute values are pessimistic and only the comparison is meaningful. FP8 roughly matches BF16; NVFP4 flips nearly half of all scheduling decisions.

FP8 tracks the BF16 curve with a roughly constant offset. The 4-bit curves decay steeply — the compounding that the prefix-run-length structure punishes.

Harness: eval_dspark_quant.py (standalone BF16/FP32 reference forward — MLA + SwiGLU + context-KV cross-attention + sequential Markov head — reimplemented from vLLM's DSpark backbone).

Limitations

Stated plainly, because this is an unusual evaluation:

• Acceptance length itself is not measured. Everything above is drift against a high-precision reference, not agreement with Kimi-K3. Drift bounds acceptance impact; it does not equal it. In particular the ceiling is conservative — a divergence occurring past the point where the target would have rejected anyway costs nothing in practice.
• The aux hidden states are synthetic. Swept over three distributions, and the ranking is stable across all three, but none of them is Kimi-K3's actual hidden-state distribution.
• Long-context behaviour is untested. A position sweep through the YaRN regime (original_max_position_embeddings 32768, factor 32) was planned and not completed. The base card reports acceptance holding at ~95 k tokens; whether FP8 holds equally there is unverified.
• Not load-tested in vLLM. The build used here has no K3DSparkModel in its registry. The config.json is verified structurally — single config group, 41 ignored modules, no target/ignore overlap, fused-module constraint respected — but has not been through a real loader.

Serving with vLLM

Requires a vLLM build that registers K3DSparkModel (the public vllm/vllm-openai:kimi-k3 image). Serve Kimi-K3 per the official recipe and point the speculative config at this repo:

--speculative-config '{"method": "dspark", "model": "kodelow/Kimi-K3-DSpark-FP8",
                       "num_speculative_tokens": 7, "attention_backend": "FLASHINFER_MLA",
                       "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}'

FP8 weight-only needs no calibration and no activation scales.

How it was made

qstream-quantize \
  --model_dir <Inferact/Kimi-K3-DSpark> \
  --output_dir ./Kimi-K3-DSpark-FP8 \
  --quant_format fp8 \
  --include_layers "mlp.gate_proj" "mlp.up_proj" "mlp.down_proj" \
                   "self_attn.o_proj" "self_attn.q_b_proj" "context_proj" \
  --device cuda --workers 1

Built with qstream @fcfe8a2. SmoothQuant is deliberately off: this FP8 path is weight-only per-output-channel, which is already invariant to per-column rescaling, so smoothing can only redistribute error, never remove it.

License

Inherits the Kimi-K3 license from the base model. This is a derivative (quantized) work of Inferact/Kimi-K3-DSpark.

---

Original model card

Model Overview

Inferact/Kimi-K3-DSpark is an MLA-native DSpark draft model that accelerates Kimi-K3 on vLLM, served through vLLM's native dspark speculative method.

The draft trains on target hidden states extracted from vLLM itself — the same engine that serves it, so the numerics it learns from are the numerics it meets at inference. TorchSpec provides the loop, streaming those hidden states from live target inference into concurrent FSDP draft training.

DSpark = a block-diffusion backbone of 5 dense layers with non-causal attention, drafting 7 tokens in a single parallel pass, a low-rank sequential Markov head supplying the intra-block dependency, and a confidence head for resource-aware scheduling. Mirroring Kimi-K3's own MLA attention means draft and target share one KV layout (a compact 576-element latent per token), so the draft's pages unify with the target's cache — KV offloading and P/D disaggregation work with no separate page format.

---

Performance

Peak bs=1 decode: 464 tok/s

Under low-entropy real reasoning workload — Kimi-K3 + DSpark on vLLM can achieve 464 tok/s using the public vllm/vllm-openai:kimi-k3 image on 4 × GB300 at bs=1 & tensor-parallel-size=16.

Speculator acceptance

Speculators do best on predictable, low-entropy work like the reasoning workload behind the 464 tok/s above, and worst on open-ended, high-entropy generation. So we measured 14 benchmarks spanning math, code synthesis, real-world software engineering, multi-turn chat, RAG and QA, multilingual text, creative writing and long-context generation — all with the Kimi-K3 chat template enabled and production sampling parameters, with temperature=0 reported alongside for reproducibility. Acceptance stays strong at long context too, verified on AA-LCR's ~95k-token multi-document prompts. That is the most comprehensive view we can give of how the speculator performs on real-world workloads.

Acceptance length, with 7 speculative tokens:

Benchmarks: GSM8K, MATH-500, AIME 2026, HumanEval, MBPP, SWE-bench Pro, MT-Bench, AA-LCR — 100 multi-document prompts of 71k–115k tokens — and six splits of NVIDIA SPEED-Bench: its throughput_16k low-entropy split at 10k-token input, plus five qualitative categories.

---

Training

Data — all responses regenerated on-policy by Kimi-K3 itself, so the draft learns the target's own reasoning traces and chat formatting. Prompts come from public datasets:

• lightseekorg/kimi-mtp-dataset — general instruction prompts
• nvidia/OpenCodeInstruct — single-turn coding
• A nine-category prompt pool from the NVIDIA Nemotron SFT/RL collections and CohereLabs/aya_dataset, spanning chat, code, multilingual, RAG/QA, math, structured output and safety. Evaluation-set prompts are excluded.

Method: DSpark with block_size=7, trained on a combined CE + L1 distribution-distillation objective against the target's post-final-norm hidden state, bf16. The draft consumes Kimi-K3 auxiliary hidden states from target layers (2, 23, 47, 71, 89) of 93. Roughly two epochs in total, on GB300 nodes.

Draft architecture, block size, sequence length, and loss weights are YAML-configurable — see the TorchSpec repo.

---

Quick Start

Requirements

For serving Kimi-K3 itself — hardware, parallelism and engine flags — follow the official vLLM recipe: recipes.vllm.ai/moonshotai/Kimi-K3.

Enable the draft

Add to your Kimi-K3 vllm serve command:

--speculative-config '{"method": "dspark", "model": "Inferact/Kimi-K3-DSpark", "num_speculative_tokens": 7, "attention_backend": "FLASHINFER_MLA", "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}'

Sampling options

vLLM offers two knobs on top of the defaults, both used in the temperature=1.0 column above:

• draft_sample_method — probabilistic samples the draft from its own distribution instead of taking its argmax. Pair it with a sampling client; use greedy when serving at temperature=0 so the draft matches the client.
• rejection_sample_method — block verifies the drafted block as a unit rather than token by token. It is a no-op under greedy decoding (at temperature=0 verification reduces to a deterministic argmax match), so it only applies to the sampling configuration.

The temperature=0 column was produced with {"draft_sample_method": "greedy"} and no rejection_sample_method.