llm-jp-4-33b-thinking-NVFP4

llm-jp-4-33b-thinking NVFP4 (W4A16)

Weight-only NVFP4 quantization of llm-jp/llm-jp-4-33b-thinking (revision 9e6928628594b205bd3ccd93c2813064aa53fe97) for vLLM on NVIDIA Blackwell. The checkpoint is 21.6 GB, 0.32× the BF16 size.

On llm-jp-eval it scores 0.673 against 0.661 for BF16 on the same harness and prompts (30 samples per dataset). On a DGX Spark, NVFP4 plus the companion DSpark drafter decodes 5.2× faster than BF16 (19.1 vs 3.6 tok/s at 128 output tokens). On an RTX PRO 6000 the same pair decodes at 103.7 tok/s (NVFP4 alone 62.8, BF16 21.4).

Quickstart

pip install "vllm==0.28.0"
hf download kel-dx/llm-jp-4-33b-thinking-NVFP4 --include "vllm_plugin/*" --local-dir nvfp4
pip install ./nvfp4/vllm_plugin          # the llmjp4 reasoning parser (source in this repo)
vllm serve kel-dx/llm-jp-4-33b-thinking-NVFP4 \
  --trust-remote-code \
  --reasoning-parser llmjp4 \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.6

With the DSpark drafter (num_speculative_tokens must be 8):

vllm serve kel-dx/llm-jp-4-33b-thinking-NVFP4 \
  --trust-remote-code --reasoning-parser llmjp4 --kv-cache-dtype fp8 \
  --max-model-len 16384 --gpu-memory-utilization 0.6 \
  --speculative-config '{"method":"dspark","model":"kel-dx/llm-jp-4-33b-thinking-NVFP4-speculator.dspark","num_speculative_tokens":8}'

These are the settings of the quality run below on a DGX Spark. The throughput run used --gpu-memory-utilization 0.35, which holds the 21.6 GB checkpoint, the 4.7 GB drafter and single-stream KV at 16k context. On the GB10's 119 GB unified memory, 0.85 starved the host and the server was killed by the out-of-memory daemon during KV allocation, so leave host headroom. Raise --max-model-len as memory allows.

Any OpenAI-compatible client works against /v1/chat/completions. The llmjp4 parser returns the Harmony analysis channel as reasoning and the final channel as content, in streaming mode as well.

Tokenizer

Use the tokenizer files in this repository. The upstream llmjp4_tokenizer.py subclasses LlamaTokenizerFast, and under transformers 5 (bundled in vLLM 0.25 and later) that class rebuilds the Unigram tokenizer as merge-less BPE and emits one token per character. The class shipped here is the one the LLM-jp 8B repositories already use; tokenizer.json is unchanged from upstream.

Quantization

• Tool: NVIDIA Model Optimizer nvidia-modelopt==0.45.0, W4A16_NVFP4_CFG, modelopt export format, served by vLLM's native NVFP4 path.
• Layout: every Linear in the 64 decoder layers (q/k/v/o, gate/up/down) in NVFP4 W4A16: FP4 weights, group 16, FP8 E4M3 block scales, FP32 per-tensor scale. lm_head, embeddings and norms stay BF16.
• No activation quantization. At batch 1 the matmuls are memory-bound GEMVs, so quantizing activations adds work without a bandwidth gain.
• Calibration: weight-only max.
• KV cache: FP8 E4M3 at serve time (--kv-cache-dtype fp8). The checkpoint carries no calibrated KV scales. NVFP4 KV is not used because vLLM's NVFP4 KV kernel corrupts decode on SM120 and SM121 (vllm#50084).

Quality (llm-jp-eval v2.1.5)

Setup: 30 samples per dataset, 58 datasets plus xlsum_ja, temperature 0, dataset-default few-shot (4 or 0), max_tokens = output length + 4096 for reasoning, no drafter, vLLM 0.28.0, same prompts for both. BF16 ran on an RTX PRO 6000 with bf16 KV, NVFP4 on a DGX Spark with fp8 KV.

Notes:

• With 30 samples per dataset and one run each, a point or two of AVG is noise. Some datasets go either way (mmlu_en 28/30 → 24/30).
• Generations that ran out of the 4096 reasoning tokens count as wrong: 55/1770 (BF16) and 56/1770 (NVFP4), mostly polymath and gpqa.
• SUM is xlsum_ja ROUGE-2 on the whole answer. The harness default cuts at the first blank line, which turned most summaries into their heading (SUM 0.033 / 0.039, AVG 0.658 / 0.670 before the fix).
• CG: jhumaneval 28/30 (BF16) vs 30/30 (NVFP4), mbpp 30/30 for both. The prompts include the test cases.
• Per-dataset scores and sample generations: benchmark/phase3-llm-jp-eval-{nvfp4,bf16}-r4096-vllm0.28.0-sumfix.json, recovery table benchmark/phase3-recovery-bf16-vs-nvfp4-sumfix.json. Files without the -sumfix suffix are the original scoring.

Throughput

ELYZA-tasks-100 prompts, temperature 0, one request at a time, exactly N generated tokens per request (ignore_eos), decode tok/s excluding time-to-first-token, mean over 100 prompts.

DGX Spark (GB10, SM121, vLLM 0.28.0, fp8 KV)

The BF16 row is from an earlier run on vLLM 0.27.1 with a pre-fix tokenizer (longer prompts); it was not repeated on 0.28.0.

RTX PRO 6000 Blackwell (SM120, 96 GB, vLLM 0.28.0, fp8 KV)

The NVFP4 legs ran at --gpu-memory-utilization 0.9, the BF16 leg at 0.85. Per-request records for every row are in benchmark/.

Limitations

• The fp8 KV cache runs with scale 1.0 because the checkpoint has no calibrated KV scales. The quality numbers above were measured in this configuration.

Versions

vLLM 0.28.0 (serving), nvidia-modelopt 0.45.0, transformers 5.5.4 (quantization) and 5.15.1 (serving), base model revision 9e69286.

License and attribution

Apache-2.0, following the base model. Base model by the Research and Development Center for Large Language Models, National Institute of Informatics (LLM-jp). Quantization, drafter and measurements by kel-dx.