ThaiLLM-30B-NVFP4

<p align="center"> <b>AGIcafet Research</b> · <a href="https://thaillm.agicafet.com">interactive report</a> · <a href="https://github.com/spped2000/thaillm-nvfp4-dgx-spark">full pipeline &amp; results</a> </p>

ThaiLLM-30B-NVFP4

NVFP4 (4-bit) quantization of ThaiLLM/ThaiLLM-30B — the Thai/English continued-pretrain of Qwen3-30B-A3B (MoE, 31B total / ~3.3B active) — produced with NVIDIA TensorRT Model Optimizer 0.43.0 and validated with a rigorous paired BF16-vs-NVFP4 evaluation on an NVIDIA DGX Spark (GB10).

• 18.1 GB (from 61.1 GB BF16 — 3.4× smaller); 16.9 GiB GPU-resident
• 2.3–2.5× faster decode on DGX Spark (27 → 63 tok/s single-stream, 69 → 175 tok/s at 4 streams), TTFT 2.0–2.7× lower
• Thai capability statistically unchanged (paired McNemar tests over identical questions)
• W4A4 NVFP4 (E2M1 + FP8 block scales, group 16): attention and all 128 experts/layer quantized; router gates + lm_head kept BF16; KV cache not quantized
• Calibration: 256 Thai Wikipedia + 256 English news documents (512×512 tokens) — half-Thai calibration per multilingual-quantization best practice
• ⚠️ Like the base model, this is a base (non-instruct) model — fine-tune before use as an assistant

Serving

# vLLM (Blackwell GPU, e.g. DGX Spark with nvcr.io/nvidia/vllm:26.05.post1-py3)
vllm serve AGIcafet/ThaiLLM-30B-NVFP4 \
  --quantization modelopt \
  --gpu-memory-utilization 0.70 \
  --attention-backend flashinfer
# vLLM auto-selects native FLASHINFER_CUTLASS NVFP4 MoE kernels on SM121/SM100.
# For extra KV headroom in production add: --kv-cache-dtype fp8

Benchmarks (paired BF16 vs NVFP4, identical questions, seed 0)

Interpretation. Pooled over 19,786 paired questions (ThaiExam counted once, via the letter-based template) the quantization cost is real but under one accuracy point, and it concentrates on English reasoning tails — no Thai task degrades significantly. A Thai-fluent 12-domain qualitative review found no systematic degradation (orthography intact, no new repetition behavior); the one caution is verbatim quotation (e.g., legal statutes), where 4-bit weights can paraphrase — use retrieval grounding or BF16 for citation-critical work.

DGX Spark performance (vllm bench serve, median of 3, no speculative decoding)

Thai context vs other local models (same protocol)

How it was made

# TensorRT Model Optimizer 0.43.0, examples/llm_ptq/hf_ptq.py
python hf_ptq.py --pyt_ckpt_path ThaiLLM/ThaiLLM-30B \
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 0 \
  --dataset thai_en_calib.jsonl \
  --export_path ThaiLLM-30B-NVFP4

Full reproducible pipeline (calibration builder, fairness-protocol eval scripts, custom letter-scored ThaiExam lm-eval tasks, paired statistics, complete technical report): github.com/spped2000/thaillm-nvfp4-dgx-spark

Roadmap — planned next phases

1. Draft-model speculative decoding: this base has no MTP head, but a same-tokenizer drafter (e.g. Qwen3-0.6B) is expected to add ~1.3–1.6× decode speed losslessly (63 → ~80–100 tok/s on DGX Spark).
2. Thai instruction SFT on the BF16 base → re-quantize with the same recipe → re-run the eval gate.
3. EAGLE-3 head training post-SFT (~1.5–2× additional).
4. FP8 KV cache in production (--kv-cache-dtype fp8) — disabled in this study only for measurement isolation.

Limitations

• Base model: requires instruction fine-tuning before assistant use (the recommended production flow is SFT the BF16 base → re-quantize with this recipe → re-run the eval gate).
• Evaluated with automatic metrics + a 12-domain expert review; no large-scale human evaluation.
• Serving requires NVFP4-capable hardware (NVIDIA Blackwell) for the measured speedups.

Attribution

Base model by the ThaiLLM project (Apache-2.0). Quantization, evaluation, and release by AGIcafet, 15 July 2026.