kvtc-pca-gemma4-26b

KVTC PCA Calibration for Gemma 4 26B MoE

Pre-computed PCA calibration data for KVTC (KV cache compression) on Gemma 4 26B MoE, calibrated on post-RoPE hidden states.

What This Does

Reduces KV cache head dimensions from 512 → 160 on the 5 full-attention layers (layers 5, 11, 17, 23, 29) of Gemma 4 26B. This gives 3.2x compression on the full-attention KV cache pool, enabling:

• 3M+ tokens in KV cache on 32GB VRAM (11.8× a single 256K conversation)
• ~160 tok/s single user on RTX 5090
• ~1,128 tok/s aggregate at 16 concurrent users

Quality

dim=160 is the quality floor. Do not go lower.

Usage with SGLang

Download the calibration file and pass it to SGLang:

# Download
huggingface-cli download roo5150/kvtc-pca-gemma4-26b kvtc_calibration_postrope.pt --local-dir .

# Launch SGLang with KVTC PCA
python3 -m sglang.launch_server \
  --model-path <your-gemma4-26b-model> \
  --kvtc-calibration-path kvtc_calibration_postrope.pt \
  --kvtc-reduced-dim 160 \
  --kv-cache-dtype fp8_e5m2 \
  --attention-backend triton \
  --mem-fraction-static 0.83 \
  --port 30000 --host 0.0.0.0

Important Notes

• CUDA graphs require mem=0.83 (0.84 OOMs, 0.85 leaves no room for graph capture on 32GB)
• Must use --attention-backend triton on SM120 (RTX 5090) — FlashInfer doesn't support variable head_dim
• Calibrated on post-RoPE data so positional encoding is preserved in dot products
• Works with both AutoRound INT4 and NVFP4 quantized Gemma 4 26B checkpoints

Hardware Tested

• NVIDIA RTX 5090 32GB (SM120 / Blackwell)
• CUDA 13.0, SGLang v0.5.x (April 2026)

Related

• SGLang SM120 Fixes — 7 bug fixes for NVFP4/SM120
• SGLang PR #22927 — NaN + 3D tensor fixes
• SGLang PR #22928 — MoE GEGLU + block scale
• SGLang PR #22929 — NVFP4 weight loading

Author

Chris (@roo5150)

License

MIT