rumik-oss-1-nvfp4-vllm

rumik-oss-1-nvfp4-vllm

NVFP4 (4-bit floating point) quantization of rumik-ai/rumik-oss-1, Rumik's open source Indic text-to-speech model, built to run through vLLM's fused Blackwell NVFP4 kernels.

Quantized with nvidia-modelopt (full weight+activation NVFP4, calibrated on representative prompts), exported in vLLM's native checkpoint format. The stop_predictor head is kept at full precision (BF16) — it's a small (~1M parameter) auxiliary head, not worth the quantization risk.

Full writeup, architecture dissection, and benchmark methodology: https://github.com/Sj0605-DataSci/rumik-oss-model-dissection

Benchmarks (NVIDIA DGX Spark, GB10 Grace Blackwell)

Measured on the same 3 prompts, comparing the original BF16 checkpoint (standard HuggingFace transformers generation) against this NVFP4 checkpoint running through vLLM with a custom model plugin (source in the repo above):

~2.3x faster decode, ~2.2x smaller on disk. RTF is wall-clock generation time ÷ audio duration produced; lower is better, 1.0 is real-time.

Getting this speedup required more than quantizing the weights: quantizing alone (with plain PyTorch, no fused kernel) actually made inference ~15x slower, because there was no compiled kernel to run the 4-bit matmuls on — see the linked writeup for the full story of what did and didn't work.

Sample audio generated by this checkpoint (durations closely match the original BF16 model's output on the same prompts) is included in the GitHub repo linked above.

Concurrent request throughput

The single-request numbers above understate what this checkpoint is actually for — vLLM's design point is serving many requests at once via continuous batching. Submitting multiple requests together in one call:

Going from 3 to 48 concurrent requests (16x more load) only increases wall time ~1.5x while aggregate throughput scales ~10.8x — this is the real payoff of a bandwidth-bound workload on a fused low-precision kernel: one weight read serves many sequences' next tokens at once. See the linked writeup for full methodology.

Usage

This checkpoint is built for vLLM and needs the custom model plugin from the repo above (the base architecture, TinyAyaForCausalLM, is a Cohere2 backbone plus a small stop-prediction head and constrained audio-token decoding — not a standard vLLM model on its own).

git clone https://github.com/Sj0605-DataSci/rumik-oss-model-dissection
cd rumik-oss-model-dissection
pip install -e vllm-plugin/
VLLM_PLUGINS=register_tinyaya python3 vllm-plugin/vllm_infer.py --repo-dir <path-to-this-checkpoint>

Requires a Blackwell GPU (compute capability ≥ 12.0) and vLLM with compiled NVFP4 kernels (verified with vllm/vllm-openai:v0.20.0).

What's unchanged from the original

• Tokenizer, Mimi audio codec, and the stop_predictor head weights are identical to the original rumik-ai/rumik-oss-1 release.
• Generation behavior (which tokens get sampled, when generation stops) is reproduced faithfully — see the linked benchmark writeup for the validation that confirms this (generated audio durations match the original model closely on the same prompts).

Known limitations

• NVFP4 checkpoint format is still marked experimental by vLLM itself.
• Concurrency tested up to 48 simultaneous requests; scaling behavior beyond that, and under mixed/realistic traffic patterns, isn't characterized yet.
• This is an independent optimization/dissection, not an official Rumik release.

All credit for the original model, training, and data to the Rumik team — see rumik-ai/rumik-oss-1 and rumik.ai/research/rumik-oss.