rumik-oss-1-FP8-Dynamic

rumik-oss-1 FP8 Dynamic

An FP8 quantization of rumik-ai/rumik-oss-1, Rumik's 3B multilingual text to speech model for 22 Indic languages and English, with expressive delivery control, inline vocalizations and 24 kHz audio output.

This checkpoint stores the weights of the transformer linear layers in 8 bit floating point (FP8) and quantizes activations dynamically at runtime. It needs no calibration data and keeps the output heads in bf16. The goal is a smaller checkpoint and lower memory use with the same behavior as the original model.

This is an independent quantization. It is not an official Rumik release. All credit for the model, training and data goes to the Rumik team.

Quantization details

Hardware requirements

Native FP8 compute needs an NVIDIA GPU with compute capability 8.9 or higher (Ada Lovelace, Hopper or Blackwell). On GPUs without FP8 support, use the original bf16 model instead.

Usage

pip install -U transformers accelerate compressed-tensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Henil1/rumik-oss-1-FP8-Dynamic"

tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, device_map="auto", trust_remote_code=True
)

The model uses custom code, so trust_remote_code=True is required. Speech generation, the voice list and the audio codec setup are unchanged from the original model, so follow the usage section of the original model card for generating and decoding audio. 100 audio tokens make one second of speech, and the Mimi codec decodes them to 24 kHz audio.

Controls (unchanged from the original)

The delivery description can also be written inline, for example <description="excited, Hindi accent, fast pace"> your text here <laugh>.

How this checkpoint was made

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

SRC = "rumik-ai/rumik-oss-1"

model = AutoModelForCausalLM.from_pretrained(
    SRC, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained(SRC, trust_remote_code=True)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*stop_predictor.*"],
)
oneshot(model=model, recipe=recipe)

model.save_pretrained("rumik-oss-1-FP8-Dynamic", save_compressed=True)
tok.save_pretrained("rumik-oss-1-FP8-Dynamic")

Limitations

• Quantization can slightly change pronunciation, prosody and pacing compared with the bf16 model. Compare a few sentences in your own languages before relying on it.
• Only the original model card's benchmarks apply to the base model. This checkpoint has not been re-benchmarked on IndicEmo, NoVA or WER and CER.
• The model uses custom architecture code and may not load in inference engines that do not support it.
• The same limitations, biases and acceptable use terms as the original model apply.

License

Research and non commercial use only, under the CC BY NC 4.0 license with the acceptable use addendum inherited from Tiny Aya Fire, unchanged from the original model. The original LICENSE and NOTICE files are included with this repository, and this repository adds FP8 weight quantization as a modification. The Mimi codec is licensed under CC BY 4.0.

Citation

@unpublished{govindu2026rumikoss1,
  title  = {{rumik-oss 1 technical report}},
  author = {Govindu Pranav and Anant Shukla and Suryansh Shakya and Aman Anand and Vatsal Bharti},
  year   = {2026},
  note   = {In preparation}
}