Realtime-Venus-Omni-heretic-ARA-NVFP4

Realtime-Venus-Omni-heretic-ARA-NVFP4

justsumguy/Realtime-Venus-Omni-heretic-ARA, quantised with NVIDIA ModelOpt 0.46.1. It is 9.1 GB on disk. The abliteration details and numbers are on that page. In short: heretic ARA branch, ARA-LoRA rank 128, trial 83, refusals 98 → 4 of 100, KL 0.249 on recheck, no healing applied.

Quantisation

• Recipes: ModelOpt NVFP4_DEFAULT_CFG for llm; FP8_DEFAULT_CFG for vpm, with patch_embedding excluded.

• llm calibration: 256 samples × 512 tokens of tatsu-lab/alpaca (shuffled, seed 0), run through the extracted text tower. Mean NLL on 8 calibration chats: BF16 3.941, NVFP4 fake-quant 4.065 (+0.123).

• vpm calibration: 80 images from johnowhitaker/imagenette2-320. Three quarters were unsliced; one quarter used max_slice_nums=4.

• vpm held-out check (16 images):

  Output	Mean cosine	Min cosine	Mean rel. L2
  vision tower, last hidden state	0.972	0.957	0.234
  resampler	0.989	0.977	0.145

• Hardware:

  • One RTX 5060 Ti 16 GB (SM120) and one RTX 3050 Ti Laptop 4 GB (SM86), with lm_head on CPU during calibration. The host had 15 GB of RAM.
  • The BF16 tower was streamed from the shards across both GPUs. It was never materialised on one card.

Loading

transformers 4.51 (the version upstream's remote code pins) has no ModelOpt quantiser, so from_pretrained cannot load this export. nvfp4_loader/ is a small loader that does it. It:

• builds the model on the meta device;
• replaces the quantised projections with NVFP4Linear / FP8Linear;
• streams the shards directly onto the GPU.

The layers run on torch._scaled_mm FP4/FP8 kernels, with a fused Triton kernel for NVFP4 activation quantisation. Tested on torch 2.8.0+cu128, on an RTX 5060 Ti (SM120) only.

import os, sys, torch
os.environ["HF_HUB_OFFLINE"] = "1"
sys.path.insert(0, "/path/to/Realtime-Venus-Omni-heretic-ARA-NVFP4")
from nvfp4_loader import load_nvfp4_model

path = "/path/to/Realtime-Venus-Omni-heretic-ARA-NVFP4"
model = load_nvfp4_model(path, device="cuda:0")
model.init_tts(model_dir=os.path.join(path, "assets/token2wav"), n_timesteps=8)
duplex = model.as_duplex(generate_audio=True, n_timesteps=8)

OMNI_NVFP4_ACT=dynamic switches activation scaling from the checkpoint's static input_scale to a per-call amax.

Realtime measurements

These were measured on one RTX 5060 Ti 16 GB. The serving sidecar is custom and not included here. On top of the loader above, it adds:

• a static KV cache (8192 slots) with CUDA graphs over the language model's and the talker's single-token steps;
• one lm_head application per step;
• a batched system-prompt prefill;
• a CUDA graph over the token2wav flow DiT chunk step, with the vocoder in FP16 and the solver cache sized to the step count.

On an RTX 5060 Ti, with these optimisations, it should run in realtime with 8 vocoder (token2wav flow-matching) steps.

License

• Model weights, config and remote code: Apache-2.0, as upstream (see LICENSE and NOTICE). The modifications are the abliteration of the language tower and the quantisation described above.
• nvfp4_loader/: MIT, Copyright (c) 2026 Schneewolf Labs.