speechless-twi-stage1-rvq-gemma4

Speechless TWI — Stage 1: RVQ for Gemma 4 Audio Encoder

Trained Residual Vector Quantizer that discretizes Gemma 4 audio encoder (1024-dim Conformer) representations into discrete semantic tokens for Twi/Akan.

Architecture

Training Data Sources

• Financial Inclusion Twi Multispeaker (~30h, ~200 speakers)
• Fante Multispeaker (~8h subsample)
• BibleTTS Asante Twi (~5h subsample, studio quality)
• BibleTTS Akuapem Twi (~5h subsample, studio quality)
• Common Voice Twi (~3h)

Results

Files

Usage

import torch, json
from huggingface_hub import hf_hub_download
from transformers import Gemma4AudioModel, Gemma4AudioFeatureExtractor

# Load RVQ
cfg = json.load(open(hf_hub_download("REPO_ID", "config_stage1.json")))
exec(open(hf_hub_download("REPO_ID", "rvq_wrapper.py")).read())
rvq = RVQWrapper(cfg["rvq_dim"], cfg["rvq_num_quantizers"], cfg["rvq_codebook_size"])
ckpt = torch.load(hf_hub_download("REPO_ID", "rvq_averaged.pt"), map_location="cpu")
rvq.load_state_dict(ckpt["rvq"])
rvq.eval()

# Load Gemma 4 encoder
encoder = Gemma4AudioModel.from_pretrained("rnagabh/gemma4-audio-encoder", torch_dtype=torch.bfloat16)
feat_ext = Gemma4AudioFeatureExtractor.from_pretrained("rnagabh/gemma4-audio-encoder")

# Encode audio → semantic tokens
import numpy as np
wav = np.random.randn(16000).astype(np.float32)  # your audio here
feats = feat_ext([wav], sampling_rate=16000, return_tensors="pt")

hook_store = {}
def hook(m, i, o): hook_store["h"] = i[0]
handle = encoder.output_proj.register_forward_hook(hook)
with torch.no_grad():
    encoder(feats["input_features"].to(torch.bfloat16))
    hs = hook_store["h"].float()
    _, indices, _ = rvq(hs)
handle.remove()

print("Token indices:", indices.shape)  # [1, Q, T]

Pipeline Context

This is Stage 1 of the Speechless pipeline for Twi:

Inference: Twi speech → Gemma 4 encoder → (same embedding space) → Gemma 4 LLM → Twi text response

Citation

Based on: Speechless: Speech Instruction Training Without Speech for Low Resource Languages (Dao et al., INTERSPEECH 2025, arXiv:2505.17417)

Trained: 2026-04-12