omniASR-CTC-1B

omniASR-CTC-1B

Wav2Vec2 CTC ASR model (1B) converted from the OmniLingual fairseq2 checkpoint omniASR_CTC_1B.

This model outputs CTC logits over a SentencePiece vocabulary and can transcribe speech in multiple languages.

Code Base

The code base for the conversion can be found here. I was only able to convert the 300M and 1B models due to GPU limitations. Contributions are welcome.

Model details

Numerical parity against the original fairseq2 checkpoint has been confirmed: outputs match to within atol=1e-4 on a held-out audio sample.

Sample transcriptions on the held-out audio clip:

Usage

from transformers import Wav2Vec2ForCTC, AutoProcessor
import torch, torchaudio

processor = AutoProcessor.from_pretrained("aadel4/omniASR-CTC-1B")
model     = Wav2Vec2ForCTC.from_pretrained("aadel4/omniASR-CTC-1B")
model.eval()

waveform, sr = torchaudio.load("audio.wav")
if sr != 16_000:
    waveform = torchaudio.functional.resample(waveform, sr, 16_000)

inputs = processor(
    waveform.squeeze().numpy(), sampling_rate=16_000, return_tensors="pt"
)
with torch.no_grad():
    logits = model(**inputs).logits          # (1, T, vocab)

pred_ids   = torch.argmax(logits, dim=-1)
transcript = processor.decode(pred_ids[0])
print(transcript)