gemma3n-audio-encoder-VQ-32k-whisper-decoder

gemma3n-audio-encoder-VQ-32k-whisper-decoder

Combine mesolitica/gemma-3n-e4b-it-audio-encoder Encoder + Projection + VQ + Projection Layer Norm + openai/whisper-large-v3-turbo Decoder.

This model to introduce VQ on top mesolitica/gemma3n-audio-encoder-whisper-decoder

This is the most compressed speech token model, 6.25 TPS with 32768 embedding size.

WanDB at https://wandb.ai/huseinzol05/gemma3n-audio-vq-whisper-decoder-v5

Training dataset

1. malaysia-ai/common_voice_17_0
2. mesolitica/Malaysian-STT-Whisper-Stage2/malaysian_multiturn_chat_assistants_segments
3. mesolitica/Malaysian-STT-Whisper-Stage2/malaysian_multiturn_chat_assistants_manglish_segments

how to audio token

from transformers import AutoFeatureExtractor, AutoModel, AutoTokenizer
import librosa

model_id = "mesolitica/gemma3n-audio-encoder-VQ-32k-whisper-decoder"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code = True, torch_dtype = 'auto').cuda()
encoder = model.model.get_encoder()
y, sr = librosa.load('common_voice_ba_26517811.mp3', sr = feature_extractor.sampling_rate)
features = feature_extractor([y], return_tensors = 'pt')
features['input_features'] = features['input_features'].cuda()
features['input_features_mask'] = features['input_features_mask'].cuda()
_, tokens = encoder(**features)
print(tokens)

tensor([ 4679, 20093,  8341,  7777, 21322, 30807,  3741, 10235,  4053,  6004,
        17969,  1095, 30875, 10580,  9639, 22731, 29890, 28581, 20118,  3688,
        29172,  3227, 23437, 22097, 11855, 13388,  8268, 17958, 18715],
       device='cuda:0')

how to decode

from transformers import AutoFeatureExtractor, AutoModel, AutoTokenizer
import librosa

model_id = "mesolitica/gemma3n-audio-encoder-VQ-32k-whisper-decoder"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code = True, torch_dtype = 'auto').cuda()
tokenizer = AutoTokenizer.from_pretrained(model_id)

y, sr = librosa.load('common_voice_ba_26517811.mp3', sr = feature_extractor.sampling_rate)
input_ids = tokenizer(
    '<|startoftranscript|><|ru|><|transcribe|><|notimestamps|>', 
    add_special_tokens = False, return_tensors = 'pt')['input_ids']
features = feature_extractor([y], return_tensors = 'pt')
features['input_features'] = features['input_features'].cuda()
features['input_features_mask'] = features['input_features_mask'].cuda()
features['attention_mask'] = features['input_features_mask']
features['decoder_input_ids'] = input_ids.cuda()

generate_kwargs = dict(
    **features,
    max_new_tokens=1024,
)
generation_output = model.generate(**generate_kwargs)
tokenizer.decode(generation_output[0])

Output,

<|startoftranscript|><|ru|><|transcribe|><|notimestamps|> Купыкта был широкое глобка шляпше на битапсы.<|endoftext|>

Evaluation

Evaluate on malaysia-ai/common_voice_17_0/test up to 115 languages with some conditions,

1. Lower case.
2. Remove punctuation.
3. Provide language tagging for decoder input ids, <|startoftranscript|><|{lang}|><|transcribe|><|notimestamps|>.

We also compared with mesolitica/gemma3n-audio-encoder-whisper-decoder,

lang: gl, samples: 9949, CER: 0.4434100651127645, Difference CER: 0.3433969941516518
lang: en, samples: 16379, CER: 0.491967304650396, Difference CER: 0.31341208233551177
lang: ar, samples: 10458, CER: 0.6646916755934748, Difference CER: 0.27259434680871414
lang: kab, samples: 14972, CER: 0.5573320818874136, Difference CER: 0.2682886040511139
lang: ml, samples: 703, CER: 0.6912507475624741, Difference CER: 0.16014039867625407
lang: kk, samples: 514, CER: 0.5271876202388266, Difference CER: 0.2652086600730568
lang: ltg, samples: 2904, CER: 0.5114767250464123, Difference CER: 0.2812152823711915
lang: fr, samples: 16145, CER: 0.4739101373137794, Difference CER: 0.33027914863974916
lang: de, samples: 16170, CER: 0.467384524036037, Difference CER: 0.36535175892058525
lang: fi, samples: 1554, CER: 0.5333326269490939, Difference CER: 0.335315832887497
lang: pt, samples: 9432, CER: 0.4608595674397654, Difference CER: 0.2621939343736506
lang: ia, samples: 1816, CER: 0.3071555483082413, Difference CER: 0.19930406136758025
lang: eu, samples: 13621, CER: 0.45598884055743977, Difference CER: 0.37437287984873835
lang: ro, samples: 3896, CER: 0.44460973132177356, Difference CER: 0.26967838759748985
lang: sw, samples: 12086, CER: 0.5073141125603544, Difference CER: 0.3255685186863707
lang: sv-SE, samples: 5247, CER: 0.5394456283128104, Difference CER: 0.2924514788242536
lang: ta, samples: 8263, CER: 0.5060987792010176, Difference CER: 0.34694330449493194
lang: et, samples: 2653, CER: 0.6462712331929156, Difference CER: 0.36193483115291364
lang: lg, samples: 11902, CER: 0.5098737387978356, Difference CER: 0.31113640878481275
lang: it, samples: 15154, CER: 0.43857836286539176, Difference CER: 0.3471486530674707
lang: mhr, samples: 15107, CER: 0.4312138084188474, Difference CER: 0.28517428511332876
lang: sr, samples: 1539, CER: 0.36365414650647815, Difference CER: 0.1774638081285035
lang: mr, samples: 1437, CER: 0.617884037776395, Difference CER: 0.3485615698903225
lang: ka, samples: 12608, CER: 0.5324935796791754, Difference CER: 0.4187018324110657
lang: es, samples: 15848, CER: 0.4148185717086169, Difference CER: 0.33025384361916765
lang: be, samples: 15878, CER: 0.47463265071345456, Difference CER: 0.39463652856607145
lang: lt, samples: 4753, CER: 0.5414425145939812, Difference CER: 0.3240438134543827
lang: ca, samples: 16389, CER: 0.35390086014896116, Difference CER: 0.27546791608919935
lang: eo, samples: 14773, CER: 0.3278106680767194, Difference CER: 0.25688312683212056
lang: tr, samples: 11235, CER: 0.47513694278183155, Difference CER: 0.3367334737853406
lang: hu, samples: 11435, CER: 0.5531108079876421, Difference CER: 0.4119711667261202
lang: ja, samples: 6033, CER: 0.9077425365464499, Difference CER: 0.23764127183638462
lang: br, samples: 2202, CER: 0.5471387074745448, Difference CER: 0.17420646298311487
lang: ne-NP, samples: 217, CER: 0.6004590890861153, Difference CER: 0.2112445297814397
lang: uz, samples: 12006, CER: 0.5162576491330815, Difference CER: 0.35494690134742385
lang: ru, samples: 10184, CER: 0.5542631381554615, Difference CER: 0.38250689895197226
lang: dv, samples: 2213, CER: 0.633197544577893, Difference CER: 0.12903655325559527
lang: tt, samples: 4953, CER: 0.5053160995205921, Difference CER: 0.3350990650524065
lang: rw, samples: 14797, CER: 0.5398179646969407, Difference CER: 0.3389878596802388
lang: bn, samples: 9327, CER: 0.5999844064017279, Difference CER: 0.31274867664704037
lang: ug, samples: 6108, CER: 0.5699968308615198, Difference CER: 0.3831918653365691
lang: rm-sursilv, samples: 1361, CER: 0.509094340151972, Difference CER: 0.25296527745515524
lang: bg, samples: 3201, CER: 0.5532583803079603, Difference CER: 0.3539457954214664
lang: ab, samples: 9108, CER: 0.515020487430682, Difference CER: 0.31094133334888874
lang: uk, samples: 9915, CER: 0.5071036327376705, Difference CER: 0.35631637565538965
lang: mt, samples: 1662, CER: 0.5800270907666782, Difference CER: 0.28852832373222753
lang: fa, samples: 10292, CER: 0.4910854796682209, Difference CER: 0.28369128500388563
lang: pl, samples: 9186, CER: 0.5653488000388763, Difference CER: 0.3781066324977953
lang: bas, samples: 541, CER: 0.5466344908529508, Difference CER: 0.19707787661940196
lang: nl, samples: 11255, CER: 0.5129207738221947, Difference CER: 0.3711368644576756
lang: zh-CN, samples: 10335, CER: 0.8541503534811906, Difference CER: 0.3012558390941098
lang: tok, samples: 2175, CER: 0.264257441179988, Difference CER: 0.19631856513296336
lang: ur, samples: 4052, CER: 0.5527530529382912, Difference CER: 0.3358566177233667
lang: sk, samples: 2593, CER: 0.4914308091605027, Difference CER: 0.2800366317695697
lang: oc, samples: 254, CER: 0.5378203009687572, Difference CER: 0.2255962440893785
lang: yue, samples: 2585, CER: 0.8365652897461145, Difference CER: 0.18534275377588494
lang: mrj, samples: 7102, CER: 0.4722343833478152, Difference CER: 0.27861843176673606
lang: fy-NL, samples: 3167, CER: 0.5206207254560982, Difference CER: 0.2974301445709563
lang: cs, samples: 9055, CER: 0.5819181413836227, Difference CER: 0.41455814978254946
lang: th, samples: 10982, CER: 0.6724854596918328, Difference CER: 0.4129565964286039
lang: ckb, samples: 5262, CER: 0.468860467596021, Difference CER: 0.27016030472749486
lang: mn, samples: 1896, CER: 0.6593068736450253, Difference CER: 0.3132768597800812
lang: ky, samples: 1604, CER: 0.5822130029865302, Difference CER: 0.33361526283537574
lang: skr, samples: 1006, CER: 0.5837053862576943, Difference CER: 0.2157136973863757
lang: hy-AM, samples: 4281, CER: 0.5646891121938136, Difference CER: 0.3779595093843453
lang: sl, samples: 1242, CER: 0.46378147019291943, Difference CER: 0.2703534666006613
lang: vi, samples: 1077, CER: 0.5804975359487894, Difference CER: 0.2589067683319407
lang: hi, samples: 3151, CER: 0.5349342355542961, Difference CER: 0.3189640150505923
lang: nan-tw, samples: 2317, CER: 0.6098843080851416, Difference CER: -0.0482288576202029
lang: id, samples: 3633, CER: 0.3899058778847679, Difference CER: 0.2771819142215215
lang: cy, samples: 5371, CER: 0.5784888838507417, Difference CER: 0.30484885620646895
lang: yo, samples: 999, CER: 0.6728052521245995, Difference CER: 0.15981862810123726
lang: sah, samples: 1455, CER: 0.6190951958424097, Difference CER: 0.4005720011218832
lang: mk, samples: 1097, CER: 0.5501147463417055, Difference CER: 0.3596358018208492
lang: cv, samples: 1288, CER: 0.6000867555243313, Difference CER: 0.30407606631319894
lang: myv, samples: 479, CER: 0.5420571659133294, Difference CER: 0.3287449784598492
lang: da, samples: 2405, CER: 0.5717583076230536, Difference CER: 0.26868679501245485
lang: lv, samples: 6738, CER: 0.5219108646049374, Difference CER: 0.34519174478462855
lang: kmr, samples: 3900, CER: 0.4669804059667253, Difference CER: 0.2594262733968632
lang: tk, samples: 545, CER: 0.6935893960564069, Difference CER: 0.30980184486217444
lang: nn-NO, samples: 370, CER: 0.5651411865163424, Difference CER: 0.2568901111922791
lang: ha, samples: 661, CER: 0.432970109056588, Difference CER: 0.1508906026016678
lang: he, samples: 260, CER: 0.8359821400656229, Difference CER: -0.018109631913784674
lang: dyu, samples: 59, CER: 0.6204141749575005, Difference CER: 0.25276691280972013
lang: gn, samples: 855, CER: 0.6362538486802313, Difference CER: 0.27103397317539274
lang: lij, samples: 694, CER: 0.47076408549771004, Difference CER: 0.14437438704867617
lang: hsb, samples: 444, CER: 0.6333311895143787, Difference CER: 0.3354545995714982
lang: pa-IN, samples: 487, CER: 0.6073532431075208, Difference CER: -0.030561642084508067
lang: el, samples: 1696, CER: 0.5511874931877256, Difference CER: 0.31450101394451857
lang: zgh, samples: 159, CER: 0.92663633796379, Difference CER: -0.07336366203621003
lang: as, samples: 551, CER: 0.6360145021414202, Difference CER: 0.227073974977329
lang: sq, samples: 472, CER: 0.5924622382567383, Difference CER: 0.28423875103637575
lang: ko, samples: 338, CER: 0.8154387136871475, Difference CER: -0.1845612863128525
lang: ga-IE, samples: 517, CER: 0.6125545827471378, Difference CER: 0.10889718493488154
lang: cnh, samples: 763, CER: 0.46585304604939853, Difference CER: 0.18366608018765435
lang: sat, samples: 147, CER: 0.3508647436523669, Difference CER: -0.6491352563476331
lang: rm-vallader, samples: 462, CER: 0.5026374521524353, Difference CER: 0.19003056570303684
lang: or, samples: 670, CER: 0.6480353275973851, Difference CER: -0.28645046418104325
lang: mdf, samples: 104, CER: 0.5467010663483317, Difference CER: 0.26044412376253007
lang: af, samples: 62, CER: 0.6056762366643446, Difference CER: 0.25758913306272413
lang: ig, samples: 4, CER: 0.7398076503923279, Difference CER: 0.05680034873583262
lang: sc, samples: 232, CER: 0.5475061471491893, Difference CER: 0.2111703730753658
lang: tig, samples: 169, CER: 0.8087287689135308, Difference CER: -0.19127123108646915
lang: te, samples: 49, CER: 0.6941330313345585, Difference CER: -0.004581521211846296
lang: ps, samples: 199, CER: 0.534365926255053, Difference CER: 0.17918601027498526
lang: am, samples: 205, CER: 0.8136632271053513, Difference CER: -0.05734604153533751
lang: ast, samples: 162, CER: 0.4756498304513374, Difference CER: 0.2557716070641014
lang: os, samples: 50, CER: 0.6690129433442099, Difference CER: 0.11239095049663006
lang: lo, samples: 33, CER: 1.0, Difference CER: 0.0
lang: az, samples: 33, CER: 0.622821182347389, Difference CER: 0.28887930573413195
lang: ti, samples: 4, CER: 0.8912698412698412, Difference CER: -0.10873015873015879
lang: vot, samples: 6, CER: 0.4945652173913044, Difference CER: 0.1466289240888784
lang: nhi, samples: 5, CER: 0.7172699343667086, Difference CER: 0.22850998463901695
lang: yi, samples: 6, CER: 0.7957748071832881, Difference CER: -0.09864690356576267
lang: tw, samples: 9, CER: 0.6437220715662427, Difference CER: 0.2404688702155286

average CER: 0.5701075528698588

Source code

Source code at https://github.com/mesolitica/malaya-speech/tree/master/session/gemma3n-audio-whisper-decoder