glm-4-voice-bridge-GLM-4.6V-Flash-alpha

glm-4-voice → GLM-4.6V-Flash: audio bridge (alpha)

Built with glm-4. A research artifact from a model-surgery experiment: giving "ears" to zai-org/GLM-4.6V-Flash (image + text, MIT) by using the speech embeddings of zai-org/glm-4-voice-9b as a donor, without touching a single host weight. The long-term goal is one model with audio, video and image; this repository covers the first step, audio. It exists to record the method, the measured numbers and the causal controls. Nothing here is a benchmark claim.

Update 2026-09-06 — the earlier negative result was wrong, and it was wrong for a boring reason. The first version of this card said open-vocabulary Portuguese ASR "does not work yet", based on runs capped at 12,000 steps. Trained for 10 epochs over the full corpus (87,550 steps), the same 131,073-parameter bridge reaches CER 0.4250 on a speaker-disjoint held-out set, against 0.7573 for the best constant transcription of the corpus and 0.7991 for a paired control given the same budget. The plateau was undertraining, not a ceiling.

1. Headline numbers [measured]

Final bridge, 87,550 steps, 300 held-out clips with speakers never seen in training. The shuffled_map row is from its curve samples (120 clips per eval) because that run died at 40,000 steps; every other row is a 300-clip final.

The random control sits at 0.7826 on its own training set, which rules out the alternative reading: it is not "learned but did not transfer", it is "did not learn".

Sentence identification, 10-way, length-matched distractors (the model scores 10 candidate transcriptions and picks the highest log-probability; distractor lengths matched to within 0.12 characters, so the duration shortcut is closed):

Choosing is far easier than generating, which is why 0.9867 coexists with CER 0.4250. The acoustic information arrives at the host essentially intact; the bottleneck is decoding it into text.

2. What it is

• Host: GLM-4.6V-Flash, entirely frozen (embedding, layers, lm_head). No new output tokens; the output vocabulary is still text only.

• Donor: glm-4-voice-9b. It provides (a) the speech tokenizer zai-org/glm-4-voice-tokenizer (Whisper encoder + VQ, 16,384 <|audio_N|> codes, 30 s window) and (b) the embedding table of those codes (ids from 152353 on).

• Bridge (the only trained module), per code c:

  E[c] = base[c] + (v[c] @ A) @ B · (α / rank), with gain = exp(gain_log) scaling the acoustic block.

  base is the map table (donor speech embeddings carried into the host input space by a ridge regression fitted on the 151,329 text tokens the two vocabularies share); v is the centered donor speech embedding; A (4096×16) and B (16×4096) start with B = 0, so at step 0 the bridge is exactly the table's zero-shot. 131,073 trainable parameters.

• Sequence: [gMASK]<sop><|user|>\n instruction \n [AUDIO] \n<|assistant|>\n<think></think> transcription <eos>. Loss falls only on the transcription and the <eos>. The empty <think></think> is not decoration: without closing the reasoning block the model "thinks" instead of answering (logit 31.5 for <think> against 16.75 for the runner-up, measured).

• Corpus: Common Voice PT, 8,755 training clips and 1,552 held-out clips, disjoint speakers.

• Paired controls in every measurement: shuffled_map (the same table rows, permuted with the same index on base and v: same information, same parameter count, donor geometry destroyed) and random (Gaussian noise with per-dimension mean and std matched). No gain is accepted without them.

3. The qualitative change is the most convincing part

At 20,000 steps the wrong hypotheses were fluent, plausible Portuguese with the right content words. At 87,550 steps they became phonetic transcription with broken word segmentation:

"Horário de Janelas" for "Rio de Janeiro"; "istores da língua dos conuns" for "História da Liga dos Comunistas". The model started following the sound and failing at orthography and word boundaries. That is the signature of an acoustic front-end working without a strong language model on top — the opposite of hallucinating a likely sentence.

4. What is in this repository

Bridge tensors: A [4096,16] f32, B [16,4096] f32, gain_log scalar f32, with metadata condicao, rank, passos (steps), lr. The embedding tables are not here (1 GB, derived from the donor weights); the script rebuilds them from the two original models.

5. How to use (research level)

Requirements: both original checkpoints, PyTorch + Transformers with glm4v support, ffmpeg, a 24 GB GPU (the host in bf16 takes ~19.7 GiB; training peaks at 21.8 GiB reserved).

# 1. tables (once): reads the host text embedding and the donor speech embedding
python scripts/g2_voice_etapa0_tables.py
# 2. VQ codes for a {audio, text} manifest
python scripts/g2_voice_etapa1_tokenize.py --manifest my-manifest.jsonl --out manifest-codes.jsonl
# 3. evaluate the final bridge (300 held-out clips, paired with 300 training clips)
python scripts/g2_voice_etapa1_asr.py --codes manifest-codes.jsonl --condition map --rank 16 \
  --load-bridge pontes/cv-pt-map-r16-10ep-final-87550.safetensors --steps 0 --final-n 300 --out eval.json
# 4. the identification test and its floor
python scripts/g2_voice_etapa1_ident.py --codes manifest-codes.jsonl --condition map --rank 16 \
  --load-bridge pontes/cv-pt-map-r16-10ep-final-87550.safetensors \
  --n-clipes 300 --n-vias 10 --distratoras comprimento --out ident.json

The scripts expect the layout of the source repository (models/zai-org__GLM-4.6V-Flash, derived_models/etapa0-audio); adjust the paths at the top of each file.

6. History of the measurements

Hard reference on the held-out set: the best constant transcription of the corpus gives CER 0.7573. Anything that does not go below that has learned nothing useful.

Caveat that matters [judgment]. Reports 005, 007 and 009 concluded negatively using the same 12,000-step budget that report 011 has now shown to be undertraining — at 10,000 steps even the real bridge sits at CER 0.80, indistinguishable from the controls. Their negative conclusions are measured in the wrong regime and need to be redone before being treated as fact. The most valuable of them is 009 (LoRA on the host + bridge), because with identification already at 0.9867 the bottleneck is plausibly decoding, which is exactly what unfreezing the backbone would attack.

7. Limitations

• Not usable ASR. CER 0.4250 on Portuguese with an unseen speaker is far from any practical use.
• Not a ready-to-use model: it needs both original checkpoints, the rebuilt tables and the scripts.
• The generalization gap opened by the end: 0.3354 on training clips against 0.4250 held-out, and 4× more exact sentences on training. It is starting to memorize speakers, though held-out kept improving to the last step.
• Portuguese only (Common Voice PT). English was tested once, at the undertrained budget, and lost.
• No video. Image is native to the host and was neither touched nor measured after the bridge.
• The bridge is input-side only: the model reads speech codes, it does not generate speech.
• No standard benchmark; the numbers are CER/WER on a speaker-disjoint held-out split of the same corpus.

8. Licenses and attribution

• The bridge weights (A, B, gain_log), the scripts and the reports in this repository: MIT (LICENSE).
• The host zai-org/GLM-4.6V-Flash is MIT.
• The donor zai-org/glm-4-voice-9b is under the glm-4-voice License (copy in licenses/glm-4-voice-LICENSE.txt): free academic use, commercial use upon registration with Zhipu, the obligation to display "Built with glm-4" and to start the name of any model created with its materials with "glm-4". The tables this method uses derive from the donor embedding; that is why they are not redistributed here and why this repository's name starts with glm-4.
• Common Voice (Mozilla) was used only as an evaluation corpus; no audio or transcription is redistributed.