qwen3.5-4b-transcript-formatter-lora

Qwen3.5-4B Transcript Formatter — LoRA adapter (distilled student)

A LoRA adapter for Qwen/Qwen3.5-4B that turns a raw speech-to-text transcript into a clean, formatted transcript — fixing punctuation, casing, fillers, disfluencies, homophones, ITN, garbled proper nouns / phonetic URLs-emails, and spoken layout commands — while preserving the speaker's words and passing through anything that is already correct.

This is a distilled 4B student in the gpt-oss-120b → 20B → 4B lineage. It reproduces the teacher's formatting behaviour at a fraction of the cost: the (thinking-capable) Qwen emits the formatted transcript directly on the answer channel, no chain-of-thought, on a 4B base that serves cheaply.

This repo ships only the LoRA adapter (~hundreds of MB), not the merged 4B. Merge it on demand against the public base — see Usage.

Model details

Training approach — cross-architecture distillation

The student is trained by distilling the distilled 20B student onto the Qwen3.5-4B base (off-policy prompt distillation). Because gpt-oss and Qwen have different tokenizers/vocabularies, on-policy token-level KL cannot cross the architecture boundary — so the transfer is done as off-policy SFT on the teacher's decoded text:

1. Teacher generation. The 20B teacher (direct final channel) produces gold formatted outputs over the transcript-formatter-curriculum prompts (all 21 categories, L0–L5 + RL-align).
2. Prompt distillation. The formatter's long rulebook lives only in the teacher traces; the Qwen student is trained on raw transcript → formatted transcript pairs (the Qwen chat template is the only wrapper), so it internalises the behaviour without a system prompt at inference.
3. Student SFT. Qwen3.5-4B is fine-tuned (LoRA rank 32, lr ≈ 4.9e-4 linear, 3 epochs, batch 128, max_len 2048) with qwen3_5_disable_thinking so it answers immediately, no reasoning trace.

21 categories total across the pooled curriculum. Full dataset: Akash-Sakala/transcript-formatter-curriculum.

Results

Held-out evaluation over 2,384 rows across all 21 categories, scored with the same two-round protocol as the teacher: deterministic Round 1 (EM / ContentOK / CER), then an independent Claude judge re-adjudicates Round-1 fails to credit cases where the model is correct but the gold label is unrealistic ("adjusted accuracy").

Student numbers measured over 2,384 held-out rows (all 21 categories), greedy decoding. 44 Round-1 fails re-adjudicated → 29 rescued, 15 confirmed fails. Source: final_eval_all_layers_Qwen.json / final_eval_round1_Qwen.json. The cross-architecture distillation reaches near-teacher parity (99.4% vs 99.5% adjusted acc) at 4B with no reasoning — the adjusted accuracy is flat across all five layers (L0–L4 all ≥ 99.0%). The only systematic residual gap is ITN clock-time normalisation (e.g. two thirty PM → 2:30 PM).

EM understates quality for the layout categories (many valid paragraph/list renderings); read ContentOK / adjusted accuracy / CER, not EM.

Usage

The formatter behaviour is prompt-distilled — the student was trained to format directly from the raw transcript, so no system prompt is needed at inference. (Optionally prepend the training system prompt for maximum fidelity.)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-4B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "Akash-Sakala/qwen3.5-4b-transcript-formatter-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")

messages = [{"role": "user",
             "content": "i spoke to aisha she confirmed the PR will be merged by friday"}]
ids = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt",
    enable_thinking=False).to(model.device)   # disable the <think> block
out = model.generate(ids, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
# → "I spoke to Aisha. She confirmed the PR will be merged by Friday."

Merge once for serving (vLLM/SGLang):

# Option A — PEFT
merged = PeftModel.from_pretrained(
    base, "Akash-Sakala/qwen3.5-4b-transcript-formatter-lora").merge_and_unload()
merged.save_pretrained("./qwen3.5-4b-transcript-formatter-merged")

# Option B — Tinker cookbook
from tinker_cookbook import weights
weights.build_hf_model(
    base_model="Qwen/Qwen3.5-4B",
    adapter_path="./adapter",            # this repo, downloaded locally
    output_path="./merged",
    dtype="bfloat16",
)

Hardware: Qwen3.5-4B fits on a single mid-VRAM GPU; at int4 ≈ 2.0 GB. Greedy decoding (temperature 0) for deterministic output. Disable thinking (enable_thinking=False) so the model emits the transcript directly.

Intended use & limitations

• Use: post-processing ASR/Whisper output into readable text (dictation, meeting notes, voice memos, emails).
• Preserves words. It edits format, not meaning — it does not paraphrase, summarise, answer, or invent content; ambiguous cases pass through unchanged.
• Limitations: English only; rare/novel proper-noun spellings can be missed; spelled-out clock times are sometimes left un-normalised (three PM vs 3:00 PM); paragraph segmentation of free-dictated emails is subjective; not a general assistant (it only formats transcripts).
• License: Apache-2.0, inherited from the base Qwen/Qwen3.5-4B.

Citation / provenance

Cross-architecture distillation with the Tinker API: teacher gpt-oss-20b-transcript-formatter-lora (itself distilled from the RL-final 120B) → off-policy SFT onto Qwen/Qwen3.5-4B, LoRA rank 32, lr ≈ 4.9e-4 linear, 3 epochs, batch 128, no reasoning, renderer qwen3_5_disable_thinking. Tinker checkpoint: 0f0e8e0d-eb7b-5333-a9be-6aaa631536ff:train:0/sampler_weights/000400. See the dataset card for the full curriculum.