tokenizer-vla-qwen3-v2

VLA Tokenizer — Qwen3 v2 (listen/speak wrapper fix + MV-Omni SNAC bands)

Successor to tokenizer-vla-qwen3 (used to train vla-1.7b-qwen3-v2). Purely additive — every token ID from the previous tokenizer is unchanged; this only registers new tokens that were missing.

Vocab size: 274,561 (151,643 Qwen3 base + 122,918 added VLA tokens, up from 257,901 / 106,258 in the previous tokenizer — +16,660 new tokens).

What changed and why

1. <listen>/</listen>/<speak>/</speak> are now atomic. The previous tokenizer was built before the 2026-07-23 decision to replace the generic <snac>/</snac> SNAC wrapper with <listen> (ambient/scene audio the model perceives — used by FineVideo-VLA and OmniVideo-100K) and <speak> (a generated reply — used by the emotional-roleplay dataset). Verified directly (tokenizers.Tokenizer.encode(), not assumed): in the old tokenizer these 4 tags split into 3-4 BPE subword pieces each; here each encodes to a single token ID.

2. MixtureVitae-Omni's full SNAC ID range is now covered. Streaming 50,000 real records from MV-Omni's mv_omni_snac_*.jsonl.gz turned up 28,672 unique <snac_N> IDs. The old tokenizer only registered the 3 bands this project's own tokenize_snac.py produces (L0/L1a/L1b, 128266-132361/132362-136457/144650-148745) — 16,384 of the 28,672 IDs found (57%), spanning 2 further 4096-wide bands (136458-144649, 148746-156937), were unregistered and would have split into multiple BPE pieces. All 28,672 are atomic here.

Without this tokenizer, essentially all <listen>/<speak>-wrapped SNAC audio project-wide (FineVideo-VLA, OmniVideo-100K, emotional-roleplay, and MixtureVitae-Omni) would silently lose the atomic-token property those projects' data pipelines assume.

Compatibility

All 257,901 token IDs from tokenizer-vla-qwen3 are unchanged (verified — this is a strict superset, additions only, no renumbering). Text tokenized with the old tokenizer remains valid under this one.

Base

Qwen/Qwen3-1.7B-Base tokenizer, extended the same way as tokenizer-vla-qwen3 (see that repo's card for the full original token-family breakdown: seed2/cosmos/avclm/agent/snac/ caption/speech).

Related