Downloads · 30 days
35
18% of all-time downloads
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit
Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit is a audio-text-to-text model from VikramPal. Use it for the audio-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
This checkpoint is broken and is published broken on purpose. Do not use it for inference. It scores 25.00% (125/500) on the single task it was built for, against 86.80% for the bf16 model it was quantized from.
Downloads · 30 days
35
18% of all-time downloads
All-time downloads
190
Public
Parameters
3.2B
11.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors11.9 GB · 100%
How the weights are stored.
I322.7B · 84%
From the Hugging Face model README
This checkpoint is broken and is published broken on purpose. Do not use it for inference. It scores 25.00% (125/500) on the single task it was built for, against 86.80% for the bf16 model it was quantized from.
If you want something that runs, take
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit,
which scores 86.20% at 14.77 GiB and does not separate from the bf16 ceiling. It has the same two
caveats as this repo: it is text output only (no speech), and it needs both
pip install dynquant and an explicit dynquant.register_hf_quantizer() call before
from_pretrained.
It is 11.08 GiB and it answers 25.00% of a 60-way classification correctly, against a chance floor of 1.667%. It is above chance and it is useless. Three of its 500 generations came back unparseable; every other arm in the results table below returned zero.
The reason to publish it is that the failure has a specific, named, reproducible mechanism — a quantization target set below the allocator's own floor budget — and there is no way for anyone else to check that account without the artifact. This repository exists so that floor-override damage can be measured by someone other than the person claiming it. It is a control, not a product.
Required, and it is two steps, not one
pip install dynquant # step 1 of 2 -- installing is NOT enough on its ownimport dynquant dynquant.register_hf_quantizer() # step 2 of 2 -- MUST run before from_pretrained
The checkpoint carries a quantization_config whose quant_method is dynquant. If the
dynquant package is not installed — or is installed but dynquant.register_hf_quantizer() is
never called — transformers does not raise. It logs a warning (Unknown quantization type, got dynquant ... Hence, we will skip the quantization.), silently sets pre_quantized=False,
and from_pretrained returns a randomly initialised model that generates fluent garbage and
scores at chance. This exact failure cost two full evaluation runs during the campaign that
produced these checkpoints: it showed as 0.0% accuracy across two arms, with 499 and 500 of 500
answers unparseable, and the checkpoints were fine the whole time. Registration is an explicit
call, deliberately not an import side effect — there is no transformers entry point that does it
for you.
Since this artifact is already known to be broken, that failure would be easy to mistake for the real defect — so do not judge the load by the score. Count the swapped modules, which separates the two cleanly: a correctly loaded 3-bit checkpoint reports 602 DynQuant modules (504 linears + 2 embeddings + 96 expert banks; the 48 routers are dequantized to dense by design), and a silently unquantized one reports 0. The real collapse still answers in parseable indices 99.4% of the time; a random-init model does not.
dynquant 0.4.0 is on PyPI and is the exact version that produced this checkpoint.
Beyond it you need transformers>=5.15 and accelerate, which pip install dynquant does not
pull:
pip install dynquant "transformers>=5.15,<6" accelerate torch
Source: https://github.com/kambojvikram/dynquant.
Only the Thinker of Qwen3-Omni was trained, merged and quantized. The Talker and code2wav stacks (3,540,613,057 parameters) were nulled out before training and are excluded from the checkpoint entirely.
This model therefore:
The class is Qwen3OmniMoeThinkerForConditionalGeneration and model_type is
qwen3_omni_moe_thinker. config.json has exactly three sub-configs — audio_config,
text_config, vision_config — and no talker.
AutoModelForCausalLM and AutoModel both fail here. qwen3_omni_moe_thinker is absent
from MODEL_MAPPING_NAMES and registered only under image-text-to-text, so both raise
ValueError: Unrecognized configuration class Qwen3OmniMoeThinkerConfig. Use
Qwen3OmniMoeThinkerForConditionalGeneration directly, or AutoModelForImageTextToText. Needs
transformers >= 5.0 (produced and scored on 5.15.0; the snippets use the v5 dtype= spelling
of torch_dtype=); no trust_remote_code is needed.
Parameter counts: whole Omni 35,259,818,545; Thinker 31,719,205,488; Talker plus code2wav
3,540,613,057. The 96 batched MoE expert banks account for 28,991,029,248 parameters, 91.399% of
the Thinker. tie_word_embeddings is false — embed_tokens and lm_head are two separate
152064x2048 tensors, priced and quantized separately.
DynQuant allocates bit widths per module against a byte budget, subject to per-role floors. On this architecture the role floors alone cost 3.418 average bits. A 3.00-bit target therefore sits below the floor budget: the floors cannot all be satisfied at any allocation, the soft floors bind, and the allocator starts downgrading modules by lowest return on investment. It reports 400 of 650 modules breached.
At 4.00 bits the floors fit with 0.582 bits to spare, every floor is respected
(violations: 0), and the score decides only what gets upgraded. At 3.00 bits the score
decides what gets cut. The two arms are not the same experiment run at two budgets: the
3-bit arm measures floor-override damage. A hard-floor allocator would simply have refused a
3.00 target; that this arm ran at all is the soft-floor policy behaving as designed, and 25.00%
is the answer to what those cuts cost.
The concrete damage:
lm_head is cut from a floor of 8 bits to 3, the only five-bit drop in the map, in the
tensor that emits the token the task is scored on.No single breach is the cause; 400 modules moved at once and either injury alone might be
sufficient. Separating them requires an arm that pins lm_head at its floor and spends the
difference elsewhere, which is a change to the floor policy rather than another run of the
existing one. That experiment is named here and was not run.
One thing the floor policy did protect: the 48 MoE routers are never breached at either
budget. MOE_ROUTER is a structural role, so the tensors that decide which experts a token
passes through keep their 8 bits while everything around them is cut. The collapse at 3 bits is
not a routing collapse. (What "keeps its 8 bits" means precisely is set out under
Quantization detail — routers are quantized to 8 bits and written packed, then dequantized
to dense at load.)
Every arm below is the first 500 items of a random.Random(0) shuffle of the SLURP test
split — not the first 500 rows as the mirror serves them, which arrive grouped by speaker and
scenario. 4 shots drawn from train at seed 0, greedy decoding, max_new_tokens=8, stop at
the first newline, add_special_tokens=False, max_prompt_tokens=4096, batch size 8, MoE
experts dispatch pinned to eager, against the same 60-intent menu (intents_sha
d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d). Chance is 1.667%. Per-item
hits are stored, so every comparison below is an exact McNemar test on paired outcomes.
| arm | what it is | accuracy | vs comparator | discordant | p |
|---|---|---|---|---|---|
omni-base | whole Omni checkpoint, bf16, no fine-tune | 79.40% (397/500) | — | — | — |
omni-sft | Thinker-only, adapter merged, bf16 | 86.80% (434/500) | +7.40 vs base | 47/10 | 7.51e-07 separated |
omni-dq4 | DynQuant 4.00-bit map | 86.20% (431/500) | −0.60 vs sft | 12/15 | 0.7011 not separated |
omni-dq3 (this repo) | DynQuant 3.00-bit map | 25.00% (125/500) | −61.80 vs sft | 3/312 | 1.56e-88 separated |
95% confidence intervals on the paired differences: base to sft [+4.51, +10.29]; sft to dq4
[−2.64, +1.44]; sft to dq3 [−66.17, −57.43].
The discordance for this arm is 3/312: it recovered three items the bf16 model missed and lost 312 it had. It returned 3 unparseable generations; every other arm in this table returned 0.
Both packed checkpoints reproduce their in-memory counterpart on all 500 items, 0 discordant, at both widths. That is a round-trip test whose predicted outcome is exact identity — same codes, same scales, same compute dtype means bit-identical weights and therefore identical greedy tokens — so it is a check that the packed path matches the encoder, not an independent replication of the accuracy.
The model answers with an index into a numbered 60-intent menu — an integer such as 37 —
not with an intent name and not with a sentence. Anything outside that shape is scored wrong.
Menu. Read scenario and action from SLURP's own annotation at
https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl
for all three splits (train, devel, test), form f"{scenario}_{action}", take the
sorted unique set. That is 60 strings, numbered from 0; sha256("\n".join(menu)) must equal
the digest above. Deriving it from one split renumbers the taxonomy and every learned index
names its neighbour. The header is:
Listen to the spoken command and classify it into one of these intents.
Answer with the number only.
0. alarm_query
...
59. weather_query
Block order. The menu and exemplars come first, then the audio, then a trailing
Intent:. The exemplars are rendered as text, never as audio, in the form
"{transcript} -> {gold_index}" under the line Examples, as text:.
Audio. A bare mono float32 ndarray at 16 kHz inside the {"type": "audio", "audio": ...}
block, with sampling_rate=16000 passed beside the block to apply_chat_template. The
{"array", "sampling_rate"} dict raises; a wrong rate does not — it silently reads the clip
at the wrong speed.
Items. random.Random(0).shuffle(examples), then the first 500 of test.
Shots. sorted(random.Random(0).sample(range(50628), 4)) over the identically shuffled
train pool — an index sample, not a prefix. Their gold indices are 27, 15, 34, 22.
Scoring. Strip any reasoning trace, then take a leading integer in [0, 60) if there is
one, else the first in-range integer anywhere in the text; anything naming no index in
[0, 60) is unparseable and counts as wrong.
The reference implementation is dynquant.eval.slurp, which registers the quantizer itself and
raises if a packed checkpoint comes back unpacked:
pip install dynquant "transformers>=5.15,<6" "datasets>=4" torchcodec
dynquant eval VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit \
--task slurp --split test --limit 500 --shots 4 --shot-seed 0 \
--model-class Qwen3OmniMoeThinkerForConditionalGeneration \
--experts-impl eager --device cuda:0 --dtype bfloat16 --out dq3.json
Reproducing it downloads the whole SLURP test split and the whole train split (50,628
recordings, pulled in full to draw four text exemplars), so budget the disk and the network.
import torch
import dynquant
dynquant.register_hf_quantizer() # REQUIRED -- see the warning above
from transformers import AutoProcessor, Qwen3OmniMoeThinkerForConditionalGeneration
repo = "VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit"
processor = AutoProcessor.from_pretrained(repo)
assert type(processor).__name__ == "Qwen3OmniMoeProcessor", type(processor).__name__
model = Qwen3OmniMoeThinkerForConditionalGeneration.from_pretrained(
repo,
dtype=torch.bfloat16, # compute dtype; the weights stay packed
device_map="cuda:0", # needs `accelerate`
)
# Confirm the load. This is the check that separates the real collapse from a silent
# random-init: 0 means register_hf_quantizer() never ran, any other number means the
# module swap was partial. Do not weaken it to `> 0`.
packed = sum(1 for m in model.modules() if type(m).__name__.startswith("DynQuant"))
assert packed == 602, f"expected 602 packed modules, got {packed}"
# Text output only. There is no Talker in this checkpoint.
Ignore the "Use this model" snippet the Hub renders above this card. It emits a bare
AutoModel*call with nodynquantimport and no registration — exactly the path that returns a randomly initialised model.
Written and tested against transformers 5.15.0, torch 2.11+cu128, dynquant 0.4.0. Neutral decode
fields must be pinned explicitly (do_sample=False, num_beams=1, repetition_penalty=1.0, length_penalty=1.0, no_repeat_ngram_size=0): on transformers 5.x, fields left unset are filled
from the checkpoint's own generation_config, so a shipped repetition_penalty survives
do_sample=False and moves the score.
This directory ships processor_config.json and no preprocessor_config.json.
AutoProcessor resolves Qwen3OmniMoeProcessor from the former; if you see an error naming
preprocessor_config.json, that filename is a red herring — the merged bf16 reference has none
either and loads fine.
Three distinct levels, which should never be mixed. The GiB conversion is attached to the level it belongs to:
| weight-tensor bytes | shard-file bytes | directory (du -sb) | |
|---|---|---|---|
| bf16 merged Thinker | 63,438,410,976 (59.08 GiB) | 63,440,876,184 | 63,454,086,373 |
| DynQuant 4-bit | 15,859,386,016 (14.77 GiB) | 15,859,696,832 | 15,871,835,299 |
| DynQuant 3-bit (this repo) | 11,894,607,776 (11.08 GiB) | 11,894,918,480 | 11,907,056,499 |
Like-for-like compression, shard-file against shard-file: 5.33x for this arm, 4.00x for the 4-bit arm.
Resident VRAM, measured with torch.cuda.memory_allocated across a load on an idle card:
11,927,683,584 B for this arm (5.32x less than the bf16 merge's 63,440,876,184 B of shard
files — bf16 resident VRAM was not itself measured, so this is not a VRAM-to-VRAM comparison);
the 4-bit arm is 15,892,454,912 B, 3.99x less on the same denominator. Residency sits 0.28% above
the stored payload for this arm and 0.21% for the 4-bit one — 33,075,808 B and 33,068,896 B. Most
of that is the routers, not metadata: all 48 are stored packed at 8 bits (270,336 B each) but
restored to dense bfloat16 at load, costing 524,288 B resident each, so 48 x 253,952 B =
12,189,696 B — 36.9% — is that restoration on both arms, identical because MOE_ROUTER is a
structural 8-bit floor at either budget. The ~20.9 MB remainder is the runtime's own bookkeeping,
not the scales and offsets, which are already counted inside the stored payload.
Those are floors, not requirements: they are weights at load, with no activations, no KV cache, no audio-encoder workspace and no ~4k-token prompt. Peak allocation over a full 500-item run was not recorded in this campaign, so size a card with headroom over the figure above rather than to it.
The saving is real — these are packed checkpoints that dequantize at compute time, not fake-quant checkpoints carrying rounded weights at full width. Had any module fallen back to fp16 the residue would be gigabytes rather than tens of megabytes. None of that redeems the artifact. It is a small 25.00% model.
Group size 128, asymmetric (per-group bf16 scale plus bf16 offset), layout linear, compute
dtype bfloat16. 650 modules in the bit map — 602 written packed, plus the 48 routers, which
are restored dense at load — covering 31,712,079,872 elements.
7,125,616 dense parameters (norms, biases, convolutions) stay bf16. The 48 MoE routers
(mlp.gate) are a third case, and the one most often misstated: they are in the map at 8
bits — they are 48 of the 48 8-bit modules in the histogram below — and they are written
packed at 8 bits (270,336 B each, inside the packed total rather than beside it). Nothing in
the packed runtime can stand where a router stands, so the loader dequantizes them back into a
dense bfloat16 weight after load, which is why a read-back reports 48 of 48 routers dense and
why the assertion above expects 602 rather than 650. Their values therefore carry 8-bit
quantization error; only their storage at runtime is dense. What the floor policy guarantees is
that a router is never breached.
Achieved average bits for this arm: 2.997055 over quantized elements, 2.999976 over all Thinker parameters. (The 4-bit arm is 3.997249 and 3.999945 respectively.)
Width histogram:
| width | this arm (3.00) | the 4-bit arm (4.00) |
|---|---|---|
| 2-bit | 125 modules, 13,596,426,240 elements | 9 modules, 1,811,939,328 |
| 3-bit | 368 modules, 12,595,818,496 | 67 modules, 5,697,961,984 |
| 4-bit | 109 modules, 5,507,252,224 | 524 modules, 23,877,120,000 |
| 8-bit | 48 modules, 12,582,912 | 50 modules, 325,058,560 |
| floor violations | 400 of 650 | 0 |
The element-weighted mean of this arm's histogram is 2.7469 bits, not 2.9971. The difference is the group metadata, which the quoted average includes: a bf16 scale plus a bf16 offset every 128 elements is 0.25 bits per element. Add the 0.25 and you land within 0.0002 of the figure above, the remainder being rows whose length does not divide evenly into groups. The same arithmetic reconciles the 4-bit arm (3.7470 + 0.25).
The adapter was trained with QLoRA over an NF4 base, peft 0.20.0: r=16, lora_alpha=32,
lora_dropout=0.05, bias none, targeting q_proj, k_proj, v_proj, o_proj, out_proj,
fc1, fc2.
The MoE expert banks were not adapted. LoRA does not reach batched 3-D expert banks, so 91.399% of the parameters were frozen and unadapted throughout. The merge touched 384 weights; all 96 expert banks are unchanged from the base model.
Run with torchrun, world size 2, effective batch 16, 500 optimizer steps, 4,188.1 s wall, on
vast.ai: 2x RTX PRO 6000 Blackwell Max-Q (94.97 GiB each, sm_120), torch 2.11+cu128,
transformers 5.15.0, dynquant-core 0.4.0. Training used 8,000 of the 50,628 SLURP train
recordings and saw 59 distinct intents out of the 60-class menu. Final train_loss
4.031702354431152; the loss curve ran 5.479 to 3.080.
The adapter was trained against NF4 and merged into bf16 — standard for QLoRA, and it means the merged checkpoint is not bit-for-bit the model that produced the signal used to allocate the bit map.
SLURP intent classification. The label is the joint scenario_action pair, 60 classes. Audio
comes from the Hub dataset
marcel-gohsen/slurp; labels come from
SLURP's own annotation at
https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl.
Two sources are used because the mirrors' own intent column is corrupted: on 1,548 of 72,396
recordings it disagrees with the scenario/action pair, having dropped the scenario, so nine
different *_query intents all collapse to query. That yields 91/71/77 classes per split
against the real 60. The scenario and action fields are clean, so the joint pair is the label.
SLURP is Bastianelli et al., EMNLP 2020, CC BY 4.0.
Beyond the fact that this checkpoint does not work:
The expert banks were never adapted. 91.399% of the Thinker's parameters — the 96 batched MoE banks, 28,991,029,248 parameters — were frozen through fine-tuning and unchanged by the merge. Whatever the fine-tune achieved, it achieved through attention and the dense projections alone. The quantizer does reach them; the trainer did not.
500 evaluation items bound the resolution of every comparison here. That is enough to
separate a 61.80-point collapse at any sensible threshold, and it is not enough to resolve small
differences; the 4-bit arm's [−2.64, +1.44] interval is the honest form of that limit.
One task, one language. Everything reported is SLURP intent classification in English, under a fixed 4-shot prompt with the full 60-intent menu in context. Nothing here says anything about this quantization's effect on any other task, any other language, or the model's vision and general-purpose capabilities, none of which were measured, and the allocation signal itself was collected on this one task.
SLURP is one collection with one recording protocol. It is not a sample of arbitrary field audio, and accuracy on it does not transfer unexamined to other microphones and acoustic conditions, telephony audio, unfamiliar accents, or spontaneous speech outside this collection.
No baseline quantizers. GPTQ, AWQ and RTN were out of scope for this campaign. There is therefore no matched-bytes competitive claim, and specifically no evidence about whether this collapse is DynQuant's allocation or simply what 3 bits does to this architecture. A uniform-width control at the same byte anchor would answer that in one run and was not run.
The base-to-SFT comparison differs in two things, not one: the base screen ran the whole Omni
checkpoint while every post-merge arm runs the Thinker alone, and only the post-merge arms carry
the merged adapter. The arms carrying the actual quantization claim (sft against dq4 against
dq3) are all Thinker-only and differ in exactly one thing.
No speech output. See the scope section; the Talker was removed before training.
tokenizer.json). Its adapter_config.json carries an empty base_model_name_or_path
and a null task_type, so nothing in the file tells PeftModel.from_pretrained what to
attach to: load the base yourself and attach to .thinker, never to the whole Omni model,
whose Talker carries matching q_proj/k_proj/v_proj/o_proj names the adapter was never fitted
against.DynQuant source, format specification and the allocator that produced this bit map: https://github.com/kambojvikram/dynquant.
The base model is
Qwen/Qwen3-Omni-30B-A3B-Instruct,
released under license: other with license_name: apache-2.0. This derivative mirrors that,
and nothing here grants rights beyond the base model's terms. SLURP is CC BY 4.0 and its terms
apply to the data, not to these weights.
SLURP:
@inproceedings{bastianelli-etal-2020-slurp,
title = {{SLURP}: A Spoken Language Understanding Resource Package},
author = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2020}
}
This artifact:
@misc{dynquant_omni_slurp_3bit,
title = {Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit: a measured floor-override collapse},
author = {Pal, Vikram},
year = {2026},
note = {DynQuant 0.4.0, 3.00-bit target below a 3.418-bit floor budget; 25.00\% on SLURP intent classification},
url = {https://github.com/kambojvikram/dynquant}
}