Downloads · 30 days
31
19% of all-time downloads
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit
Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit is a audio-text-to-text model from VikramPal. Use it for the audio-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
Required, and it is two steps, not one This checkpoint declares quantizationconfig with quantmethod: "dynquant". If the dynquant package is not installed — or is installed but dynquant.registerhfquantizer() is never c…
Downloads · 30 days
31
19% of all-time downloads
All-time downloads
162
Public
Parameters
4.2B
15.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors15.9 GB · 100%
How the weights are stored.
I323.7B · 88%
From the Hugging Face model README
Required, and it is two steps, not one
pip install dynquant # step 1 of 2 -- installing is NOT enough on its ownimport dynquant dynquant.register_hf_quantizer() # step 2 of 2 -- MUST run before from_pretrainedThis checkpoint declares
quantization_configwithquant_method: "dynquant". If thedynquantpackage is not installed — or is installed butdynquant.register_hf_quantizer()is never called —transformersdoes not raise. It logs a warning (Unknown quantization type, got dynquant ... Hence, we will skip the quantization.), silently setspre_quantized=False, andfrom_pretrainedreturns a randomly initialised model that generates fluent nonsense and scores at chance. This exact failure cost two full evaluation runs during the campaign that produced these artifacts, on a box where the package was installed. Registration is an explicit call and deliberately not an import side effect: there is no transformers entry point that does it for you. There is a load-time assertion in the snippet below; use it.
dynquant0.4.0 on PyPI is the exact version that produced this checkpoint. Source: https://github.com/kambojvikram/dynquant
A 4.00-bit DynQuant quantization of a QLoRA fine-tune of
Qwen/Qwen3-Omni-30B-A3B-Instruct,
trained for SLURP spoken-intent classification. Thinker only: it listens, it does not speak.
Audio, image, video and text in; text out. The Talker and code2wav stacks were removed before
training and are not in this checkpoint.
The checkpoint is 14.77 GiB of shard files against 59.08 GiB for the bf16 merge it was made from — 4.00x fewer bytes — and scores 86.20% (431/500) on the evaluation protocol below, against 86.80% (434/500) for that bf16 parent. On paired per-item outcomes the difference is −0.60 points at p = 0.7011: an exact McNemar test does not separate the quantized model from the model it came from.
Of the four repositories in this campaign, this is the one to take if you want to run the model.
Only the Thinker was trained, merged and quantized. The Talker and code2wav stacks were nulled out before training — 3,540,613,057 parameters, excluded — so this checkpoint:
The class is Qwen3OmniMoeThinkerForConditionalGeneration, model_type is
qwen3_omni_moe_thinker, and config.json carries exactly three sub-configs — audio_config,
text_config, vision_config. There is no talker config.
AutoModelForCausalLM and AutoModel both fail here. qwen3_omni_moe_thinker is absent
from MODEL_MAPPING_NAMES and is registered only under image-text-to-text, so both raise
ValueError: Unrecognized configuration class Qwen3OmniMoeThinkerConfig. Use the Thinker class
directly, or AutoModelForImageTextToText. Needs transformers >= 5.0 (this checkpoint was
produced and scored on 5.15.0, and the snippets below use the v5 dtype= spelling of
torch_dtype=); no trust_remote_code is needed, the architecture is native.
| parameters | |
|---|---|
| whole Omni checkpoint | 35,259,818,545 |
| Thinker (this model) | 31,719,205,488 |
| Talker + code2wav (removed) | 3,540,613,057 |
| the 96 batched MoE expert banks | 28,991,029,248 (91.399% of the Thinker) |
tie_word_embeddings is false: embed_tokens and lm_head are two separate 152064x2048
tensors, and are priced and quantized separately.
The directory ships processor_config.json and no preprocessor_config.json, which is
correct for this model family — AutoProcessor resolves Qwen3OmniMoeProcessor from it
directly. An OSError naming preprocessor_config.json is a red herring; the file that decides
is processor_config.json.
What you need. ~14.8 GiB of VRAM for the weights (measured resident: 15,892,454,912 B),
plus activations for the audio encoder and a ~4k-token prompt. A single 24 GB card (3090 / 4090
/ A10G) is the practical floor at batch 1; below that you need device_map="auto" across two
cards or CPU offload. Disk: 14.77 GiB. Peak allocation over a full 500-item run was not recorded
in this campaign, so treat the resident figure as a floor rather than a requirement.
Beyond dynquant you need transformers>=5.15 and accelerate — pip install dynquant
installs neither, so install them yourself:
pip install dynquant "transformers>=5.15,<6" accelerate torch
Ignore the "Use this model" snippet the Hub renders above this card.
library_name: transformersmakes it emit a bareAutoModel*call with nodynquantimport and no registration, which is exactly the path that returns a randomly initialised model. Use the code below.
import torch
import dynquant
from transformers import AutoProcessor, Qwen3OmniMoeThinkerForConditionalGeneration
dynquant.register_hf_quantizer() # NOT optional, and NOT done by pip. Without this call
# transformers cannot resolve quant_method="dynquant",
# logs a warning, sets pre_quantized=False, and returns a
# randomly initialised model without raising.
repo = "VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit"
processor = AutoProcessor.from_pretrained(repo)
assert type(processor).__name__ == "Qwen3OmniMoeProcessor", type(processor).__name__
model = Qwen3OmniMoeThinkerForConditionalGeneration.from_pretrained(
repo,
dtype=torch.bfloat16, # compute dtype; the weights stay packed
device_map="auto", # needs `accelerate`
)
# Do not skip this, and do not weaken it to `> 0`. A partial swap -- some modules packed,
# some left dense -- passes a `> 0` check and is exactly the case where the score drifts
# without an error.
packed = sum(1 for m in model.modules() if type(m).__name__.startswith("DynQuant"))
assert packed == 602, (
f"expected 602 packed modules (504 DynQuantLinear + 2 DynQuantEmbedding + "
f"96 DynQuantExpertBank), got {packed}. 0 means dynquant.register_hf_quantizer() "
f"was never called and this model is randomly initialised; any other number "
f"means the module swap was partial."
)
The 48 MoE routers are not in that 602: they are dequantized back to a dense weight at load, by design (see What the quantizer did).
The fine-tune taught one prompt and one output format. The model answers with an index into a
numbered 60-intent menu — an integer such as 37 — not with an intent name and not with a
sentence. The menu is the 60 scenario_action strings sorted lexicographically and numbered from
0, under this header:
Listen to the spoken command and classify it into one of these intents.
Answer with the number only.
0. alarm_query
1. alarm_remove
...
59. weather_query
A generation is scored by stripping any reasoning trace, then taking a leading integer in
[0, 60) if there is one, else the first in-range integer anywhere in the text; anything naming
no index in [0, 60) is unparseable and counts as wrong. Outside this prompt shape nothing here
measures the model at all: both the fine-tune and the bit allocation were made from this one
task, and no general-capability arm was run.
Block order matters — the menu and exemplars come first, then the audio, then a trailing
Intent:. That is what the model was trained and scored on:
MENU = intent_menu(INTENTS) # the 60-line numbered menu, header included; see Results
SHOTS = [f"{ex.transcript} -> {ex.answer}" for ex in four_shots] # TEXT transcripts, not audio
preamble = MENU + "\n\nExamples, as text:\n" + "\n".join(SHOTS)
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": preamble + "\n\nNow the spoken command:"},
{"type": "audio", "audio": clip}, # bare mono float32 ndarray at 16 kHz; the
# {"array", "sampling_rate"} dict raises, and a
# wrong rate does not -- it reads the clip at the
# wrong speed and scores like a weak model
{"type": "text", "text": "Intent:"},
],
}
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
sampling_rate=16000, # passed beside the block, not inside it
).to(model.device)
# `do_sample=False` alone is not enough on transformers 5.x: unset fields are filled from
# the checkpoint's own generation_config, so a shipped repetition_penalty survives and
# silently moves the score. Pin the neutral fields.
out = model.generate(
**inputs,
max_new_tokens=8,
do_sample=False,
num_beams=1,
repetition_penalty=1.0,
length_penalty=1.0,
no_repeat_ngram_size=0,
)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
# e.g. "37" -> INTENTS[37]
Every arm below is the first 500 items of a random.Random(0) shuffle of the SLURP test
split — not the first 500 rows as the mirror serves them, which arrive grouped by speaker
and scenario and would sample one corner of the label space. 4 shots, greedy decoding,
max_new_tokens=8, stop at the first newline, add_special_tokens=False,
max_prompt_tokens=4096, batch size 8, MoE experts dispatch pinned to eager, against the same
60-intent menu (intents_sha
d04b663b407e9f5b5be80c9d11160c391c7b68f516c9da957aaca026138fc86d). Chance is 1.667%. Per-item
hits are stored, so every comparison is an exact McNemar test on paired outcomes rather than
a comparison of two independent proportions.
| arm | what it is | accuracy | vs comparator | discordant | p |
|---|---|---|---|---|---|
omni-base | whole Omni checkpoint, bf16, no fine-tune | 79.40% (397/500) | — | — | — |
omni-sft | Thinker-only, adapter merged, bf16 | 86.80% (434/500) | +7.40 vs base | 47/10 | 7.51e-07 separated |
omni-dq4 | this model, DynQuant 4.00-bit map | 86.20% (431/500) | −0.60 vs sft | 12/15 | 0.7011 not separated |
omni-dq3 | DynQuant 3.00-bit map | 25.00% (125/500) | −61.80 vs sft | 3/312 | 1.56e-88 separated |
95% confidence intervals on the paired differences: base to sft [+4.51, +10.29], sft to dq4
[−2.64, +1.44], sft to dq3 [−66.17, −57.43]. This model returned 0 unparseable generations;
every other arm in this table returned 0 except the 3-bit one, which returned 3.
scenario and action from SLURP's own annotation at
https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl
for all three splits (train, devel, test), form f"{scenario}_{action}", take the
sorted unique set. That is 60 strings; sha256("\n".join(menu)) must equal the digest above.
Deriving it from one split renumbers the taxonomy and every learned index names its
neighbour.marcel-gohsen/slurp, resampled to 16 kHz, joined to the
annotation on slurp_id. Ignore the mirror's own intent column entirely — it is corrupted.random.Random(0).shuffle(examples), then take the first 500 of test.sorted(random.Random(0).sample(range(50628), 4)) over the identically shuffled
train pool — an index sample, not a prefix of the shuffle. Rendered as text, never as
audio: "{transcript} -> {gold_index}" under the line Examples, as text:. Their gold
indices are 27, 15, 34, 22.All of it is dynquant.eval.slurp, and the harness is the reference implementation — it
registers the quantizer itself and raises if a packed checkpoint comes back unpacked:
pip install dynquant "transformers>=5.15,<6" "datasets>=4" torchcodec
dynquant eval VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit \
--task slurp --split test --limit 500 --shots 4 --shot-seed 0 \
--model-class Qwen3OmniMoeThinkerForConditionalGeneration \
--experts-impl eager --device cuda:0 --dtype bfloat16 --out dq4.json
# the bf16 ceiling, and the paired McNemar test against the arm above
dynquant eval VikramPal/Qwen3-Omni-30B-A3B-Thinker-bf16 \
--task slurp --split test --limit 500 --shots 4 --shot-seed 0 \
--model-class Qwen3OmniMoeThinkerForConditionalGeneration \
--experts-impl eager --device cuda:0 --dtype bfloat16 --out sft.json --compare dq4.json
Reproducing it downloads the whole SLURP test split and the whole train split (50,628
recordings, pulled in full to draw four text exemplars) and fetches labels from
raw.githubusercontent.com, so budget the disk and the network.
The correct claim is the interval, not the point estimate. At 4.00x fewer bytes this comparison excludes damage worse than 2.64 points. It does not establish that the damage is zero. 500 items is what buys the resolution, and a 0.60-point loss sits well inside the noise that sample size leaves; a larger evaluation could resolve a real, small regression that this one cannot see.
The packed checkpoint published here was also loaded back and scored against its in-memory counterpart: 0 discordant items out of 500. That is a round-trip test whose predicted outcome is exact identity — the packed path stores the same codes and scales and dequantizes at the same compute dtype, so greedy decoding must emit identical tokens — and not an independent replication. A single discordant item would have meant the packed path diverges from the encoder somewhere. None does, at either width.
DynQuant allocates a per-module bit width under a global average-bits budget, subject to per-role floors, instead of applying one uniform width. At a 4.00-bit target on this architecture the role floors fit with 0.582 bits to spare, so the allocation decides only what gets upgraded, and no floor is violated.
| setting | value |
|---|---|
| group size | 128 |
| scheme | asymmetric (per-group bf16 scale + bf16 offset) |
| layout | linear |
| compute dtype | bfloat16 |
| modules in the bit map | 650 (602 packed + the 48 routers, restored dense at load), covering 31,712,079,872 elements |
| average bits | 3.997249 over quantized elements; 3.999945 over all Thinker parameters |
| floor violations | 0 |
7,125,616 parameters — norms, biases, convolutions — stay bf16. The 48 MoE routers (mlp.gate)
are a third case, and the one most often misstated: they are in the map at 8 bits — they are
the 48 of the 50 8-bit modules in the histogram below — and they are written packed at 8
bits (270,336 B each, inside the packed total rather than beside it). Nothing in the packed
runtime can stand where a router stands, so the loader dequantizes them back into a dense
bfloat16 weight after load, which is why a read-back reports 48 of 48 routers dense and why
the assertion above expects 602 rather than 650. Their values therefore carry 8-bit
quantization error; only their storage at runtime is dense. What the floor policy guarantees is
that a router is never breached: MOE_ROUTER is a structural role, so the tensor that decides
which experts a token passes through keeps its 8 bits at both budgets.
Width histogram:
| width | modules | elements |
|---|---|---|
| 2-bit | 9 | 1,811,939,328 |
| 3-bit | 67 | 5,697,961,984 |
| 4-bit | 524 | 23,877,120,000 |
| 8-bit | 50 | 325,058,560 |
The element-weighted mean of that histogram is 3.7470 bits, not 3.9972. The difference is the group metadata, which the quoted average includes: a bf16 scale plus a bf16 offset every 128 elements is 0.25 bits per element. Add the 0.25 and you land within 0.0003 of the figure above, the remainder being rows whose length does not divide evenly into groups.
Sensitivity pricing made no difference at this budget: re-allocating from measured channel
moments rather than the rank-product proxy changed 0 of 650 module widths at 4.00 bits and
produced an identical byte total. At 3.00 bits it moved 23. Read that null narrowly — the
measured moments exist for only 49 of 654 modules, 0.324 B of 31.719 B parameters (1.02%)
(the 48 routers plus lm_head), because the moment hook's shape guard rejects every module
bitsandbytes replaced with a flat 4-bit weight and expert banks are excluded as 3-D. Both
allocations priced the other 99% of the parameters from the same proxy.
Three distinct measurements, which should not be mixed. The GiB conversion is attached to the level it belongs to:
| weight-tensor bytes | shard-file bytes | directory (du -sb) | |
|---|---|---|---|
| bf16 merged Thinker | 63,438,410,976 (59.08 GiB) | 63,440,876,184 | 63,454,086,373 |
| this model (4-bit) | 15,859,386,016 (14.77 GiB) | 15,859,696,832 | 15,871,835,299 |
| DynQuant 3-bit | 11,894,607,776 (11.08 GiB) | 11,894,918,480 | 11,907,056,499 |
Like-for-like — shard files against shard files — the compression is 4.00x.
Resident VRAM, measured with torch.cuda.memory_allocated across a load on an idle card, is
15,892,454,912 B: 3.99x less than the bf16 merge's 63,440,876,184 B of shard files
(bf16 resident VRAM was not itself measured, so this is not a VRAM-to-VRAM comparison), and
0.21% above the stored payload. That overage is 33,068,896 B, and most of it is the routers,
not metadata: the 48 of them are stored packed at 8 bits (270,336 B each) but restored to dense
bfloat16 at load, costing 524,288 B resident each, so 48 x 253,952 B = 12,189,696 B — 36.9% of
the overage — is that restoration alone. The ~20.9 MB remainder is the runtime's own bookkeeping;
it is not the scales and offsets, which are already counted inside the stored payload.
The saving is real.
This is a packed checkpoint that dequantizes at compute time, not a fake-quant checkpoint
carrying rounded weights at full width. Had any module fallen back to fp16 the residue would be
gigabytes rather than tens of megabytes.
A QLoRA adapter was trained over an NF4 base with peft 0.20.0: r=16, lora_alpha=32,
lora_dropout=0.05, bias="none", targeting q_proj, k_proj, v_proj, o_proj, out_proj,
fc1, fc2.
The MoE expert banks were not adapted. LoRA does not reach batched 3-D expert banks. Under LoRA every base weight is frozen; only rank-16 adapters on 384 projections took a gradient, and the 91.399% of parameters in the 96 expert banks carried no adapter at all. The merge accordingly touched 384 weights and left all 96 expert banks unchanged. Whatever this fine-tune learned, it learned in the attention and dense-MLP projections and in the audio tower.
Run configuration: torchrun, world size 2, effective batch 16, 500 optimizer steps, 4,188.1 s
wall. 8,000 of the 50,628 SLURP training recordings were used, covering 59 distinct intents out
of the 60-class menu. Final train_loss 4.031702354431152; the loss curve ran 5.479 to 3.080.
Hardware and stack: vast.ai, 2x RTX PRO 6000 Blackwell Max-Q (94.97 GiB each, sm_120),
torch 2.11+cu128, transformers 5.15.0, dynquant-core 0.4.0.
SLURP intent classification. The label is the joint scenario_action pair — 60 classes.
Audio comes from the Hub dataset
marcel-gohsen/slurp. Labels come from
SLURP's own annotation files at
https://raw.githubusercontent.com/pswietojanski/slurp/master/dataset/slurp/{split}.jsonl.
Two sources are used because the mirrors' own intent column is corrupted: on 1,548 of 72,396
recordings it disagrees with the scenario/action pair, having dropped the scenario, so nine
different *_query intents all collapse to query. That yields 91/71/77 classes per split
against the real 60. The scenario and action columns are clean, so the joint pair is the label.
The expert banks were never adapted. 91.399% of the parameters — 28,991,029,248 of 31,719,205,488 — went through fine-tuning frozen, because LoRA does not reach batched 3-D expert banks. The quantizer does reach them; the trainer did not.
500 evaluation items bound the resolution. The 4-bit result is "not separated from bf16",
which is a statement about what this test can see. The interval [−2.64, +1.44] excludes damage
worse than 2.64 points and says nothing more than that.
One task, one language. Everything measured here is SLURP intent classification in English, under a fixed 4-shot prompt with the full 60-intent menu in context. There is no evidence in this card about general instruction following, other audio tasks, other languages, or the vision path, and the allocation was made from a signal collected on this one task. Treat any other use as unmeasured.
SLURP is one collection with one recording protocol. Accuracy on it does not transfer unexamined to other microphones and acoustic conditions, telephony audio, unfamiliar accents, or spontaneous speech outside this collection.
No competing quantizers were run. GPTQ, AWQ and RTN were out of scope for this campaign, so there is no matched-bytes comparison against another method — only against the bf16 parent.
The adapter was trained against an NF4 base and merged into bf16, which is standard for QLoRA and means the merged checkpoint is not bit-for-bit the model that produced the allocation signal.
The base-to-SFT row differs in two things, not one: the base screen ran the whole Omni checkpoint while every post-merge arm runs the Thinker alone, and only the post-merge arms carry the merged adapter. The rows carrying this card's actual claim — sft against dq4 — are both Thinker-only and differ in exactly one thing.
No speech output. See the scope section; the Talker was removed before training.
| repo | what it is |
|---|---|
VikramPal/Qwen3-Omni-30B-A3B-Thinker-QLoRA | the QLoRA adapter alone, 112,207,855 B (107 MiB). It was fitted against an NF4 quantization of the Talker-nulled Thinker, so attach it to .thinker and never to the whole Omni model, whose Talker carries matching q_proj/k_proj/v_proj/o_proj names it was never fitted against. If you just want to run the model, take this repo, not the adapter — the adapter is the smallest download and the largest memory requirement |
VikramPal/Qwen3-Omni-30B-A3B-Thinker-bf16 | the merged bf16 Thinker this model was quantized from, 59.08 GiB, 86.80% |
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-4bit | this repo — the 4.00-bit map, 86.20%, 14.77 GiB |
VikramPal/Qwen3-Omni-30B-A3B-Thinker-DynQuant-3bit | the 3.00-bit map, 11.08 GiB, 25.00% — published as a reproduction of a measured collapse, not a usable model |
DynQuant's role floors alone cost 3.418 average bits on this architecture. A 3.00-bit target is
therefore below the floor budget: the soft floors bind, the allocator downgrades by lowest
return on investment, 42.9% of the parameters land at 2 bits, and lm_head is cut from 8 bits
to 3. That arm reports 400 of 650 modules in breach, against 0 here. It measures floor-override
damage; it is not the 4-bit experiment at a lower budget. The 4-bit arm, where the floors fit,
is the one that carries a quantization result.
license: other with license_name: apache-2.0, inherited from
Qwen/Qwen3-Omni-30B-A3B-Instruct;
the base model's terms govern, and nothing here grants rights beyond them. The SLURP dataset is
CC BY 4.0, and its terms apply to any redistribution of the data.
SLURP:
@inproceedings{bastianelli-etal-2020-slurp,
title = {{SLURP}: A Spoken Language Understanding Resource Package},
author = {Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena},
booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2020}
}
DynQuant: https://github.com/kambojvikram/dynquant, version 0.4.0.
Base model: https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct