Qwen3.5-35B-A3B-DynQuant-3bit

Qwen3.5-35B-A3B Intent Router (DynQuant 3-bit)

A general intent router: it reads a user turn plus an intent catalog supplied in the prompt and returns exactly one intent id from that catalog. The catalog is an input, not a trained-in label set, so the same checkpoint routes for a catalog it has never seen -- which is what the held-out BANKING77 column below measures.

Fine-tuned from Qwen/Qwen3.5-35B-A3B, then quantized with DynQuant to 3.00 bits/weight (12.105 GiB, 5.33x smaller than bf16).

Read this before you load it

Three things about this checkpoint will produce wrong results silently if you do not know them.

1. It is text-only. The base is a multimodal qwen3_5_moe checkpoint. What is published here is the text tower alone -- model_type is qwen3_5_moe_text, and the vision config and the MTP speculative-decoding head are not present. 34,660,610,688 parameters against the base checkpoint's 35,951,822,704. If you need vision or MTP, use the base model.

2. A packed DynQuant directory requires register_hf_quantizer() before you load it. transformers has no entry-point discovery for quantization methods. Without the call below it does not recognise quant_method in config.json, skips the quantization, and returns a randomly initialised model without raising an exception. There is no traceback and no warning you are likely to see -- the model just generates nonsense. Always:

import dynquant
assert dynquant.register_hf_quantizer()      # must come first, and must be asserted

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "VikramPal/Qwen3.5-35B-A3B-DynQuant-3bit", dtype=torch.bfloat16, device_map={"": "cuda:0"},
    trust_remote_code=True,
    experts_implementation="eager",   # grouped_mm needs sm_90; drop this on Hopper+
)
tok = AutoTokenizer.from_pretrained("VikramPal/Qwen3.5-35B-A3B-DynQuant-3bit")

To check the load actually took, count modules holding packed buffers -- not packed_module_names(), which reports 352 of 472 on this architecture because it does not count the 80 batched expert banks or the 40 routers:

import json
priced = set(json.load(open("dynquant_manifest.json"))["layers"])

def at(name):                      # priced names are modules; banks resolve via the parent
    try:
        return model.get_submodule(name)
    except AttributeError:
        parent, _, leaf = name.rpartition(".")
        return getattr(model.get_submodule(parent), leaf, None)

live = sum(1 for n in priced if getattr(at(n), "qweight", None) is not None)
assert live == 472, "%d of 472 -- this model is NOT packed" % live

3. vLLM cannot serve this checkpoint. Two independent blockers: vLLM refuses fused MoE weights at layer 0, before any bit width is read, and embed_tokens is packed -- vLLM builds its input embedding without consulting a quantization config at all. Use transformers.

How to prompt it

The system prompt carries the catalog; the user turn carries the utterance. The model was trained on a fixed system template (published in this repo as system_template.txt) and renders through the chat template with enable_thinking=False. Scoring at eval time used add_special_tokens=False because the template already emits them -- passing both prepends a second BOS and measures a different model.

messages = [
    {"role": "system",  "content": SYSTEM_TEMPLATE.format(catalog="\n".join(intent_ids))},
    {"role": "user",    "content": "my card hasn't turned up yet"},
]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=24, do_sample=False)
print(tok.decode(out[0, enc.input_ids.shape[1]:], skip_special_tokens=True))
# -> card_arrival

Greedy decoding, and 24 new tokens is enough for every label in these catalogs. The router answers with a bare intent id, or out_of_scope when the catalog does not cover the turn, or a clarifying question when the turn conjoins two intents.

Results

1,500 held-out items, 150 per (split, catalog) group, greedy decoding, strict exact match against the gold intent id.

Invented ids are answers that are not in the catalog the model was given -- the failure mode that makes a router unusable downstream, because the caller has no branch for them.

Strict requires the reply to be exactly the intent id and nothing else; right id present also accepts the id surrounded by other text. The two are identical for Qwen3.5-35B-A3B (no fine-tune), this router, bf16, this router, DynQuant 4-bit, so those strict numbers are accuracy outright. They separate on this router, DynQuant 3-bit (22 of 1500 items), which reached the correct id and then kept writing -- a note, a caveat, or a markdown table. Whether that counts as a loss is the caller's choice: a router that reads the first line of the reply recovers those items, one that requires a bare label does not. Both numbers are given so neither reading has to be taken on trust.

Paired McNemar against the un-quantized bf16 router on the same 1,500 items (199 items where only that arm was right, 20 where only this one was): -11.93 points, exact binomial p = 2.85e-38. The test is paired because both arms answer the identical items; the ~1281 items they agree on carry no information about which is better.

The direction of the truncation matters here and is worth stating plainly: it is this arm that ran into the 24-token cap, on 41 of 1500 items, while the un-quantized bf16 router never did. So the gap above is not an artifact of the cap being unfair to the reference -- if anything the cap is what this arm's damage looks like. Restricted to the 1459 items this arm did not truncate, the gap is -9.80 points (p = 4.61e-29). Scoring every arm by whether the right id appears anywhere in the reply -- the most generous reading available, and the one that forgives a model for explaining itself -- this arm reaches 80.6% against 91.1%, a gap of -10.47 points. Quantization at this width does not merely move answers, it changes how the model ends them.

By catalog

banking77 is never seen in training -- neither its utterances nor its 77 intent ids. It is the column that says whether this is a router or a 77-way classifier wearing one.

By item kind

By language

Quantization

Allocated by DynQuant 0.5.0: a per-module bit width chosen from a gradient-variance (plasticity) signal harvested during the fine-tune itself, spent under a global byte budget by a greedy ROI knapsack.

VRAM is genuinely reduced: the weights stay packed in memory and are not materialised to bf16 at load. Measured resident on the eval run that produced the numbers above -- torch.cuda.memory_allocated immediately after from_pretrained -- 12.144 GiB, against 64.56 GiB for the bf16 arm (5.32x).

At this budget lm_head and model.embed_tokens are pinned to their floors (8-bit and 4-bit). DynQuant leaves both breachable by default, which is correct arithmetic for a tied model where the head can be a quarter of all parameters -- here the model is untied and the head is 1.47%, so pinning it costs almost nothing and prevents the allocator from paying for expert bits with output-vocabulary precision.

Structural floors

This target is below the model's own floor budget, and that is the most important thing on this card. DynQuant assigns every role a minimum width; honouring all of them on this architecture costs 3.44 average bits. This arm was allocated at 3.00, a shortfall of 0.44 bits, so the allocator had to breach floors to fit: 191 of 472 modules, holding 55.9% of all quantized parameters, were given less than their role's minimum.

Read this checkpoint as what a 3.00-bit budget does to this model when its architecture wanted 3.44, not as a clean 3.00-bit allocation. The accuracy above is a true measurement of this artifact, but attributing the loss to "3-bit quantization" in general would be wrong: most of it is the price of overriding floors on the paths that were breached, and a model whose floors fit inside the same budget would not pay it. It also caps what the shuffled control below can possibly show: with 56% of the quantized mass under floor, the breach dominates and which module the signal picked has little room left to matter.

What the signal is worth

The honest control for a bit allocator is not fp16 -- it is the same widths assigned to the wrong modules. This arm ships with a within-role shuffled control allocated under the same policy at the same budget: identical width histogram, identical byte count, 155 of 472 modules (40.9% of quantized parameters) receiving a different width.

This is a null result, and it is reported as one. At this budget the signal-driven assignment and a shuffled one are not distinguishable: +1.13 points with p = 0.314, on 253 items where the two arms disagree at all (135 that only this arm gets right, 118 only the control). The widths are landing where the role floors and the byte budget put them, and the measured signal is not demonstrably doing the choosing. Two things make that unsurprising on this model rather than a defect: the expert banks are 92.9% of everything being quantized and share one score per layer, so most of the mass has far fewer independent decisions than it has parameters; and the floors already pin the modules a router most depends on. What the null does establish is that this checkpoint's accuracy is a property of the budget and the policy, reproducible without trusting the signal -- which is the more useful claim for anyone deciding whether to run it.

How it fails

The errors are not the kind an accuracy figure implies. This arm returns an id that is not in the catalog it was given on 54 of 1500 items (3.6%), against 4 for the bf16 router -- and 29 of those 54 are a garbled spelling of the exact label the item wanted, not a different label. The most common:

That matters more downstream than the accuracy delta does: a caller that switches on the returned id has a branch for every catalog entry and none for top_up_reversed. Validate this checkpoint's output against the catalog before dispatching on it.

The damage is concentrated rather than spread: cs_oos (25 items) falls to 48.0% here against 92.0% for the bf16 router. Its own shuffled control scores 80.0% on that same slice -- better than the signal-driven allocation by 32.0 points. The overall comparison between the two is a null, so one slice running the other way is well inside what that null permits; it is reported rather than dropped, because dropping it is how a control ends up only ever agreeing.

Training

LoRA (r=32, alpha=64, dropout=0.05) on bf16 base weights -- not QLoRA; the base is not quantized during training, so the harvested gradient signal describes the tensors that are actually quantized afterwards. Adapters on every attention and MLP projection including the linear-attention in_proj_* family, merged into the base before quantization.

Training data is assembled from five public intent datasets, each turned into catalog-in-prompt form and augmented with the rule behaviours the router has to get right (refusing to guess when the intent was removed from the catalog, asking when two intents are conjoined, staying silent about intents that are not offered).

BANKING77 appears only in the test split -- it is the generalization measurement, and training on it would destroy the only evidence that the catalog is really an input.

Limitations

• Text only. No vision tower, no MTP head (see the top of this card).
• Not servable by vLLM. Two independent blockers: vLLM's fused-MoE guard, and a packed embed_tokens that vLLM never consults a quantizer about.
• Six languages. en/de/es/fr/hi/th. Other languages are untested and the catalog ids are English regardless of the utterance language.
• Catalog size. Tested at 60-151 intents. Much larger catalogs will not fit the context the same way and are unmeasured here.
• One intent per turn. Conjoined intents are trained to produce a clarifying question, not two labels.
• experts_implementation="eager" is needed below sm_90. The default grouped-MoE path requires Hopper or newer.

Citation

The quantization method:

@misc{dynquant2026,
  title  = {DynQuant: Dynamic-Signal Quantization for Extreme LLM Compression},
  author = {Pal, Vikram and Kour, Manpreet},
  year   = {2026}
}

The base model is Qwen/Qwen3.5-35B-A3B; please cite Qwen as well, and the five source datasets listed above.