Qwen3.5-35B-A3B-DynQuant-4bit

Qwen3.5-35B-A3B Intent Router (DynQuant 4-bit)

A general intent router: it reads a user turn plus an intent catalog supplied in the prompt and returns exactly one intent id from that catalog. The catalog is an input, not a trained-in label set, so the same checkpoint routes for a catalog it has never seen -- which is what the held-out BANKING77 column below measures.

Fine-tuned from Qwen/Qwen3.5-35B-A3B, then quantized with DynQuant to 4.00 bits/weight (16.140 GiB, 4.00x smaller than bf16).

Read this before you load it

Three things about this checkpoint will produce wrong results silently if you do not know them.

1. It is text-only. The base is a multimodal qwen3_5_moe checkpoint. What is published here is the text tower alone -- model_type is qwen3_5_moe_text, and the vision config and the MTP speculative-decoding head are not present. 34,660,610,688 parameters against the base checkpoint's 35,951,822,704. If you need vision or MTP, use the base model.

2. A packed DynQuant directory requires register_hf_quantizer() before you load it. transformers has no entry-point discovery for quantization methods. Without the call below it does not recognise quant_method in config.json, skips the quantization, and returns a randomly initialised model without raising an exception. There is no traceback and no warning you are likely to see -- the model just generates nonsense. Always:

import dynquant
assert dynquant.register_hf_quantizer()      # must come first, and must be asserted

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "VikramPal/Qwen3.5-35B-A3B-DynQuant-4bit", dtype=torch.bfloat16, device_map={"": "cuda:0"},
    trust_remote_code=True,
    experts_implementation="eager",   # grouped_mm needs sm_90; drop this on Hopper+
)
tok = AutoTokenizer.from_pretrained("VikramPal/Qwen3.5-35B-A3B-DynQuant-4bit")

To check the load actually took, count modules holding packed buffers -- not packed_module_names(), which reports 352 of 472 on this architecture because it does not count the 80 batched expert banks or the 40 routers:

import json
priced = set(json.load(open("dynquant_manifest.json"))["layers"])

def at(name):                      # priced names are modules; banks resolve via the parent
    try:
        return model.get_submodule(name)
    except AttributeError:
        parent, _, leaf = name.rpartition(".")
        return getattr(model.get_submodule(parent), leaf, None)

live = sum(1 for n in priced if getattr(at(n), "qweight", None) is not None)
assert live == 472, "%d of 472 -- this model is NOT packed" % live

3. vLLM cannot serve this checkpoint. Two independent blockers: vLLM refuses fused MoE weights at layer 0, before any bit width is read, and embed_tokens is packed -- vLLM builds its input embedding without consulting a quantization config at all. Use transformers.

How to prompt it

The system prompt carries the catalog; the user turn carries the utterance. The model was trained on a fixed system template (published in this repo as system_template.txt) and renders through the chat template with enable_thinking=False. Scoring at eval time used add_special_tokens=False because the template already emits them -- passing both prepends a second BOS and measures a different model.

messages = [
    {"role": "system",  "content": SYSTEM_TEMPLATE.format(catalog="\n".join(intent_ids))},
    {"role": "user",    "content": "my card hasn't turned up yet"},
]
text = tok.apply_chat_template(messages, tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=24, do_sample=False)
print(tok.decode(out[0, enc.input_ids.shape[1]:], skip_special_tokens=True))
# -> card_arrival

Greedy decoding, and 24 new tokens is enough for every label in these catalogs. The router answers with a bare intent id, or out_of_scope when the catalog does not cover the turn, or a clarifying question when the turn conjoins two intents.

Results

1,500 held-out items, 150 per (split, catalog) group, greedy decoding, strict exact match against the gold intent id.

Invented ids are answers that are not in the catalog the model was given -- the failure mode that makes a router unusable downstream, because the caller has no branch for them.

Strict requires the reply to be exactly the intent id and nothing else; right id present also accepts the id surrounded by other text. The two are identical for Qwen3.5-35B-A3B (no fine-tune), this router, bf16, this router, DynQuant 4-bit, so those strict numbers are accuracy outright. They separate on this router, DynQuant 3-bit (22 of 1500 items), which reached the correct id and then kept writing -- a note, a caveat, or a markdown table. Whether that counts as a loss is the caller's choice: a router that reads the first line of the reply recovers those items, one that requires a bare label does not. Both numbers are given so neither reading has to be taken on trust.

Paired McNemar against the un-quantized bf16 router on the same 1,500 items (22 items where only that arm was right, 15 where only this one was): -0.47 points, exact binomial p = 0.324. The test is paired because both arms answer the identical items; the ~1463 items they agree on carry no information about which is better.

By catalog

banking77 is never seen in training -- neither its utterances nor its 77 intent ids. It is the column that says whether this is a router or a 77-way classifier wearing one.

By item kind

By language

Quantization

Allocated by DynQuant 0.5.0: a per-module bit width chosen from a gradient-variance (plasticity) signal harvested during the fine-tune itself, spent under a global byte budget by a greedy ROI knapsack.

VRAM is genuinely reduced: the weights stay packed in memory and are not materialised to bf16 at load. Measured resident on the eval run that produced the numbers above -- torch.cuda.memory_allocated immediately after from_pretrained -- 16.179 GiB, against 64.56 GiB for the bf16 arm (3.99x).

Structural floors

Every module sits at or above the minimum width DynQuant assigns its role: 0 floor violations. The budget pays for the architecture's own requirements with room to spare, so what follows measures allocation inside a feasible budget rather than the cost of overriding one.

What the signal is worth

The honest control for a bit allocator is not fp16 -- it is the same widths assigned to the wrong modules. This arm ships with a within-role shuffled control allocated under the same policy at the same budget: identical width histogram, identical byte count, 53 of 472 modules (16.6% of quantized parameters) receiving a different width.

This is a null result, and it is reported as one. At this budget the signal-driven assignment and a shuffled one are not distinguishable: +0.47 points with p = 0.281, on 31 items where the two arms disagree at all (19 that only this arm gets right, 12 only the control). The widths are landing where the role floors and the byte budget put them, and the measured signal is not demonstrably doing the choosing. Two things make that unsurprising on this model rather than a defect: the expert banks are 92.9% of everything being quantized and share one score per layer, so most of the mass has far fewer independent decisions than it has parameters; and the floors already pin the modules a router most depends on. What the null does establish is that this checkpoint's accuracy is a property of the budget and the policy, reproducible without trusting the signal -- which is the more useful claim for anyone deciding whether to run it.

Training

LoRA (r=32, alpha=64, dropout=0.05) on bf16 base weights -- not QLoRA; the base is not quantized during training, so the harvested gradient signal describes the tensors that are actually quantized afterwards. Adapters on every attention and MLP projection including the linear-attention in_proj_* family, merged into the base before quantization.

Training data is assembled from five public intent datasets, each turned into catalog-in-prompt form and augmented with the rule behaviours the router has to get right (refusing to guess when the intent was removed from the catalog, asking when two intents are conjoined, staying silent about intents that are not offered).

BANKING77 appears only in the test split -- it is the generalization measurement, and training on it would destroy the only evidence that the catalog is really an input.

Limitations

• Text only. No vision tower, no MTP head (see the top of this card).
• Not servable by vLLM. Two independent blockers: vLLM's fused-MoE guard, and a packed embed_tokens that vLLM never consults a quantizer about.
• Six languages. en/de/es/fr/hi/th. Other languages are untested and the catalog ids are English regardless of the utterance language.
• Catalog size. Tested at 60-151 intents. Much larger catalogs will not fit the context the same way and are unmeasured here.
• One intent per turn. Conjoined intents are trained to produce a clarifying question, not two labels.
• experts_implementation="eager" is needed below sm_90. The default grouped-MoE path requires Hopper or newer.

Citation

The quantization method:

@misc{dynquant2026,
  title  = {DynQuant: Dynamic-Signal Quantization for Extreme LLM Compression},
  author = {Pal, Vikram and Kour, Manpreet},
  year   = {2026}
}

The base model is Qwen/Qwen3.5-35B-A3B; please cite Qwen as well, and the five source datasets listed above.