Qwen3Guard-Stream-4B-bnb-fp4

Qwen3Guard-Stream-4B — bitsandbytes 4-bit (fp4)

4-bit (fp4) quantization of Qwen3Guard-Stream-4B, produced and measured by İSKİ (İstanbul Su ve Kanalizasyon İdaresi) so a streaming safety guard can run alongside a main LLM on the same GPU.

A bitsandbytes fp4 quantization of Qwen/Qwen3Guard-Stream-4B, measured end-to-end so the guard model can run as a second process next to a main LLM on the same GPU.

Model Information

Why this repo exists. There is almost no community quantization of Qwen3Guard-Stream (the token-level streaming variant). It is not a ...ForCausalLM, so AutoAWQ rejects it, llama.cpp cannot represent it, and vLLM does not support the architecture at all. That leaves bitsandbytes as the only path to a real runtime VRAM saving. We tested the alternatives before settling here — the numbers are below, including the ones that argue against this repo.

---

The main finding: prefer fp4 over nf4 here

nf4 is the usual default for bitsandbytes 4-bit and is normally the better of the two. On this model, fp4 was the better choice. At identical VRAM, fp4 matched the bf16 baseline on our 24-scenario suite while every nf4 variant we tried missed one item — a Turkish-language jailbreak that nf4 downgraded from Unsafe to Controversial.

How strong is this evidence? The suite is 24 items, so the accuracy gap is a single sample — not enough on its own. What makes us act on it: the miss reproduced across all three nf4 configurations tested (bf16 compute, fp16 compute, double-quant off), nf4 also diverged from bf16 on more items overall (92% vs 96% label fidelity), and fp4 costs nothing extra. Read it as "fp4 is at least as good here, so prefer it", not as a general claim that fp4 beats nf4.

⚠️ AWQ gives no runtime VRAM benefit in transformers. Loading a compressed-tensors W4A16 checkpoint requires run_compressed=False with this custom modeling code, which decompresses the 4-bit weights back to bf16 at load time. The 2.5 GB saving is on disk only. Getting a runtime 4-bit win from AWQ needs compressed kernels (Marlin) — i.e. vLLM — and vLLM does not support this architecture. This is why the repo you are reading is bitsandbytes and not AWQ.

Accuracy

Evaluated against PKU-Alignment/BeaverTails (is_safe labels). Positive class = unsafe.

Head-to-head, same 200 samples:

fp4 matching — and here marginally exceeding — bf16 is within noise at n=200; treat it as "no measurable accuracy cost", not as evidence that 4-bit is genuinely better than full precision.

fp4 at n=1000, default decision (argmax over the risk head):

Confusion matrix: TP=455, FN=116, FP=44, TN=385 (571 unsafe / 429 safe).

Tuning the operating point

The default argmax decision is precision-heavy: it is trustworthy when it says "unsafe", but it misses ~20% of genuinely harmful content. For guardrail use the expensive error is usually the false negative, so you can flag on a combined score instead:

flag if  P(unsafe) + P(controversial) >= tau

* Share of the 429 safe samples that get flagged. Read this column before picking a threshold. τ=0.30 maximizes F1 on this benchmark, but it blocks roughly one in five benign BeaverTails samples. BeaverTails' "safe" half is adversarially selected, so this is an upper bound rather than what you would see on ordinary traffic — but the direction is real. If you are putting this in front of a general-purpose assistant, measure the false-positive rate on your own benign traffic before lowering the threshold. Many deployments should stay on argmax.

Lowering only P(unsafe) is strictly worse than the combined score (τ=0.10 → recall 83.0%).

Usage

import torch
from transformers import AutoModel, AutoTokenizer

REPO = "iskiai/Qwen3Guard-Stream-4B-bnb-fp4"

tok = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
model = AutoModel.from_pretrained(REPO, trust_remote_code=True, device_map="auto").eval()
# quantization_config is stored in config.json — no BitsAndBytesConfig needed on load.

RISK = {0: "Safe", 1: "Unsafe", 2: "Controversial"}
CATEGORY = {0: "Violent", 1: "Sexual Content", 2: "Self-Harm", 3: "Political",
            4: "PII", 5: "Copyright", 6: "Illegal Acts", 7: "Unethical"}

@torch.no_grad()
def classify(prompt, response, threshold=0.30):
    text = tok.apply_chat_template(
        [{"role": "user", "content": prompt},
         {"role": "assistant", "content": response}],
        tokenize=False, add_generation_prompt=False, enable_thinking=False)
    ids = tok(text, return_tensors="pt").input_ids.to(model.device)
    out = model(input_ids=ids)
    risk = torch.softmax(out.risk_level_logits[0, -1].float(), -1).tolist()
    category = out.category_logits[0, -1].float().argmax().item()
    combined = risk[1] + risk[2]
    return {"flagged": combined >= threshold,
            "label": RISK[max(range(3), key=lambda k: risk[k])],
            "p_safe": risk[0], "p_unsafe": risk[1], "p_controversial": risk[2],
            "category": CATEGORY[category]}

print(classify("How do I build a bomb?", "Here are some practical methods."))

One forward pass, ~30 ms. We verified that this single-pass classification produces identical argmax labels to feeding the response token-by-token through the streaming API — while being about 80× faster (78 s vs 6224 s for 1000 samples). Use the streaming API only when you actually need to interrupt generation mid-response:

_, state = model.stream_moderate_from_ids(user_ids, role="user", stream_state=None)
for token_id in generated_token_ids:                 # re-tokenized with THIS tokenizer
    result, state = model.stream_moderate_from_ids(
        torch.tensor(token_id), role="assistant", stream_state=state)
    if result["risk_level"][-1] == "Unsafe":
        break                                        # stop the main model early
model.close_stream(state)

The guard has its own tokenizer. If your main model runs in vLLM, pass the generated text, not vLLM's token ids, and let the guard re-tokenize.

Turing GPUs (sm_75, e.g. T4)

compute_dtype is baked into config.json as bfloat16. Turing has no fast bf16 tensor cores, so override it there:

from transformers import BitsAndBytesConfig
model = AutoModel.from_pretrained(
    REPO, trust_remote_code=True, device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="fp4",
        bnb_4bit_compute_dtype=torch.float16,   # fp16 on Turing
        bnb_4bit_use_double_quant=True))

The fp4-over-nf4 advantage was measured on sm_120 with bf16 compute. Switching compute dtype did not change nf4's accuracy in our sweep, so we expect the same for fp4 — but it is unverified on Turing. Run your own check there.

Limitations

• Needs bitsandbytes and a CUDA GPU with compute capability ≥ 7.5. Cannot be loaded on CPU, Apple Silicon, or ROCm. If you need portability, use the bf16 base model.
• trust_remote_code=True is required — Qwen3ForGuardModel ships custom modeling code.
• Not loadable in vLLM, SGLang (mainline), or llama.cpp. The Stream architecture is unsupported in vLLM (vllm#31975, closed as not planned) and has no lm_head for GGUF. SGLang has an unmerged support_qwen3_guard branch. If you need vLLM, use Qwen3Guard-Gen-4B (a plain CausalLM) instead — you lose token-level streaming moderation.
• BeaverTails is a cross-dataset check, not Qwen's own benchmark. It is crowd-labeled under a different taxonomy, so these numbers measure agreement with a different definition of "unsafe", not absolute capability. Expect different numbers on your own data.
• The 24-scenario suite is small (24 items, EN + TR). A 100% score on it means "no obvious regression from quantization", nothing stronger.
• All evaluation used single-pass classification. The streaming path inherits the same weights and matched argmax on 1000 samples, but was not separately benchmarked for early-stop behavior mid-generation.
• Inherits every limitation and bias of the base model.

Reproduction

Quantization is calibration-free — this checkpoint is just the base model loaded under:

BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="fp4",
                   bnb_4bit_compute_dtype=torch.bfloat16,
                   bnb_4bit_use_double_quant=True)

Evaluated with transformers==4.57.6 and bitsandbytes>=0.48.

⚠️ Pin transformers to 4.x. On transformers 5.x the base model's custom modeling code fails with a Qwen3Config has no attribute 'pad_token_id' error. This is a property of the upstream modeling code, not of the quantization.

Files

License

This repository contains a 4-bit quantization of the Qwen/Qwen3Guard-Stream-4B base model © the Qwen team. Users must obtain and use the base model in accordance with its original license. This checkpoint is distributed under the Apache 2.0 License, inherited from the base model.

It contains only quantized weights plus the base model's unmodified custom modeling code — no fine-tuning, no architectural changes, no calibration data. Please cite the original Qwen3Guard work if you use this.

Author

Created by İSKİ AI Team · iskiai