Qwen3Guard-Stream-8B

VERY IMPORTANT NOTE!

These don't work!

The Qwen3Guard-Stream models are specialised versions of the Qwen3 series, designed specifically for real-time safety and moderation tasks rather than general-purpose language generation.

These models incorporate custom architectural components - such as non-standard attention mechanisms, proprietary gating layers, or unique tokenizers - that deviate significantly from the Transformer-based architectures (like LLaMA, Mistral, or Gemma) that the GGUF format and the llama.cpp runtime were explicitly engineered to support.

Because GGUF is not a universal model serialisation format but rather one tightly coupled to the structural assumptions of models compatible with llama.cpp, models like Qwen3Guard-Stream cannot be directly converted to GGUF, regardless of quantisation level.

The absence of official support in the llama.cpp codebase for Qwen's architecture - especially its attention variants, rotary embedding scheme, and vocabulary layout - means that even a successful file conversion would result in runtime errors or incorrect inference behavior.

This last point is where I ran into trouble - I modified llama.cpp to convert the Qwen3Guard-Gen and SafeRL models, but the Stream models will not work.

The good news though is that you can easily get the Stream version working, and I have provided a working example that you can run on any computer.

Please take a look at [https://github.com/geoffmunn/Qwen3Guard] and especially the chat_demo.html file to see it in action. You will need a python environment and the ability to install some pip modules.

🔍 How It Works

Feed it streaming text, and it returns JSON like:

{"safe": false, "categories": ["hate"], "confidence": 0.85, "partial": true}

Or when safe:

{"safe": true, "categories": [], "confidence": 0.99, "partial": false}

Risk Categories Detected

• violence
• hate
• sexual
• self-harm
• illegal
• spam

Streaming-Specific Fields

• partial: true if input is incomplete
• confidence: confidence score (0.0–1.0)
• Early warning: may flag risk before sentence ends

💡 Why Use This?

Imagine a user starts typing:

"I hate people who are different—"

Even before they finish, Qwen3Guard-Stream-8B detects rising risk and outputs:

{"safe":false,"categories":["hate"],"partial":true,"confidence":0.82}

Your app can:

• Warn the user
• Trigger parental controls
• Pause AI response generation

Perfect for:

• Kids’ apps
• Wearables
• Offline educational tools
• Edge-based moderation

🔗 Relationship to Other Safety Models

Part of a layered safety ecosystem:

Layered Safety Architecture

User Input (Streaming)
    ↓
[Qwen3Guard-Stream-8B] ← fast pre-filter on device
    ↓ (if safe)
[Cloud LLM or Local Generator]
    ↓
Safe Response

💡 Tip: Run this model on-device for instant filtering; forward only safe prompts to cloud.

Review and comparison

I asked the same questions across all 3 models. 0.6B is extremely fast - if you need to run this on a laptop or low-powered server, 0.6B is extremely impressive. However, for grey areas it will be very conservative and will return objectively wrong assessments.

If you have a more powerful server available, there is no real difference between 4B and 8B. However you will need to do a customisation which will accept 'Controversial' categories as being safe.

Both 4B and 8B will offer conservative opinions on ethical gray areas, but they're different so it probably depends on what your use cases are and what your server can accommodate.

You can test this yourself with the demo script I have here: https://github.com/geoffmunn/Qwen3Guard

Author

👤 Geoff Munn (@geoffmunn)
🔗 Hugging Face Profile

Disclaimer

Community conversion for local inference. Not affiliated with Alibaba Cloud.