Qwen3.8-4B-Distill-MLX-4bit

Qwen3.8-4B-Distill-MLX-4bit — Q4-class / ~4-bit MLX

MLX-native quantized version of Qwen3.8-4B-Distill, built with oMLX Universal Dynamic Quantization (oQe). Standard mlx-lm safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift.

At a glance: 2.3GB · 65.4 tok/s · 2.684 GB peak on M1 Max · 4-bit oQe

Quantized and benchmarked by RolanDorisTech on an M1 Max —<br> more on YouTube: Local AI Made Easy.

Contents

• Quick Facts
• What are oQ and oQe?
• How to Use
• Family
• Measured Performance
• Architecture & Compatibility
• Base Model Provenance
• Limitations
• Credits
• References

Quick Facts

What are oQ and oQe?

oQ measures each layer's real quantization sensitivity and allocates bits accordingly, instead of a flat uniform budget. oQe adds activation-importance weighting (imatrix) on top.

<details> <summary>Full methodology, benchmarks, and research citations</summary>

Not all weights matter equally. Layers like lm_head, token embeddings, and the first and last transformer blocks are disproportionately sensitive: a small quantization error there can distort the output distribution and degrade reasoning. Other layers tolerate much more aggressive compression with little effect.

Conventional uniform quantization uses a largely fixed precision policy, whereas oQ measures layer sensitivity and allocates a mixed-precision budget accordingly.

oQ replaces that flat budget with a measured one. It runs calibration data through the model, measures each layer's real sensitivity to quantization error, and allocates more bits to layers where error hurts most and fewer bits where it doesn't.

oQe goes further: it adds an activation-importance calibration pass and uses per-channel activation statistics (an imatrix) to weight quantization error, giving greater importance to errors on channels that matter more during calibration.

Why it matters (published oMLX benchmark on Qwen3.5-35B-A3B, MMLU, 300 samples):

The 2-bit uniform result was substantially lower in this test.

Note: This benchmark evaluates oQ on Qwen3.5-35B-A3B; it documents the oQ methodology and is not a quality benchmark for these oQe Qwen3.8 Distill models.

For full methodology see: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md

Related quantization research also shows that sensitivity- and activation-aware weighting can improve weight-only quantization results:

• SqueezeLLM (Kim et al., arXiv:2306.07629) reports LLaMA-7B C4 perplexity of 28.26 for uniform RTN, 18.08 for sensitivity-agnostic non-uniform quantization, and 7.75 for sensitivity-based non-uniform quantization.
• vLLM's LLM Compressor, using its imatrix_mse observer, reduces WikiText-2 perplexity on Llama-3.1-8B (W4A16) from 6.96 to 6.85, and to 6.83 with GPTQ. Docs: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix/

The takeaway: in this specific MMLU test, oQ achieved higher accuracy than uniform quantization at the same nominal bit width. This benchmark is on Qwen3.5-35B-A3B and documents the oQ methodology; it is not a quality benchmark for these Qwen3.8 Distill oQe models.

Output is standard MLX safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift. Runtime compatibility should be verified against the specific application/version.

These Qwen3.8 Distill models in this release use oQe.

</details>

How to Use

Install:

pip install mlx-lm

Or if you use uv:

uv pip install mlx-lm

Generate:

mlx_lm.generate --model RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit --prompt "Explain oQ vs oQe" --max-tokens 250 --temp 0.6 --top-p 0.95 --top-k 20

LM Studio: Search and download RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit

Prompt format uses Qwen3 chat template with <think> tags. chat_template.jinja is included.

Family

All three sizes are community <u>distills</u> of Qwen/Qwen3.8-2.4T-A95B into Qwen3.5 architecture, NOT official Alibaba releases.

Note: This is a <u>community distillation</u>, NOT an official Alibaba/Qwen release.

Measured Performance

Local measurement by RolanDorisTech on M1 Max 64GB; not an upstream Empero or Qwen benchmark.

(Accuracy benchmarks to come — community evaluations like MMLU, GSM8K, HumanEval etc. will be published in a future update when time allows; current numbers are local speed only. Planning to queue overnight in oMLX with extended thinking ON.)

Test prompt: "Q: If 2x + 3 = 11, what is x? Think step by step inside <think> tags." — Result correct x=4, preserved.

For this Qwen3.8-4B-Distill-MLX-4bit model: 37 tokens prompt at 133.7 tok/s, 65.4 tok/s gen, peak 2.684 GB

Architecture & Compatibility

• Architecture: Qwen3.5 hybrid — repeating groups of 3 Gated DeltaNet blocks followed by 1 Gated Attention block. 248,320-token vocabulary (padded).
• Native context: 262,144 tokens inherited from Qwen3.5 base.
• Requirements: recent mlx-lm / transformers with Qwen3.5 + Gated DeltaNet support. For PyTorch/CUDA, needs flash-linear-attention and causal_conv1d kernels — without them, linear-attention layers fall back to slow PyTorch ops.
• Sampler: temperature=0.6, top_p=0.95, top_k=20 (Qwen3.5 recommended). Greedy decoding can cause repetition loops.
• Output: The model is trained to open responses with a <think> block; allow up to 16,384 new tokens and strip the <think>...</think> span when hiding reasoning from end users.

Base Model Provenance

Note: This model is quantized from empero-ai/Qwen3.8-4B-Distill, a community model developed by Empero, not from an official Alibaba/Qwen 4B checkpoint.

• Teacher: Qwen/Qwen3.8-2.4T-A95B teacher
• Student base: Qwen/Qwen3.5-4B
• Training: SFT (off-policy distillation) on ~45,000 teacher traces
• Lineage: Qwen/Qwen3.8-2.4T-A95B → empero-ai/Qwen3.8-4B-Distill → MLX BF16 → oQe quant

For full lineage: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill

The Qwen3.8 open-model releases documented by Qwen include Qwen/Qwen3.8-2.4T-A95B and Qwen/Qwen3.8-27B; this release should not be described as an Alibaba/Qwen "Qwen3.8-4B" checkpoint.

Limitations

Accuracy benchmarks (MMLU, GSM8K, etc.) for these specific released models have not been run yet — only inference speed above is measured. Benchmarking is planned but takes significant compute time per model; results will be added when available.

Distilled reasoning model. May hallucinate. Lossy relative to BF16. Quantization quality depends on model, quantization level, and calibration; no direct same-size accuracy comparison with uniform g32/g64 is claimed here.

Text-trained/fine-tuned; vision behavior is inherited from the Qwen3.5 base and was not evaluated for this release.

Native context: 262,144 tokens inherited from Qwen3.5; practical usable context depends on the inference runtime, KV-cache implementation, and available memory.

Credits

• Base: empero-ai/Qwen3.8-4B-Distill (safetensors) — itself a full-parameter distillation of Qwen/Qwen3.8-2.4T-A95B into Qwen/Qwen3.5-4B, Apache 2.0. GGUF repo empero-ai/Qwen3.8-4B-Distill-GGUF is a separate quantized artifact for llama.cpp, not used as source for this MLX conversion. Released by Empero under Apache-2.0; upstream card states license inherited from base.
• Quantization: oMLX Universal Dynamic Quantization (oQe). Built by RolanDorisTech, September 23, 2026.
• Machine: M1-Max Mac Studio 64GB, 32 Core GPU, macOS 27.0
• Tool: oMLX Quantization panel
• Base conversion: Qwen3.8-4B-Distill BF16 MLX master 7.1GB (converted from empero-ai/Qwen3.8-4B-Distill safetensors, not GGUF)
• Method: oQe ON, Reuse ON, Automatic cache, Strict OFF, Preserve MTP OFF, bfloat16 for norms and scales
• Protection: lm_head 8-bit, embedding and early/late layers boosted per the oQe quantization configuration

References

Methodology:

• oQ docs: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md
• SqueezeLLM paper: https://arxiv.org/abs/2306.07629
• vLLM imatrix docs: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix/

Base model lineage:

• Empero 9B Distill: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill
• Empero 2B Distill: https://huggingface.co/empero-ai/Qwen3.8-2B-Distill
• Empero 4B Distill: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill
• GGUF variant (not used as source): https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF

Official Qwen releases:

• Official Qwen3.8 repo: https://github.com/QwenLM/Qwen3.8
• Qwen3.8-2.4T-A95B: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
• Qwen3.8-27B: https://huggingface.co/Qwen/Qwen3.8-27B
• Qwen3.5-9B base: https://huggingface.co/Qwen/Qwen3.5-9B

More from YouTube series Local AI Made Easy — @RolanDorisTech