Qwen3.8-27B-mxfp8-mlx

Qwen3.8-27B-mxfp8-mlx

This repository contains Qwen3.8-27B quantized to MXFP8 format using mlx-vlm.

It provides an ideal middle ground for Apple Silicon users who want near-lossless FP16 accuracy with half the memory footprint and significantly faster token generation compared to full-precision weights.

---

⚡ Quickstart & Usage

1. Installation

Ensure you have mlx-vlm installed:

pip install --upgrade mlx-vlm

2. Python Inference

from mlx_vlm import generate, load
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load model and processor
model_id = "zherebetskyy/Qwen3.8-27B-mxfp8-mlx"
model, processor = load(model_id)
config = load_config(model_id)

# Prepare prompt & image input
image = ["[http://images.cocodataset.org/val2017/000000039769.jpg](http://images.cocodataset.org/val2017/000000039769.jpg)"]
prompt = "Describe the contents of this image in detail."

formatted_prompt = apply_chat_template(processor, config, prompt, num_images=1)

# Generate response
output = generate(model, processor, formatted_prompt, image, max_tokens=256)
print(output)

3. Running via CLI

python -m mlx_vlm.generate \
  --model zherebetskyy/Qwen3.8-27B-mxfp8-mlx \
  --image <PATH_TO_IMAGE> \
  --prompt "Describe this image." \
  --max-tokens 200

---

🎯 Why MXFP8 is Superior for Accuracy

When choosing a quantized variant of Qwen3.8-27B, balancing memory, token generation speed, and reasoning accuracy is key. Below is how MXFP8 compares to alternative quantizations:

🔬 Technical Comparison Across Variants

Key Reasons to Choose MXFP8:

1. Microscaling Block Architecture (E8M0 Scaling): Unlike traditional uniform integer quantization (such as standard INT4 or INT8), standard OCP Microscaling (MX) formats divide tensors into fine-grained blocks (typically size 32) and assign a shared 8-bit scale factor (E8M0). This allows sub-block values to adjust scaling dynamic ranges dynamically without truncating high-magnitude features.
2. Exponential Floating-Point Distribution (E4M3 Layout): Language and vision models suffer accuracy loss in low-bit representations because weight distributions concentrate near zero while containing critical outlier values in active channels. MXFP8's E4M3 dynamic range preserves high-energy activation and weight outliers effortlessly.
3. Elimination of Quantization Degradation in Complex Reasoning: While 4-bit formats (4-bit, NVFP4) work well for simple chat, they often experience severe performance degradation on multi-step reasoning, mathematical tasks, structural JSON parsing, and dense visual processing. MXFP8 preserves full-precision quality on benchmark reasoning tasks.
4. Optimized for Modern Apple Silicon Unified Memory: With an approximate RAM requirement of ~28–30 GB, this model fits comfortably on 64GB+ Apple Silicon devices (M1/M2/M3/M4 Max & Ultra), allowing maximum inference speed with virtually no accuracy loss.

---

💻 System Requirements

• Architecture: Apple Silicon (M-Series)
• RAM Required: ~30 GB Unified Memory
• Software: mlx-vlm >= 0.6.13
• Python Version: 3.10 or higher

---

🤝 Acknowledgments & Credits

• Original base model developed by the Qwen Team (Qwen/Qwen3.8-27B).
• Quantized using Apple's open-source MLX framework.