Qwen2.5-Coder-3B-it-MLX-nvfp4

Qwen2.5-Coder-3B-it-MLX-nvfp4

This repository contains the 4-bit NVFP4 quantized weights for Qwen/Qwen2.5-Coder-3B-it, optimized for low-latency inference on Apple Silicon using the oMLX framework.

Qwen2.5-Coder-3B-it is the ultra-lightweight entry in the Qwen2.5 coding specialist series. Despite its exceptionally compact 3 billion parameter footprint, it inherits the advanced architectural and training enhancements of the broader Qwen2.5-Coder family, making it uniquely suited for fast, edge-based autocomplete, inline code generation, and low-resource deployments.

---

🚀 Efficiency & Performance Advantages

By combining the highly efficient 3B-it parameter base model with a 4-bit NVFP4 quantization mapping, this variant achieves:

• ⚡ Blazing-Fast Generation (TPS): Exceptional token generation and prefill speeds, allowing for near-instantaneous IDE code completions.
• 📉 Minimal Memory Footprint: Extremely small VRAM utilization, freeing up system resources to comfortably run alongside heavy local developer environments.
• ⚙️ Seamless Mac Optimization: Native acceleration when coupled with modern execution layers like oMLX on Apple Silicon.

---

🛠️ Deployment & Execution Quickstart

To utilize this model on macOS, ensure you are running an inference wrapper configured to handle nvfp4 metadata structures.

Running with oMLX

# Execute local evaluation benches natively via terminal:
omlx bench --model your-hf-username/Qwen2.5-Coder-3B-it-MLX-nvfp4 --prompt "Write a Python function to clear a list."

<img src="/bkideas/Qwen2.5-Coder-3B-MLX-nvfp4/resolve/main/benchmark.svg" alt="Benchmark table" width="100%" />