Downloads · 30 days
49
5% of all-time downloads
Qwen2.5-32B-TQ8
ekovshilovsky/Qwen2.5-32B-TQ8
Qwen2.5-32B-TQ8 is a text generation model from ekovshilovsky. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
TurboQuant-compressed version of Qwen/Qwen2.5-32B for near-lossless inference on Apple Silicon.
All-time downloads
937
Public
Parameters
28.9B
36.7 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors36.7 GB · 100%
Parameter types
How the weights are stored.
U828.9B · 100%
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwen2.5-32B-TQ8
TurboQuant-compressed version of Qwen/Qwen2.5-32B for near-lossless inference on Apple Silicon.
Compressed with turboquant-mlx-core using the TurboQuant algorithm (Zandieh et al., ICLR 2026).
Quality
| Metric | Value |
|---|---|
| fp16 PPL | 1.41 |
| TQ8 PPL | 1.42 |
| PPL delta | 0.09% |
| Compression | 56% of original size |
Quantization Config
| Property | Value |
|---|---|
| Method | TurboQuant 4+4 residual (8 effective bits) |
| Rotation | Dual Walsh-Hadamard (per-pass seeds) |
| Codebooks | Per-layer Lloyd-Max fitted |
| Sensitive layers | First/last 4 at fp16 |
| Block size | Adaptive (largest power-of-2 dividing in_features) |
Usage
# Serve via SwiftLM (dequants to BF16 on first load, cached for subsequent runs)
SwiftLM --model ekovshilovsky/Qwen2.5-32B-TQ8 --port 5413
# Dequant to fp16 for use with any MLX/HuggingFace loader
tq-dequant ./Qwen2.5-32B-TQ8 ./Qwen2.5-32B-fp16
Hardware Requirements
- Apple Silicon Mac (M1 Pro+ recommended)
- 64 GB unified memory minimum (34 GB model + KV cache + overhead)
- macOS 14+
Original Model
This is a quantized version of Qwen/Qwen2.5-32B by Alibaba Cloud. The original model is released under the Apache 2.0 License. All original model terms and conditions apply.
Quantization
Quantization performed by Eugene Kovshilovsky using turboquant-mlx-core (MIT License).
- Task
- Text Generation
- Library
- mlx
- Type
- qwen2
- License
- apache-2.0
- Created
- Apr 16, 2026
- Updated
- Apr 17, 2026