Downloads · 30 days
160
16% of all-time downloads
notnaurw/Qwen3.8-27B-oQ5e-SAGE-mtp
Qwen3.8-27B-oQ5e-SAGE-mtp is a image-text-to-text model from notnaurw. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as apache-2.0.
<h1 align="center"Qwen3.8-27B — oQ5e / SAGE (MLX, 5-bit, MTP preserved, bf16)</h1 <p align="center" <img src="https://img.shields.io/badge/Qwen-Qwen3.8-6E56CF?style=for-the-badge&logo=qwen&logoColor=white" alt="Qwen Q…
Downloads · 30 days
160
16% of all-time downloads
All-time downloads
1K
Public
Parameters
27.8B
31.7 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors21.1 GB · 100%
How the weights are stored.
U3226.9B · 97%
From the Hugging Face model README
A mixed-precision 5-bit MLX quantization of Qwen/Qwen3.8-27B, intended for generalist fidelity with the original vision architecture preserved, and to be served with Lightning MTP enabled in oMLX.
SAGE (Streamed Activation-guided Grid-allocated Error-compensation) establishes the fidelity-first SAGE quantization pipeline around one question:
Given a constrained mixed-precision budget, how much of the bf16 source model's behavior can be preserved by measuring where quantization causes the most error, rather than distributing precision uniformly?
The premise is straightforward: not every tensor responds to reduced precision in the same way, and not every quantization error has the same effect on the model's behavior. A uniform precision layout treats those differences largely alike. SAGE instead measures them, using the available budget to preserve the parts of the source model that are most sensitive to quantization.
SAGE builds on oMLX's oQe backend with a multi-stage, activation-guided post-training quantization pipeline. It combines calibration, sensitivity-aware constrained bit-budget grid allocation, and full layer-wise second-order error compensation on the realized quantized model, matched to the model's native dimensions. These stages address different parts of the same problem: reducing quantization error, directing additional precision where it contributes most, and limiting unnecessary changes to the precision layout, group sizes, and final checkpoint size.
The objective is not simply to produce a smaller model, nor to create capabilities absent from the bf16 source. It is to preserve the source model's existing behavior as faithfully as possible within a practical precision budget, using measured sensitivity rather than uniform allocation as the basis for those decisions.
Vision is preserved, along with the bf16 MTP head. The result is a multimodal checkpoint that is designed to track the bf16 source closely on <u>generalist text</u> while retaining the original architecture and accepting the compromises inherent to reduced precision. SAGE also establishes the measurement and allocation framework that would inform subsequent experiments across the lineage, including investigations into calibration distributions, architectural pruning, and precision-budget allocation.
SAGE remains the generalist reference point for the lineage: SAGE-ARCS investigates how the calibration distribution redirects fidelity toward agentic workloads, while SAGE-PATHS explores how unused architectural capacity can be converted into additional text-model precision. SAGE establishes the foundation against which those different choices can be understood, rather than prescribing a single allocation strategy for every workload.
21.13 GB (19.68 GiB) · 6.0836 bpw · MTP head preserved at bf16 (15 parameter tensors).
<table class="sage-table" style="display:table;width:100%;table-layout:fixed;"> <thead> <tr> <th class="lead" style="width:26%;">SAGE5 <code>oQ5e</code> Collection</th> <th>bpw</th> <th>size</th> <th>purpose</th> <th>vision</th> <th>mtp</th> </tr> </thead> <tbody> <tr class="ship"><td class="lead"><strong><a href="https://huggingface.co/notnaurw/Qwen3.8-27B-oQ5e-SAGE-mtp">SAGE</a></strong></td><td><strong>6.0836</strong></td><td><strong>21.13 GB</strong></td><td><strong>Generalist · Multimodal</strong></td><td><strong>Preserved</strong></td><td><strong>Preserved</strong></td></tr> <tr><td class="lead"><a href="https://huggingface.co/notnaurw/Qwen3.8-27B-oQ5e-SAGE-ARCS-mtp">SAGE-ARCS</a></td><td>6.0836</td><td>21.13 GB</td><td>Agentic · Multimodal</td><td>Preserved</td><td>Preserved</td></tr> <tr><td class="lead"><a href="https://huggingface.co/notnaurw/Qwen3.8-27B-oQ5e-SAGE-PATHS-mtp">SAGE-PATHS</a></td><td>6.1366</td><td>20.96 GB</td><td>Agentic · Text-only</td><td><em>Pruned</em></td><td>Preserved</td></tr> <tr><td class="lead"><a href="https://huggingface.co/notnaurw/Qwen3.8-27B-oQ5e-SAGE-CREST-mtp">SAGE-CREST</a></td><td>6.5879</td><td>22.50 GB</td><td>Agentic · Text-only</td><td><em>Pruned</em></td><td>Preserved</td></tr> </tbody> </table>[!TIP] Four checkpoints, four distinct research questions: SAGE: given a fixed mixed-precision budget, how much bf16 fidelity can be recovered? SAGE-ARCS: holding the architecture and budget fixed, how does calibration distribution change where fidelity is preserved? SAGE-PATHS: holding the targeted workload fixed, what happens when unused modality capacity is removed and its budget reinvested into text-model precision? SAGE-CREST: holding the targeted workload and the pruned architecture fixed, can targeted precision beat a uniform 6-bit build while staying smaller than it?
Against a vanilla oQ5e produced by oMLX directly, the first couple of rows are all the same 5.8429 bpw, byte counts 164 apart:
Head-KL and Top-1 agreement improve consistently at every stage: head-KL decreases while Top-1 agreement increases. The two metrics measure different aspects of fidelity — head-KL tracks how closely the quantized model reproduces the source probability distribution, while Top-1 measures how often both models choose the same next token.
</details>The byte-neutral stages reduces divergence from the bf16 source by 21.0% with zero additional bytes — paired t = −6.52, p = 2.8e-07 over 32 held-out batches. Bit-widths, group sizes, and byte count remain unchanged through these stages.
The final SAGE allocation increases the trunk from 5.8429 to 5.9429 bpw (+0.32 GiB) and reduces head-KL to 31.4% below the stock baseline (paired t = −9.74, p = 6.0e-11), while Top-1 agreement increases to 0.9523.
That additional tenth of a bit removes 0.00158 nats of divergence, compared with 0.00317 removed by the three preceding byte-neutral stages combined. In other words, the final allocation provides a substantial additional fidelity gain for a relatively small increase in size.
The target was not to produce a conventional 6-bit model, but to maximize fidelity to the bf16 source at roughly 6 bpw. The SAGE trunk lands at 5.9429 bpw; restoring the MTP head to bf16 brings the shipped model to 6.0836 bpw. The trunk therefore remains a 5-bit base with sensitivity-selected higher-precision overrides.
These are all of them, same 32 batches / 130,944 tokens, same reference cache:
<table class="sage-table dense" style="display:table;width:100%;table-layout:fixed;"> <thead> <tr> <th class="lead" style="width:26%;">rung</th> <th>bpw</th> <th>head-KL ↓</th> <th>top-1 ↑</th> <th>top-5 ↑</th> <th>ΔCE ↓</th> <th>ppl ratio</th> <th>bf16 mass in its top-1024 (same for every rung)</th> </tr> </thead> <tbody> <tr><td class="lead">stock oMLX <code>oQ5e</code></td><td>5.8429</td><td>0.01509</td><td>0.9360</td><td>0.9993</td><td>+0.00754</td><td>1.0076</td><td>0.9809</td></tr> <tr><td class="lead"><strong>S</strong>treamed, <strong>A</strong>ctivation-guided calibration</td><td>5.8429</td><td>0.01462</td><td>0.9366</td><td>0.9993</td><td>+0.00642</td><td>1.0064</td><td>0.9809</td></tr> <tr><td class="lead"><strong>G</strong>rid allocation</td><td>5.8429</td><td>0.01375</td><td>0.9389</td><td>0.9993</td><td><strong>+0.00254</strong></td><td>1.0025</td><td>0.9809</td></tr> <tr><td class="lead"><strong>E</strong>rror compensation</td><td>5.8429</td><td>0.01193</td><td>0.9429</td><td>0.9994</td><td>+0.00374</td><td>1.0037</td><td>0.9809</td></tr> <tr><td class="lead"><strong>SAGE</strong> pipeline output</td><td><strong>5.9429</strong></td><td><strong>0.01035</strong></td><td><strong>0.9523</strong></td><td>0.9994</td><td><strong>+0.00203</strong></td><td><strong>1.0020</strong></td><td>0.9809</td></tr> <tr class="ship"><td class="lead">bf16 MTP head <strong>grafted</strong></td><td><strong>6.0836</strong></td><td><strong>0.01035</strong></td><td><strong>0.9523</strong></td><td>0.9994</td><td><strong>+0.00203</strong></td><td><strong>1.0020</strong></td><td>0.9809</td></tr> </tbody> </table>ΔCE is the change in cross-entropy against the bf16 source, in nats per token; ppl ratio is the same quantity as a perplexity multiplier. Both are absolute measures of how much worse the model predicts real text, where head-KL and top-1 measure agreement with the source distribution.
Final build against stock, paired on the same 32 batches, exact t(31):
<table class="sage-table" style="display:table;width:100%;table-layout:fixed;"> <thead> <tr> <th class="lead" style="width:26%;">metric</th> <th class="delta">Δ vs stock</th> <th>t(31)</th> <th>p</th> <th>batches favouring</th> </tr> </thead> <tbody> <tr><td class="lead">head-KL</td><td class="delta">−0.00475</td><td>−9.74</td><td>6.1e-11</td><td>31/32</td></tr> <tr><td class="lead">top-1</td><td class="delta">+0.01630</td><td>+24.44</td><td>8.0e-22</td><td>32/32</td></tr> <tr><td class="lead">ΔCE</td><td class="delta">−0.00550</td><td>−3.27</td><td>2.7e-03</td><td>25/32</td></tr> <tr><td class="lead">top-5</td><td class="delta">+0.00010</td><td>+1.78</td><td>0.085</td><td>17/32 — <strong>not significant</strong></td></tr> </tbody> </table>Two things this table is meant to be transparent above:
"head-KL" is KL(bf16 ‖ quantized) over the bf16 model's top-1024 logits, renormalised on both sides. Those 1024 entries are 0.41% of the 248,320-token vocabulary but hold 98.1% of the reference's probability mass (median 99.6%, 1st percentile 81.5%). So it measures how closely this build reproduces the part of the source distribution that carries the probability, and is blind to the remaining ~1.9% tail.
Every row above was scored against one common reference cache built from the bf16 checkpoint, under identical settings:
<table class="sage-table kv" style="display:table;width:100%;table-layout:fixed;"> <colgroup><col style="width:34%;"></colgroup> <tbody> <tr><td class="lead">corpus</td><td><code>oqe_code_multilingual</code>, held-out tail</td></tr> <tr><td class="lead">samples × sequence</td><td>128 × 1024 tokens</td></tr> <tr><td class="lead">batch size</td><td>4 (32 batches)</td></tr> <tr><td class="lead">top-k</td><td>1024</td></tr> <tr><td class="lead">seed</td><td>123</td></tr> </tbody> </table>head-KL renormalises both distributions inside the top-1024, which cancels any change in how much probability that window holds longitudinally. To find out what that hid, the full 248,320-entry reference distribution was stored (fp32, 61 GB, 64 × 1024 tokens) and the divergence computed exactly, then split into its three parts.
The audit was run on the shipped build and, for comparison, on the 5.8429 bpw error-compensation rung and on stock. Its instrument control computes the head terms two ways — from the stored top-1024 logits and from the full log-probabilities — and requires them to agree: identity_gap OK at 2.9e-05 nats/token against a 1e-04 threshold, so the tail column below is a measurement and not storage noise.
Nats per token, each a per-token mean, so head + mass-shift + tail = KL_full. The last three columns compare each build against the stock 5-bit baseline in the first row.
For the shipped build the first term splits exactly into head 0.010924 and mass-shift 0.000502 (re-scored 2026-09-24; the stock and 5.8429 bpw builds are no longer on disk, so their split is not re-measured).
The full-vocabulary result closely matches the top-1024 result. The shipped build improves by 29.78% under the truncated metric and 29.34% over the full 248,320-token vocabulary, a difference of only 0.44 percentage points. Most of the improvement is concentrated where the probability actually is, and the truncated metric can therefore be trusted to rank.
Comparisons are valid within each table, never across.
Reading the decomposition:
head is 94.9% of the total on the shipped build. The top-1024 is 0.41% of the vocabulary but carries almost all of the divergence, because the bf16 source places 98.1% of its probability there.mass-shift is 4.4% (0.000502) — the quantized model holds slightly less probability inside the source's top-1024 than the source does. Real and one-directional: positive in 32 of 32 batches, paired t(31) = +4.12.tail is 0.78% — the remaining 247,296 entries contribute +0.000090 nats. Small, but not zero, and larger in relative terms than on either comparison build (stock 0.35%, earlier rung −0.01%).So head-KL understates true divergence by 2.82% on the shipped build (0.011191 against 0.011516); the gap is the mass shift and the tail. The three terms sum to the directly computed full-vocabulary KL to within 1.6e-09 nats/token.
max |per-batch diff| = 0.000e+00 over 32 batches). MTP head does not participate in the scored forward path. The head's precision is justified by draft acceptance and decode throughput.This build is made for oMLX: it was quantized through oMLX's oQe backend and is meant to be served by it, with MTP on.
Install oMLX with Homebrew, or see the oMLX repository for the other ways to install and run it:
brew install jundot/omlx/omlx
omlx serve discovers models as sub-directories of --model-dir (default ~/.omlx/models); the directory name becomes the model id:
hf download notnaurw/Qwen3.8-27B-oQ5e-SAGE-mtp --local-dir ~/.omlx/models/Qwen3.8-27B-oQ5e-SAGE-mtp
omlx serve --model-dir ~/.omlx/models --port 8000
Then open the oMLX admin dashboard (http://127.0.0.1:8000/admin), open Qwen3.8-27B-oQ5e-SAGE-mtp's model settings and turn on Lightning MTP.
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen3.8-27B-oQ5e-SAGE-mtp", "messages": [{"role": "user", "content": "Hello"}]}'
Loads under a standard mlx-lm / mlx-vlm stack, like any oMLX oQe build:
pip install mlx-vlm
python -m mlx_vlm.generate --model <this-repo> --max-tokens 256 --prompt "Hello"
Base model © the Qwen team, Apache-2.0 — see LICENSE. Quantization performed with oMLX 0.6.3. SAGE is an adapter layer over oMLX's oQe backend.
Qwen Team. Qwen3.8-Max: A New Bar for Coding and Cowork. August 2026. https://qwen.ai/blog?id=qwen3.8