Qwen3.8-27B-RAM-24GB-MLX

Qwen3.8-27B — 24GB (MLX, vision-preserving)

Mixed-precision MLX build of Qwen/Qwen3.8-27B, prepared by baa.ai.

Built at the predicted local (efficiency knee) operating point — the best quality-per-GB point on the RAM budget curve. This is a vision-language model: the full vision tower is retained (quantized at 8-bit) so image understanding is preserved, while the language decoder uses per-tensor mixed precision from RAM sensitivity probing.

Family: this is the efficiency-knee build. A quality-ceiling companion (29GB, 31.61 GB on disk, 9.24 bpw) was also built and scored the same 90.0% on our 100-question MMLU screen — the larger build bought no measurable accuracy, so it was retired on 2026-09-28 and this 24GB build is the recommended choice. For a 16 GB GPU see the 13GB GGUF, which holds a 32K context.

Metrics

Benchmarks

MMLU — 100-question calibrated subset spanning all 57 subjects, enable_thinking=False (direct-answer), identical harness for both models:

The quantized build is statistically indistinguishable from the full BF16 model on this set (a ±1/100 difference is within noise) — i.e. no measurable MMLU degradation at ~2.2× smaller. This is a smoke-test subset, not the full 14k-question MMLU; treat it as a quality-retention check rather than a leaderboard figure.

Agent-safety screen

Quality benchmarks (perplexity, MMLU, data-free fidelity probes) can miss an agentic failure mode where a coherently compressed model invents procedure steps in SOP/agent execution — the finding of Fidelity Is Not Safety (code). This build was screened with that repo's tools:

1. Data-free coherence × rate gate (agent_safety_gate.py) — compares the quantization error dW = W_bf16 − dequant(W_quant) against the BF16 source across the language decoder (400 linear tensors):

VERDICT: PASS — RAM's mixed-precision is a low-coherence operator; the error is neither low-rank nor large, so it does not exhibit the coherent-error signature associated with agentic procedure-invention.

2. Behavioral canary (canary.py) — synthetic SOP/agent execution, 24 SOPs × 3 seeds = 144 paired events, invented_x = cross-procedure step confabulation (lower is safer):

Both pass the paper's reliability gate (recall ≥ 0.8, invented_x ≈ 0). The quantized build invents zero procedure steps — identical to the full model (Δ invented_x = 0.000), confirming the gate's PASS behaviorally: RAM's compression introduced no agentic-safety regression.

Caveat: the gate's thresholds are calibrated on 7–8B dense models and are advisory; this is a 27B hybrid linear-attention VLM (out of the paper's controlled battery), and the screen covers the language decoder, not the vision tower. Treat as a screen, not a certificate.

Recommended inference settings

Qwen3.8 is a reasoning ("thinking") model — it emits a reasoning trace before the final answer.

sampler_params = {
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": 20,
    "max_tokens": 8192,
}

Usage (vision + text)

This is a VLM — load it with mlx-vlm:

pip install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("baa-ai/Qwen3.8-27B-RAM-24GB-MLX")
config = load_config("baa-ai/Qwen3.8-27B-RAM-24GB-MLX")

prompt = apply_chat_template(
    processor, config,
    "Describe this image in detail.",
    num_images=1,
)
out = generate(
    model, processor, prompt,
    image=["/path/to/image.png"],
    max_tokens=512, temperature=0.7, verbose=False,
)
print(out.text if hasattr(out, "text") else out)

Text-only prompts work too — pass num_images=0 and omit image=.

Hardware

• Apple Silicon Mac with ~32 GB unified memory recommended.

How it was built

Quantized with the RAM pipeline: a functional probe measures per-tensor sensitivity (CKA / cosine divergence, SQNR, argmax-flip rate) at each candidate bit width, an MCKP knapsack solver allocates bits under a size budget, and the language decoder is built at the efficiency-knee operating point. The vision tower was held at 8-bit to preserve image understanding.

License

Apache 2.0 — inherited from Qwen/Qwen3.8-27B.

---

Quantized by baa.ai