Apriel-1.5-15b-Thinker-3bit-MLX

Apriel-1.5-15B-Thinker — MLX 3-bit (Apple Silicon)

Format: MLX (Mac, Apple Silicon)
Quantization: 3-bit (balanced footprint ↔ quality)
Base: ServiceNow-AI/Apriel-1.5-15B-Thinker
Architecture: Pixtral-style LLaVA (vision encoder → 2-layer projector → decoder)

This repository provides a 3-bit MLX build of Apriel-1.5-15B-Thinker for on-device multimodal inference on Apple Silicon. In side-by-side tests, the 3-bit variant often:

• uses significantly less RAM than 6-bit,
• decodes faster, and
• tends to produce more direct answers (less “thinking out loud”) at low temperature.

If RAM allows, we also suggest trying 4-bit/5-bit/6-bit variants (guidance below) for tasks that demand more fidelity.

Explore other Apriel MLX variants under the mlx-community namespace on the Hub.

---

🔎 Upstream → MLX summary

Apriel-1.5-15B-Thinker is a multimodal reasoning VLM built via depth upscaling, two-stage multimodal continual pretraining, and SFT with explicit reasoning traces (math, coding, science, tool-use).
This MLX release converts the upstream checkpoint with 3-bit quantization for smaller memory and quick startup on macOS.

---

📦 Contents

• config.json (MLX config for Pixtral-style VLM)
• mlx_model*.safetensors (3-bit shards)
• tokenizer.json, tokenizer_config.json
• processor_config.json / image_processor.json
• model_index.json and metadata

---

🚀 Quickstart (CLI)

Single image caption

python -m mlx_vlm.generate \
  --model <this-repo-id> \
  --image /path/to/image.jpg \
  --prompt "Describe this image in two concise sentences." \
  --max-tokens 128 --temperature 0.0 --device mps --seed 0

🔀 Model Family Comparison (2-bit → 6-bit)

TL;DR: Start with 3-bit for the best size↔quality trade-off. If you need finer OCR/diagram detail and have RAM, step up to 4-bit/5-bit. Use 6-bit only when you have headroom and you explicitly instruct concision.

📊 Quick Comparison

*Indicative for a ~15B VLM under MLX; exact numbers vary with device, image size, and context length.

---

🧪 Example (COCO 000000039769.jpg — “two cats on a pink couch”)

Settings: --temperature 0.0 --max-tokens 100 --device mps. Results vary by Mac model and image resolution; trend is consistent.

---

🧭 Choosing the Right Precision

• I just want it to work on my Mac: 👉 3-bit
• Tiny fonts / invoices / UI text matter: 👉 4-bit, then 5-bit if RAM allows
• I need every drop of quality and have ≥16 GB free: 👉 6-bit (add “Answer directly; do not include reasoning.”)
• I have very little RAM: 👉 2-bit (expect noticeable quality loss)

---

⚙️ Suggested Settings (per variant)

Anti-CoT prompt add-on (any bit-width):

“Answer directly. Do not include your reasoning steps.”

(Optional) Add a stop string if your stack supports it (e.g., stop at "\nHere are my reasoning steps:").

---

🛠️ One-liners (swap model IDs)

# 2-bit
python -m mlx_vlm.generate --model <2bit-repo> --image img.jpg --prompt "Describe this image." \
  --max-tokens 96 --temperature 0.0 --device mps --seed 0

# 3-bit (recommended default)
python -m mlx_vlm.generate --model <3bit-repo> --image img.jpg --prompt "Describe this image in two sentences." \
  --max-tokens 128 --temperature 0.0 --device mps --seed 0

# 4-bit
python -m mlx_vlm.generate --model <4bit-repo> --image img.jpg --prompt "Summarize the document and read key totals." \
  --max-tokens 160 --temperature 0.1 --device mps --seed 0

# 5-bit
python -m mlx_vlm.generate --model <5bit-repo> --image img.jpg --prompt "Extract the fields (date, total, vendor) from this invoice." \
  --max-tokens 192 --temperature 0.1 --device mps --seed 0

# 6-bit
python -m mlx_vlm.generate --model <6bit-repo> --image img.jpg \
  --prompt "Answer directly. Do not include your reasoning steps.\n\nDescribe this image clearly." \
  --max-tokens 192 --temperature 0.0 --device mps --seed 0