mlx-expert-sniper

MLX Expert Sniper

Run 26B+ MoE vision-language models on a 16 GB Apple Silicon Mac.

A single-machine inference engine that streams Mixture-of-Experts weights from SSD on demand. The active resident set stays under 4 GB even though the full model is 13–17 GB. Confirmed working with Gemma 4-26B-A4B (vision) + Falcon Perception (grounded segmentation) on a 16 GB M4 Mac Mini.

┌────────────────────────────────────────────────┐
│  http://localhost:8500   web chat + REST API   │
├────────────────────────────────────────────────┤
│  /api/chat_vision   chained Gemma → Falcon     │
│  /api/falcon        direct grounded segm       │
│  /api/turbo_chat    Gemma vision → Qwen brain  │
└─────────────┬──────────────────────────────────┘
              │
   ┌──────────┴──────────┐
   ▼                     ▼
┌────────────────┐  ┌──────────────────────┐
│ Gemma 4-26B    │  │ Falcon Perception    │
│ A4B vision     │  │ 0.6B segmentation    │
│ ~3 GB resident │  │ ~1.5 GB resident     │
│ via Sniper     │  │ via mlx-vlm          │
│ (SSD streaming)│  │                      │
└────────────────┘  └──────────────────────┘

Why this exists

MoE models only activate ~3–15% of their parameters per token. The other 85–97% sit idle in RAM. Expert Sniper unloads cold experts to SSD and pages them in on demand, so a 26B model behaves like a 4B model for memory pressure.

The sniper-streamed 4-bit gives you smaller RAM than vanilla 2-bit with no quality loss.

Install (from scratch on a fresh Apple Silicon Mac)

Requires macOS 14+, M1/M2/M3/M4 (16 GB+ RAM recommended), Python 3.10+, ~30 GB free disk.

# 1. Clone the deploy repo
git clone https://github.com/walter-grace/mac-code
cd mac-code/research/expert-sniper/distributed
python3 -m venv venv && source venv/bin/activate
pip install -e .
pip install mlx mlx-vlm fastapi uvicorn pillow huggingface_hub

# 2. Download stock Gemma 4 (one time, ~13 GB)
huggingface-cli download mlx-community/gemma-4-26b-a4b-it-4bit \
    --local-dir ~/models/gemma4-source

# 3. Split for SSD streaming (one time, ~5 minutes)
python3 split_gemma4.py \
    --input ~/models/gemma4-source \
    --output ~/models/gemma4-stream
# Produces: ~/models/gemma4-stream/{pinned.safetensors, bin/layer_XX.bin}

# 4. Falcon Perception downloads automatically on first run
#    (~1.5 GB, from tiiuae/Falcon-Perception)

# 5. Launch
python3 -m mac_tensor.cli ui --vision --falcon \
    --stream-dir ~/models/gemma4-stream \
    --source-dir ~/models/gemma4-source \
    --port 8500

Open http://localhost:8500 in a browser. Drop an image, ask Gemma to describe it, then click Ground for Falcon to outline objects precisely.

Three modes — pick the right flag

REST endpoints

Verified hardware

What's chained vision agent (the killer feature)

When you load both Gemma 4 + Falcon Perception, the server exposes a chained reasoning loop:

You: "Is the blue player offside in this image?"
       │
       ▼
Gemma:  "I need to find players, identify the second-to-last defender,
         and compare the attacker's position. Let me ground the players."
       │ tool_call
       ▼
Falcon: → returns 22 player bboxes with centroids
       │
       ▼
Gemma:  "Sorting by x-coordinate... the second-to-last defender is at x=0.65,
         the attacker is at x=0.71. The attacker is past the defender → offside."

This pattern works for any open-ended visual reasoning — the VLM picks what to look for, the segmenter measures it precisely.

Architecture

research/expert-sniper/distributed/
├── mac_tensor/                          ← the deploy package
│   ├── cli.py                           ← `mac-tensor` CLI
│   ├── server.py                        ← FastAPI app
│   ├── vision_engine.py                 ← Gemma 4 sniper wrapper
│   ├── falcon_perception.py             ← Falcon mlx-vlm wrapper
│   ├── agent.py                         ← chained vision agent loop
│   └── static/chat.html                 ← web UI
├── split_gemma4.py                      ← weight-splitting tool
├── README.md                            ← this file
└── ...

Credits

• Gemma 4-26B-A4B by Google DeepMind — Apache 2.0
• Falcon Perception by TII — Apache 2.0
• MLX by Apple Machine Learning Research — MIT
• mlx-vlm by Prince Canuma — MIT
• Expert Sniper streaming engine by Walter Grace

License

Apache 2.0 (the deploy code). Model weights are subject to their respective licenses (Gemma Terms of Use, Falcon Apache 2.0).

Star the GitHub repo

🌟 https://github.com/walter-grace/mac-code

Citation

@software{mlx_expert_sniper_2026,
  author = {Walter Grace},
  title  = {MLX Expert Sniper: Streaming MoE inference for Apple Silicon},
  year   = {2026},
  url    = {https://github.com/walter-grace/mac-code},
}