Agents-A1-8bit

Agents-A1 — MLX (8-bit)

MLX 8-bit quantization of InternScience/Agents-A1 (affine, group size 64). The source is bf16; this is a uniform mlx quantization.

Agents-A1 is a Qwen3.5-MoE vision-language agent model (qwen3_5_moe, Qwen3_5MoeForConditionalGeneration): 40 decoder layers, 256 routed experts per layer + a shared expert, hidden size 2048, with a vision tower and video preprocessing.

Running it

Multimodal (VLM) — load with mlx-vlm (mlx-lm can't load multimodal architectures):

pip install mlx-vlm
python -m mlx_vlm.generate --model mlx-community/Agents-A1-8bit \
  --prompt "What is 17 * 24? Think step by step." --max-tokens 512
# with an image:
python -m mlx_vlm.generate --model mlx-community/Agents-A1-8bit --image img.jpg --prompt "Describe this image."

Loads and runs in stock mlx-vlm — no patched code needed at inference.

Conversion notes

I first tried oMLX's data-driven oQ quantization, but it doesn't work for this checkpoint: oQ writes the MoE experts in a per-expert layout that omlx's own loader can't read back (parameters not in model), so the quantized model fails to load. This build therefore uses standard mlx-vlm quantization instead — uniform 8-bit, group size 64 — which loads cleanly in both stock mlx-vlm & oMLX.

Throughput

Measured with oMLX's benchmark harness on a Macbook Pro M5 Max 128GB 40 GPU — gen 128 tokens, cold prefill (unique prompt prefix per request, no cache reuse).

Single request (batch 1) — decode tok/s by context

TTFT (cold prefill) is ~precision-independent — ≈0.3 s @1k, 3 s @8k, 21 s @32k, 63 s @64k, ~225 s @128k — prefill is compute-bound, not weight-bound.

Continuous batching (1k context) — aggregate decode tok/s

Aggregate across the batch; per-request rate is that value divided by the batch size.

Smoke test

17 x 24 -> correct (408), coherent, no repetition.

Other precisions

License

apache-2.0, inherited from the base model.