amiko-hermit-1.0-27b-vl-mlx-3bit

Amiko Hermit 1.0 (MLX mixed 3-bit)

27B vision-language model for tool use. Fine-tuned from Qwen3.8-27B on OpenHermit traces, then converted to mixed 3-bit MLX for Apple Silicon.

Sibling of the 4-bit MLX build. This one is for 16 GB Macs. Vision tower is kept (bfloat16).

Quant

mlx-vlm mixed_3_4 (affine, group size 64):

• language: 3-bit, with 4-bit on sensitive down_proj / v_proj plus embed_tokens and lm_head
• vision encoder: left in bfloat16 (mlx-vlm skips multimodal modules)
• average 3.91 bits/weight, ~12 GB on disk (3 shards)
• smoke generate peak 13.7 GB unified memory on an M5 Max

The 4-bit sibling is ~15 GB / 4.70 bpw. Do not use mlx-lm; vision needs mlx-vlm.

Eval

Held-out tool-call split from mars2titan/amiko-openhermit-toolsft: first assistant tool JSON in the first 260 eval sessions, greedy decode, 256 new tokens. First brace-matched JSON object scored (think tags stripped). 4-bit numbers are from that checkpoint's card.

JSON format drops more than tool-name choice. 4-bit stays the quality default; 3-bit is the 16 GB Mac build.

Use with MLX

pip install -U mlx-vlm

python -m mlx_vlm generate \
  --model mars2titan/amiko-hermit-1.0-27b-vl-mlx-3bit \
  --max-tokens 256 --temperature 0.0 \
  --prompt "Describe this image." --image <path>

python -m mlx_vlm.server --model mars2titan/amiko-hermit-1.0-27b-vl-mlx-3bit

Specs

• Architecture: Qwen3.5 vision-language (Qwen3_5ForConditionalGeneration)
• Quant: affine mixed 3/4-bit, group size 64, 3.91 bits/weight, ~12 GB
• Train: 1 epoch, 250 steps, QLoRA r=16 α=32. Loss 3.18 → 0.62, token accuracy 47% → 85%
• License: Apache 2.0