luthor-8b-mlx-4bit

Luthor 8B — MLX 4bit

4-bit-quantised MLX build of Luthor 8B, a Qwen3-8B fine-tune for driving terminal and file-editing tools in an agent loop. For Apple Silicon.

Recommended default. Best size/speed/quality balance.

Status: evaluated — it did NOT pass its ship gate. 0/10 on held-out tasks, same as stock Qwen3-8B.

This build

All builds

Measured on an Apple M3 (24 GB), mlx-lm 0.31.3, 150-token generation at --temp 0.0.

A note on speed

Larger quantisations are slower here, not faster. Apple Silicon inference is memory-bandwidth-bound, so fewer bytes per weight means more tokens per second. 8-bit is the highest fidelity and the slowest.

What didn't ship

A mixed_2_6 build (3.2 GB, 3.284 bits/weight) was produced and discarded: it degenerates into repeated tokens rather than coherent text. Post-training quantisation below ~4 bits/weight breaks this model. Reaching ternary-class compression — as Bonsai 2 does at ~1.58 bits/weight — requires quantisation-aware training, not post-training conversion.

Usage

pip install mlx-lm
mlx_lm.generate --model IAMIbrahim/luthor-8b-mlx-4bit \
  --prompt "The test suite fails with ImportError. What is your first step?" \
  --max-tokens 512

from mlx_lm import load, generate
model, tokenizer = load("IAMIbrahim/luthor-8b-mlx-4bit")
messages = [
    {"role": "system", "content": SYSTEM_PROMPT_WITH_TOOLS},   # Hermes-style <tools> block
    {"role": "user", "content": "Fix the failing test."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

Declare tools exactly as at training time — the model emits <tool_call> and expects <tool_response> back.

Conversion

python -m mlx_lm convert --hf-path IAMIbrahim/luthor-8b \
  --mlx-path luthor-8b-4bit -q --q-bits 4 --q-group-size 64

See the base model card for training data, hyperparameters and limitations.

Evaluation

This model did not pass its ship gate: 0/10 on held-out tasks, identical to stock Qwen3-8B, with worse protocol adherence (completed 0/10 vs 8/10). Published as a negative result and a reproducible pipeline, not as an improvement over the base model. Full numbers and analysis on the base model card.