antares-1b-mlx-8bit

antares-1b-mlx-8bit

8-bit MLX quantization of fdtn-ai/antares-1b (Cisco Foundation AI's Antares 1B, built on IBM Granite 4.0 1B) for Apple Silicon.

• Format: MLX, 8-bit affine, group size 64 (~8.5 bits/weight)
• Size: ~1.7 GB weights, ~1.9 GB peak inference memory
• Converted with: mlx-lm 0.31.3
• Base model: fdtn-ai/antares-1b — agentic vulnerability localization

Usage

pip install mlx-lm
mlx_lm.generate --model dpoled/antares-1b-mlx-8bit \
  --prompt "List common causes of SQL injection." --max-tokens 256

from mlx_lm import load, generate

model, tok = load("dpoled/antares-1b-mlx-8bit")
msgs = [{"role": "user", "content": "Explain a buffer overflow in one sentence."}]
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=256, verbose=True))

It is a reasoning model — output may include a </think> segment before the final answer.

Example run

mlx_lm.generate --model dpoled/antares-1b-mlx-8bit \
  --prompt "List common causes of SQL injection." --max-tokens 1024

Output (abridged):

I'll list the most common causes of SQL injection, focusing on input handling
and query construction vulnerabilities.

1. Using string concatenation to build SQL queries
2. Using dynamic SQL with user input
...

Measured on Apple Silicon: prompt 39 tokens @ ~840 tok/s, generation 297 tokens @ ~183 tok/s, peak memory ~1.88 GB.

Conversion notes

The base checkpoint needed two fixes before mlx_lm.convert (0.31.3) would run:

1. Flatten rope_theta — the config (transformers 5.8.0) nests it under rope_parameters, while mlx-lm's granitemoehybrid expects a flat rope_theta.
2. Strip the redundant lm_head.weight — it is byte-identical to model.embed_tokens.weight (tied embeddings); mlx-lm ties the output projection and rejects the extra tensor.

License

Apache 2.0, inherited from the base model. See fdtn-ai/antares-1b.