distill-E4B-it-4-bit-MLX

distill-E4B-it-4-bit-MLX

MLX mixed-precision 4-bit (text-only) build of samuelfaj/distill-E4B-it-4-bit, for fast, accurate local inference on Apple Silicon. Distilled for CLI / command-output compression → single-line {"response": <str>, "confidence": <int 0-100>}.

✅ Recommended build. Upgraded from plain 4-bit to a Q4_K_M-style mixed recipe (lm_head + sensitive v_proj/down_proj at 6-bit, rest 4-bit). Pair it with the constrained decoding snippet below for near-perfect format compliance.

Details

Evaluation

Set: eval_gold_v1 — 130 held-out CLI-compression cases. Decode: greedy.

Mixed precision + constrained decoding recovers ~60% of the quantization gap to fp16 and pushes format compliance above the fp16 baseline (+10 pts vs plain 4-bit). Confidence calibration is unchanged (ECE ≈ 0.29) — treat confidence as a soft signal.

Usage (with constrained decoding)

The model targets a strict {"response","confidence"} JSON contract. Prefilling the opening of that object forces the schema and suppresses any reasoning preamble — this is what drives JSON compliance from 92% → 99%.

pip install mlx-lm

import json
from mlx_lm import load, generate

model, tok = load("samuelfaj/distill-E4B-it-4-bit-MLX")

def ask(prompt: str) -> dict:
    text = tok.apply_chat_template(
        [{"role": "user", "content": prompt}],
        add_generation_prompt=True, tokenize=False,
    )
    prefix = '{"response": "'                       # constrained decode: pin the contract
    raw = prefix + generate(model, tok, prompt=text + prefix, max_tokens=800, verbose=False)
    return json.loads(raw)

print(ask('Did the tests pass? Return PASS or FAIL, then failing tests.\n\n'
          'PASS src/auth.test.ts\nFAIL src/queue.test.ts\n1 passed, 1 failed.'))
# {'response': 'FAIL src/queue.test.ts', 'confidence': 98}

Plain (unconstrained) decoding also works but is ~10 pts less format-compliant:

python -m mlx_lm generate --model samuelfaj/distill-E4B-it-4-bit-MLX \
  --prompt "Is this safe? SAFE/REVIEW/UNSAFE.\n\nDROP TABLE users;" --max-tokens 256

Other builds

3-bit (⚠️ degraded) · 2-bit (❌ broken) · comparison vs distill-1.7B