gemma-4-12B-it-mxfp8-mlx

gemma-4-12B-it-mxfp8-mlx

Brainwaves

         arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.385,0.527,0.766,0.509,0.386,0.664,0.579
q8-hi    0.394,0.522,0.774,0.520,0.370,0.674,0.583
qx86-hi  0.391,0.526,0.777,0.521,0.366,0.677,0.590
mxfp4    0.371,0.517,0.660,0.508,0.368,0.677,0.581

Quant    Perplexity      Peak Memory   Tokens/sec
mxfp8   175.766 ± 3.092  19.42 GB      463
mxfp4   283.801 ± 5.361  13.47 GB      484

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("gemma-4-12B-it-mxfp8-mlx")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)