Llama-Breeze2-8B-Instruct-text-only-mlx-4Bit

MXLouis/Llama-Breeze2-8B-Instruct-text-only-mlx-4Bit

The Model MXLouis/Llama-Breeze2-8B-Instruct-text-only-mlx-4Bit was converted to MLX format from voidful/Llama-Breeze2-8B-Instruct-text-only using mlx-lm version 0.26.4.

Use with mlx

pip install mlx-lm

Inference Approach 1: CLI chat with system prompt

Use the chat CLI and set a system prompt. If your model relies on custom tokenizer/chat template logic, also pass --trust-remote-code.

python -m mlx_lm chat \
  --model MXLouis/Llama-Breeze2-8B-Instruct-text-only-mlx-4Bit \
  --trust-remote-code \
  --system-prompt "You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan."

Then type your query at the >> prompt, for example:

>> 請用繁體中文簡短介紹 Swift Package Index。

Inference Approach 2: Python API with system prompt

Programmatically include the system prompt in the messages and apply the chat template.

from mlx_lm import load, generate

model, tokenizer = load("MXLouis/Llama-Breeze2-8B-Instruct-text-only-mlx-4Bit")

SYSTEM_PROMPT = (
    "You are a helpful AI assistant built by MediaTek Research. "
    "The user you are helping speaks Traditional Chinese and comes from Taiwan."
)

user_prompt = "請用繁體中文簡短介紹 Swift Package Index。"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_prompt},
]

# If the tokenizer provides a chat template, apply it to build the prompt string.
if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
else:
    # Fallback: simple concatenation if no chat template is available
    prompt = f"System: {SYSTEM_PROMPT}\nUser: {user_prompt}\nAssistant:"

response = generate(model, tokenizer, prompt=prompt, verbose=True)
print(response)