Hermes-3-Llama-3.1-405B-mlx-2Bit

mrtoots/Hermes-3-Llama-3.1-405B-mlx-2Bit

The Model mrtoots/Hermes-3-Llama-3.1-405B-mlx-2Bit was converted to MLX format from NousResearch/Hermes-3-Llama-3.1-405B using mlx-lm version 0.26.4.

Toots' Note:

Not super impressed at this 2bit quantization.

Did it for testing for storywriting usecases and was average tbh. Maybe someone else will have better luck with more testing or better off using Hermes 4 or a higher quant.

Leaving it up for people that want to run and attempt a go.

🦛 <span style="color:#800080">If you want a free consulting session, </span>fill out this form <span style="color:#800080">to get in touch!</span> 🤗

Use with mlx

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mrtoots/Hermes-3-Llama-3.1-405B-mlx-2Bit")

prompt="hello"

if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)