Huihui-gemma-4-31B-it-v2-MLX-8bit

thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit

This model was converted to MLX format from huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 using mlx-vlm version 0.4.4. Refer to the original model card for more details on the model.

Use with mlx

pip install -U mlx-vlm

python -m mlx_vlm.generate --model thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit --max-tokens 100 --temperature 0.0 --prompt "Describe this image." --image <path_to_image>

Use with Python/mlx

from mlx_vlm.utils import load_config

model, processor = load("thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit")
config = load_config("thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit")

image = ["https://huggingface.co/thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit/resolve/main/images/fuji.jpg"]
prompt = "Can you describe the image in detail?"

formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

output = generate(model, processor, formatted_prompt, image,max_tokens=1024)
print(output)

Convert to mlx model

from mlx_vlm.convert import convert

convert("huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2", 
        mlx_path="/Users/thierrydekerk/MLX/Huihui-gemma-4-31B-it-abliterated-v2-MLX-8bit", 
        q_bits=8, 
        quantize=True, 
        upload_repo="thdekerk/Huihui-gemma-4-31B-it-v2-MLX-8bit"
        )