Llama-3.2-1B-Instruct-GGUF

Llama-3.2-1B-Instruct — GGUF

Built with Llama.

A Q4_K_M GGUF quantization of Meta's Llama-3.2-1B-Instruct, for use with llama.cpp and compatible runtimes (Ollama, LM Studio, llama-cpp-python, etc.).

Q4_K_M is the sweet spot for a 1B model — smallest footprint with still-solid quality.

Usage

llama.cpp

llama-cli -hf andreolf/Llama-3.2-1B-Instruct-GGUF:Q4_K_M -p "Hello!"

Ollama

ollama run hf.co/andreolf/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

llama-cpp-python

from llama_cpp import Llama
llm = Llama.from_pretrained(
    repo_id="andreolf/Llama-3.2-1B-Instruct-GGUF",
    filename="Llama-3.2-1B-Instruct-Q4_K_M.gguf",
    n_ctx=4096,
)
print(llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}]
)["choices"][0]["message"]["content"])

How it was made

Converted to an F16 GGUF with convert_hf_to_gguf.py, then quantized to Q4_K_M with llama-quantize from llama.cpp. No fine-tuning or modification was applied to the weights.

License & attribution

This is a quantized redistribution of Meta's Llama-3.2-1B-Instruct and is governed by the Llama 3.2 Community License. Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. All credit for the underlying model goes to Meta; this repo only provides a GGUF conversion for convenience.