gemma-3-4b-it-Q4_K_M-GGUF

Gemma 3 4B IT — Q4_K_M GGUF

Quantized version of google/gemma-3-4b-it using llama.cpp Q4_K_M quantization.

Self-quantized from the official weights — not sourced from a third party.

Model Details

Why Q4_K_M?

Q4_K_M uses a mixed quantization strategy (some layers at Q4, key layers at Q6) which gives significantly better output quality than naive INT4 while staying at roughly the same file size.

Usage

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="alefalab/gemma-3-4b-it-Q4_K_M-GGUF",
    filename="gemma-3-4b-it-Q4_K_M.gguf",
    n_ctx=4096,
)

response = llm.create_chat_completion(messages=[
    {"role": "user", "content": "Analyze this CV for a Software Engineer role: ..."}
])
print(response["choices"][0]["message"]["content"])

Android Integration

This model is used in the CVPilot Android app via llama.cpp Android bindings. The GGUF format is inference-engine agnostic — compatible with llama.cpp, llama-android, Ollama, LM Studio, and any other GGUF-compatible runtime.

License

This model inherits the Gemma Terms of Use.