G9v3-3B-Q4_K_M-GGUF

G9v3-3B Q4_K_M GGUF — 6-core CPU tier

A CPU-tier-validated Q4_K_M cut of ai9stars/G9v3-3B (dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling), packaged for llama.cpp on mid-range machines. The GGUF is quantized by mradermacher/G9v3-3B-GGUF; this repo adds the serving template and measured performance numbers for the 6-core tier. Companion repo: Itopoly/G9v3-3B-Q3_K_M-GGUF (the 4-core tier cut).

What's in this repo

Measured performance (6 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 6 threads, with llama.cpp llama-server:

• Decode: ~25 tok/s at short context (~15 tok/s at 4K ctx, ~10 tok/s at 8K ctx — CPU decode degrades ~5 ms/token per 1K of context)
• Prefill: ~73 tok/s — a 5.5K-token prompt costs ~75–90 s once; the prefix cache makes repeat requests near-instant
• A/B vs Q3_K_M at identical settings: Q3 ≈ 26 tok/s, Q4 ≈ 25 tok/s — on a 3B, quant level barely moves decode; the bandwidth difference is too small. Q4 wins on quality (visibly fewer 3-bit artifacts, coherent reasoning, correct "Paris" sanity answer). Threads matter more than quant on a 3B.
• -fa OFF recommended — flash-attention is slower on CPU

RAM need: ≥10 GB free for 32K context (KV cache is 52 KiB/token on this model); 131K context fits a 12 GB box.

Run with llama.cpp

# get the files
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF G9v3-3B.Q4_K_M.gguf --local-dir .
huggingface-cli download Itopoly/G9v3-3B-Q4_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b-q4_k_m)
llama-server -m G9v3-3B.Q4_K_M.gguf --alias g9v3-3b-q4_k_m \
  -t 6 -tb 8 -c 32768 -fa off --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

Reasoning text arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve this GGUF without g9v3_chat_template_low.jinja: the stock template drops tool-result messages, and the model re-calls the same tool forever (an observed loop bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

• No-think mode: temperature=0.7, top_p=0.95; think mode: temperature=0.9, top_p=0.95.
• Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time ~75–90 s first hit per distinct large prefix (cached afterwards).
• No idle eviction: the model and prompt cache stay resident, so second requests on the same prefix are near-instant.
• Tool calling works (verified single-turn + both tool-history formats), but tool-call quality on a 3B is tier-limited — the 39B family models are the tool-heavy choice.

Credits

• Original model: ai9stars/G9v3-3B (Apache-2.0)
• GGUF quantization: mradermacher/G9v3-3B-GGUF
• CPU-tier validation, template, and packaging: Itopoly (see also our Q3_K_M 4-core cut)