G9v3-3B-Q3_K_M-GGUF

G9v3-3B Q3_K_M GGUF — 4-core CPU tier

A CPU-tier-validated Q3_K_M cut of ai9stars/G9v3-3B (dense ~3B, LlamaForCausalLM, 131K context, think/no-think modes, XML tool calling), packaged for llama.cpp on low-core machines. The GGUF itself is quantized by mradermacher/G9v3-3B-GGUF; this repo adds the serving template and measured performance numbers for the 4-core tier.

What's in this repo

Measured performance (4 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz run at 4 threads to simulate a 4-core box, with llama.cpp llama-server:

• Decode: ~25 tok/s (think-mode ~14 tok/s) — comfortably above the ~10 tok/s interactive floor for a reasoning model
• Prompt processing: ~55 tok/s — a 5.5K-token prompt costs ~100 s once; the prefix cache makes repeat requests ~0.6 s
• Sanity check: reasoning is separated from the answer, tool calls emit structured tool_calls, and legacy OpenAI function_call history is normalized by the template

RAM need: ≥6 GB. Context memory: KV cache is 52 KiB/token on this model — 8192 ctx fits an 8 GB box, 32768 ctx fits 16 GB.

Run with llama.cpp

# get the files
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3-3B.Q3_K_M.gguf --local-dir .
huggingface-cli download itopoly/G9v3-3B-Q3_K_M-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve (OpenAI-compatible on /v1/chat/completions, model name: g9v3-3b)
llama-server -m g9v3-3B.Q3_K_M.gguf --alias g9v3-3b \
  -t 4 -tb 4 -c 8192 --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

Reasoning text arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve this GGUF without g9v3_chat_template_low.jinja: the stock template drops tool-result messages, and the model re-calls the same tool forever (an observed loop bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

• No-think mode: temperature=0.7, top_p=0.95; think mode: temperature=1.0, top_p=0.95.
• Prefill is the bottleneck on CPU — keep prompts ≤ 1–2K tokens, or accept a one-time ~2 min first hit per distinct large prefix (cached afterwards).
• Tool calling works (verified single-turn + both tool-history formats), but tool-call quality on a 3B is tier-limited; the 39B family models are the tool-heavy choice.

Credits

• Original model: ai9stars/G9v3-3B (Apache-2.0)
• GGUF quantization: mradermacher/G9v3-3B-GGUF
• CPU-tier validation, template, and packaging: itopoly