G9v3-39A5B-IQ4_XS-GGUF

G9v3-39A5B IQ4_XS GGUF — 16-core CPU tier

An imatrix IQ4_XS cut of ai9stars/G9v3-39A5B (39B MoE, ~5.4B active params/token, 131K context, think/no-think modes, tool calling) sized for the first CPU tier that can hold the big model: 16 cores / ≥32 GB RAM. The GGUF is imatrix-quantized by linuxid10t/G9v3-39A5B-GGUF; this repo adds the serving template and measured performance numbers.

⚠️ Read "llama.cpp requirements" below before downloading — stock llama.cpp cannot load this architecture; you need the g9v3 fork.

What's in this repo

llama.cpp requirements (important)

Upstream llama.cpp has no g9v3 architecture support. Serving this GGUF requires:

1. The fork: linuxid10t/llama.cpp, branch feature/g9v3-support
2. A required local patch: in src/llama-context.cpp, graph_max_nodes() must include LLM_ARCH_G9V3 in the MoE branch (n_tokens * 40 budget) — otherwise the server aborts at startup with GGML_ASSERT(obj_new) failed at any context size.

The fork build also serves plain llama-arch GGUFs (e.g. our 3B cuts) — one binary works for all.

Measured performance (8 threads, x86-64 AVX2)

Measured on an 8-vCPU EPYC @ 2.0 GHz with llama.cpp llama-server:

• Decode: ~13 tok/s, prefill: ~43 tok/s (a 16-core box projects ~15–20 tok/s)
• RSS: ~19.9 GiB — validated at 8192 context; a 32 GB box can go higher (KV cache is 38 KiB/token on this model: 32K ≈ 1.2 GiB/seq, 131K ≈ 4.75 GiB/seq)
• Sanity: France question → reasoning separated, answer "The capital of France is Paris."

With only ~5.4B active params/token, a 39B-class MoE runs on plain CPU — slower than the 3B dense cuts but far more capable at tool-calling and multi-step work.

Run

# get the files
huggingface-cli download Itopoly/G9v3-39A5B-IQ4_XS-GGUF G9v3-39A5B-IQ4_XS.gguf --local-dir .
huggingface-cli download Itopoly/G9v3-39A5B-IQ4_XS-GGUF g9v3_chat_template_low.jinja --local-dir .

# serve with the g9v3 fork build (see requirements above)
llama-server -m G9v3-39A5B-IQ4_XS.gguf --alias g9v3-39a5b-iq4_xs \
  -t 8 -tb 8 -c 8192 -fa off --port 8000 --host 0.0.0.0 \
  --chat-template-file g9v3_chat_template_low.jinja

8 threads is what we validated; on a 16-core box try -t 12..16. OpenAI-compatible on /v1/chat/completions; reasoning arrives in message.reasoning, the answer in content.

Why the chat template matters

Don't serve without g9v3_chat_template_low.jinja: the stock template drops tool-result messages and the model re-calls the same tool forever (an observed loop bug). The bundled template normalizes legacy tool history so multi-turn tool use works.

Notes

• Think mode: temperature=1.0, top_p=0.95; no-think mode: temperature=0.7, top_p=0.95.
• -fa off recommended — flash-attention is slower on CPU.
• No idle eviction: model and prompt cache stay resident; repeat requests on the same prefix are near-instant.
• The MoE family ladder: 3B (this repo's 3B siblings), 39A5B (this repo). Middle rungs (20A5B / 10A5B expert-pruned cuts) are planned — see our W4A16 vLLM repo for the GPU tier: Itopoly/G9v3-39A5B-W4A16.

Credits

• Original model: ai9stars/G9v3-39A5B (Apache-2.0)
• GGUF quantization (imatrix): linuxid10t/G9v3-39A5B-GGUF
• CPU-tier validation, template, and packaging: Itopoly (GPU serving: our W4A16 quant)