G9v3-39A5B-GGUF

G9v3-39A5B GGUF

This repository contains unofficial BF16, Q8_0, and Q4_K_M GGUF conversions of ai9stars/G9v3-39A5B, a preview Mixture-of-Experts language model from AI9Stars.

The conversion was created and tested with a llama.cpp implementation of the G9v3 architecture available on the feature/g9v3-support branch. Until G9v3 support is merged upstream, standard llama.cpp builds will not load this model.

Available files

The IQ4_XS file was quantized with an importance matrix. The matrix is published as G9v3-39A5B.imatrix.gguf (SHA-256 3072a061c77d31657fe4c01a528cf0a67f2abc700f329d4f569c18f6b810861d) so the quantization can be reproduced or used as a base for further IQ quantizations.

Choose the file according to your hardware and quality requirements:

The memory recommendations include headroom for the runtime, context, and operating system. Actual requirements vary with context size and llama.cpp settings.

Model details

The base model is a preview release. Refer to the ai9stars/G9v3-39A5B model card for the original description and usage guidance.

llama.cpp setup

Build the G9v3-enabled branch:

git clone --branch feature/g9v3-support --single-branch \
  https://github.com/linuxid10t/llama.cpp.git
cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp/build --target llama-cli llama-server -j

Command line

Both GGUF files contain the model's chat template. Start an interactive conversation with the Q4_K_M file using:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -cnv

For BF16, use the same command with the other filename:

./llama.cpp/build/bin/llama-cli \
  -m G9v3-39A5B-BF16.gguf \
  -c 4096 -t 32 -cnv

Start with a modest context such as 4,096 tokens, then increase it according to your available memory and workload.

OpenAI-compatible server

./llama.cpp/build/bin/llama-server \
  -m G9v3-39A5B-Q4_K_M.gguf \
  -c 4096 -t 32 -tb 32 \
  --host 127.0.0.1 --port 8080

Replace the model filename with G9v3-39A5B-BF16.gguf to serve the BF16 version on a machine with sufficient memory.

Example request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "G9v3-39A5B-Q4_K_M.gguf",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0.7,
    "top_p": 0.95,
    "max_tokens": 256
  }'

Bind the server to a non-loopback address only after configuring appropriate authentication and network controls.

Conversion provenance

Converted from the 2026-08-19 revision of the source BF16 checkpoint (Hugging Face commit 70213b14ea559298507e46e36e53d9c4c76144fc, 16 safetensors shards). The GGUF files were created and quantized with:

python convert_hf_to_gguf.py G9v3-39A5B \
  --outfile G9v3-39A5B-BF16.gguf \
  --outtype bf16

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q8_0.gguf \
  Q8_0 32

./build/bin/llama-quantize \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-Q4_K_M.gguf \
  Q4_K_M 32

The importance matrix for IQ4_XS was computed on the BF16 GGUF over 165 chunks (2048 tokens each, ~338k tokens) of a curated calibration mix (prose, code, technical, and multilingual text, including Chinese, from the froggeric/imatrix collection):

./build/bin/llama-imatrix \
  -m G9v3-39A5B-BF16.gguf \
  -f g9v3-calib.txt \
  -c 2048 --chunks 256 -t 32 \
  -o G9v3-39A5B.imatrix.gguf

./build/bin/llama-quantize \
  --imatrix G9v3-39A5B.imatrix.gguf \
  G9v3-39A5B-BF16.gguf \
  G9v3-39A5B-IQ4_XS.gguf \
  IQ4_XS 32

Conversion environment:

• llama.cpp base commit: 9d57ce456 (b10436)
• G9v3 support commit: 77b13da1a
• GGUF version: V3
• Tensor count: 568 in each file
• BF16 weight size: 74,371.15 MiB, 16.01 bits per weight
• Q8_0 weight size: 39,553.33 MiB, 8.51 bits per weight
• Q4_K_M weight size: 22,462.90 MiB, 4.84 bits per weight
• IQ4_XS weight size: 19,899.92 MiB, 4.28 bits per weight

Validation and performance

The Q4_K_M model was tested end-to-end with both llama-cli and llama-server on a CPU-only system. The performance figures below apply only to Q4_K_M.

The exact rate will vary with prompt length, thread count, memory speed, build options, and operating system.

Validation performed (2026-08-19 checkpoint revision):

• Successful load and text generation from the Q4_K_M file
• Successful load and text generation from the IQ4_XS file (imatrix quantization): correct reasoning and answer, 11.29 tokens/s generation
• Successful /v1/chat/completions request with separate reasoning content
• BF16 GGUF metadata and all 568 tensor shapes loaded successfully
• BF16 file successfully used as the source for both the Q8_0 and Q4_K_M quantizations (all 568 tensors streamed without error)
• Importance matrix covers all 37 MoE layers and 904 tensor entries; only three layer-24 expert tensors at 99.69% activation coverage (one rarely routed expert), all other entries fully populated
• Converter dry run passed against all 16 source checkpoint shards after rebasing onto llama.cpp b10436
• Synthetic G9v3 CPU architecture test: zero numerical error
• GGUF serialization round-trip test: passed

The BF16 file was not generation-benchmarked on the 64 GiB validation machine; it requires a system with more memory for representative testing.

Sampling

The base model authors recommend the following settings:

Limitations

• Q4_K_M is a lossy quantization and may perform differently from the BF16 file and original checkpoint.
• The BF16 GGUF preserves source precision but has much higher memory and storage requirements.
• G9v3 is a preview model and may produce inaccurate, biased, or unsafe output.
• Outputs should be independently reviewed before use in medical, legal, financial, safety-critical, or other high-stakes settings.
• This GGUF currently depends on the linked G9v3-enabled llama.cpp branch.
• Long-context capability was not benchmarked as part of this conversion test.

License and attribution

The original model is released by AI9Stars under the Apache License 2.0. This conversion retains that license. See the base model repository for its complete license terms and attribution information.

This conversion is independently produced and is not an official AI9Stars release.