Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B GGUF

Unofficial GGUF conversions of Nanbeige/Nanbeige4.2-3B, pinned to source revision 451ed48c3273ecef7ea8faaa43c31ce529763bb1.

Nanbeige4.2-3B is not an ordinary Llama stack. It reuses 22 physical decoder layers for two loops, applies the shared final RMSNorm at the loop boundary, and needs 44 logical KV-cache slots. These files therefore use the custom GGUF architecture name nanbeige and require the included llama.cpp patch.

Files

The source checkpoint contains 4,169,800,704 physical parameters. The Q4_K_M file is about 4.93 bits per weight; Q8_0 is about 8.51 bits per weight.

Runtime

Apply the patch to the exact tested llama.cpp revision:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout cf512566dc89d7a2bdefd43f6d4b85a1c18e93a7
git apply /path/to/llama.cpp-nanbeige.patch
cmake -B build
cmake --build build --config Release

Then run either quant:

./build/bin/llama-cli \
  -m Nanbeige4.2-3B-Q4_K_M.gguf \
  -c 4096 -n 256 \
  -p "Hello"

Stock llama.cpp builds that do not include Nanbeige support will reject the nanbeige architecture. The patch adds conversion metadata, tensor loading, loop-boundary normalization, and 44-layer cache handling; it does not replace model weights.

Validation

Built and tested on Windows with llama.cpp cf512566dc89d7a2bdefd43f6d4b85a1c18e93a7 and an AMD Ryzen 9 7950X3D.

• Both files loaded as architecture nanbeige, block count 44, loop count 2.
• Q4_K_M: prompt benchmark 92.77 tok/s; generation benchmark 15.67 tok/s.
• Q8_0: prompt benchmark 53.95 tok/s; generation benchmark 9.37 tok/s.
• Deterministic eight-token generation completed for both files.
• Interactive Q8_0 sample measured 68.6 tok/s prompt and 10.2 tok/s generation.

These are CPU smoke/throughput tests, not quality benchmarks. The original model advertises a 262,144-token context, but local validation used a short 512-token context.

License

The base model is Apache-2.0 licensed. Review the upstream model card for its full terms and intended-use information.