Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B-GGUF

Quantized GGUF versions of Nanbeige/Nanbeige4.2-3B for efficient local inference via llama.cpp, LM Studio, and Ollama.

Nanbeige4.2-3B is a loop transformer (num_loops=2) trained on Chinese/English data. 3B parameters, 256K context, optimized for general-purpose chat and instruction following.

Quantization overview

This repo ships best quality per compression band — no Q2, no I-quants, no XL variants. Just the cleanest K-quant in each size band plus the lossless baselines.

All K-quants use an importance matrix (imatrix) calibrated against Project Gutenberg text for better quality at low bit-widths.

Running

llama.cpp (CLI)

llama-cli -m Nanbeige4.2-3B-Q4_K_M.gguf -c 4096 --color -i   --temp 0.1 --top-k 50 --repeat-penalty 1.1

llama.cpp (one-liner via HF)

llama-cli -hf nicolasembleton/Nanbeige4.2-3B-GGUF:Q4_K_M -c 4096 --color -i

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="Nanbeige4.2-3B-Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    n_gpu_layers=99,  # offload all layers to GPU if available
)
print(llm("Hello, how are you?", max_tokens=256)["choices"][0]["text"])

Ollama

Create a Modelfile:

FROM ./Nanbeige4.2-3B-Q4_K_M.gguf

Then:

ollama create nanbeige4.2-3b -f Modelfile
ollama run nanbeige4.2-3b

In-browser (ONNX Runtime Web + Transformers.js)

For browser-based inference, use ONNX Runtime Web directly — no web-llm needed.

Option 1: Our ONNX export (cross-browser, including Apple Safari)

nicolasembleton/Nanbeige4.2-3B-ONNX — BF16 weights with a working num_loops=2 unroll. Works in Chrome, Firefox, Edge (WebGPU fast path) and Apple Safari 17+ macOS (WASM fallback).

import * as ort from "onnxruntime-web";

const session = await ort.InferenceSession.create(
  "https://huggingface.co/nicolasembleton/Nanbeige4.2-3B-ONNX/resolve/main/model.onnx",
  { executionProviders: ["webgpu", "wasm"] },  // Safari 17 macOS falls back to WASM
);

const inputIds = BigInt64Array.from(/* your token ids */);
const feeds = {
  input_ids: new ort.Tensor("int64", inputIds, [1, inputIds.length]),
  attention_mask: new ort.Tensor("int64", new BigInt64Array(inputIds.length).fill(1n), [1, inputIds.length]),
  position_ids: new ort.Tensor("int64", [...Array(inputIds.length).keys()].map(BigInt), [1, inputIds.length]),
};
const { logits } = await session.run(feeds);

Option 2: Community WebGPU-only export (Chrome, Firefox, Edge)

Michionlion/Nanbeige4.2-3B-ONNX-WebGPU — Q4F16 WebGPU-optimized, smaller (~3 GB) but no WASM fallback.

import { pipeline } from "@huggingface/transformers";

const generator = await pipeline(
  "text-generation",
  "Michionlion/Nanbeige4.2-3B-ONNX-WebGPU",
  {
    device: "webgpu",
    dtype: "q4f16",
    model_file_name: "model_webgpu_mlp",
    use_external_data_format: 2,
  },
);
const output = await generator("Hello, how are you?", { max_new_tokens: 256 });

Architecture note: Nanbeige's num_loops=2 (two passes per physical layer) was previously only available in community ONNX exports via custom kernels (MatMulNBitsMlp). Our export unrolls the loop at the Python level — 44 sequential layer calls with shared 22 weights — producing a standard ONNX graph that runs in stock ONNX Runtime Web.

Note: This GGUF repo is for native/server-side inference (llama.cpp, Ollama, LM Studio). The ONNX repo above covers browser inference with cross-browser support including Apple Safari WASM fallback.

Architecture

NanbeigeForCausalLM — loop transformer with 22 layers, num_loops=2 (two passes per layer). 48 heads, 8 KV heads, 3072 hidden, 166144 vocab, 256K context.

Built with llama.cpp b10276 (Aug 2026) — the first release to include Nanbeige architecture support.

Files

• *.gguf — quantized model files
• README.md — this file

License

Inherited: Apache 2.0 license (see Nanbeige/Nanbeige4.2-3B).

Citation

@misc{nanbeige42-3b-gguf,
  title = {{Nanbeige4.2-3B-GGUF}},
  author = {{Nanbeige, quantizations by nicolasembleton}},
  year = {{2026}},
  howpublished = {{Hugging Face}},
  note = {{GGUF quantizations of Nanbeige4.2-3B. For browser inference use nicolasembleton/Nanbeige4.2-3B-ONNX (cross-browser including Apple Safari) or Michionlion/Nanbeige4.2-3B-ONNX-WebGPU (WebGPU-only).}},
}}