Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B-GGUF

GGUF quantizations of Nanbeige4.2-3B for use with llama.cpp, KoboldCpp, LM Studio, Jan, Open WebUI, Ollama (via GGUF import), and other GGUF-compatible inference engines.

Base Model: Nanbeige/Nanbeige4.2-3B

About

This repository provides a collection of GGUF quantizations of Nanbeige4.2-3B optimized for local inference across a wide range of hardware configurations.

The model was converted from the original Hugging Face weights to GGUF format using the latest available llama.cpp conversion tools. Both traditional K-quants and importance-aware IQ-quants are included to provide a balance between quality, memory usage, and inference speed.

Available Quantizations

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Full Precision

Recommended Quantizations

For 4 GB RAM

• IQ2_M
• Q2_K_S

For 6 GB RAM

• IQ3_M
• Q3_K_M

For 8 GB RAM

• IQ4_NL
• Q4_K_M

For Maximum Quality

• Q6_K
• Q8_0
• BF16

Usage with llama.cpp

./llama-cli \
  -m Nanbeige4.2-3B-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms."

LM Studio

1. Download your preferred GGUF file.
2. Place it in the LM Studio models directory.
3. Refresh models.
4. Load and start chatting.

Ollama

Create a Modelfile:

FROM ./Nanbeige4.2-3B-Q4_K_M.gguf

Then:

ollama create nanbeige4-3b -f Modelfile

Run:

ollama run nanbeige4-3b

Quantization Notes

• K-Quants were generated using llama-quantize.
• IQ-Quants were generated using importance matrix (imatrix) quantization for improved quality retention at lower bitrates.
• Quantizations were produced using the latest available llama.cpp release at build time.

Credits

Base Model

All model weights, architecture, training, and tokenizer credits belong to the original authors of:

Nanbeige/Nanbeige4.2-3B

GGUF Conversion & Quantization

Converted and quantized for the community by:

Nithin Sai Kumar (NANI-Nithin)

Disclaimer

This repository only provides converted and quantized GGUF files. Please refer to the original model repository for:

• Training details
• Evaluation results
• License information
• Intended use guidance
• Safety considerations

Users must comply with the original model license and usage restrictions.

Support the Original Authors

If you find this model useful, please consider supporting the original creators by starring, downloading, and contributing feedback to the original repository:

https://huggingface.co/Nanbeige/Nanbeige4.2-3B