MiniCPM5-2B-EXL3-Quantized

MiniCPM5-2B EXL3 (4.0 bpw)

[!WARNING] Unofficial Community Quantization: This repository provides community-quantized EXL3 weights for openbmb/MiniCPM5-2B. It is not affiliated with, maintained by, or endorsed by OpenBMB or ModelBest. For the official model card, research papers, and original unquantized weights, refer to the official OpenBMB repository.

---

1. Smaller File. Closer to the Original.

EXL3 (ExLlamaV3) departs from conventional scalar quantization (such as AWQ, standard GPTQ, or GGUF block rounding). Rather than rounding weights independently into scalar bins with per-group scales every 16 or 32 values, EXL3 utilizes multidimensional lattice vector quantization guided by calibration Hessians.

Why 4 bits in EXL3 really is 4 bits

In conventional block formats (like GGUF Q4_K_M or NVFP4), scale metadata must be stored every 16 to 32 weights, adding non-trivial bookkeeping bloat. A nominal 4-bit model often requires 4.5 to 4.8 bits per weight on disk.

In EXL3, quantization traverses the weights along optimized codebook paths with global channel scales, meaning 4.0 bpw truly equals 4.0 bits per weight without per-block scale bloat.

Output Drift & Quality Benchmark

Drift is measured via KL divergence between output token distributions relative to unquantized BF16 (lower is better), alongside the percentage of positions where the quantized model selects the exact same top-1 token as the original unquantized model:

At 4 bits, EXL3 exhibits roughly one-third the drift of standard scalar 4-bit formats like NVFP4 or Q4_K_M, while producing a noticeably smaller file on disk.

---

2. Empirical Verification on NVIDIA Tesla T4

The weights in this repository have been tested end-to-end on an NVIDIA Tesla T4 (Turing sm_75, 15 GB VRAM) using greedy sampling.

Measured Performance

• Active Peak VRAM: 1.29 GB during active token generation.
• Generation Speed: 68.0 to 70.0 tokens per second.
• Time to First Token (Cached): 57.7 ms.

Sample Outputs from Live Generation

Prompt 1: Primary Colors

User: What are the three primary colors in traditional color theory? Answer in one sentence.
Assistant: The three primary colors are red, blue, and yellow.

(63 tokens, 67.9 tok/s, Peak VRAM: 1.29 GB)

Prompt 2: Photosynthesis

User: Explain photosynthesis in two concise sentences.
Assistant: Photosynthesis is the process by which plants convert sunlight into energy. This energy is used to produce glucose, which plants need to survive.

(63 tokens, 69.9 tok/s, Peak VRAM: 1.29 GB)

Prompt 3: Multi-step Reasoning

User: If a train travels at 60 mph for 2.5 hours, how far does it travel? Show the calculation.
Assistant: The distance traveled is calculated by multiplying the speed by the time. So, the distance = 60 mph * 2.5 hours = 150 miles. The answer is 150 miles.

(63 tokens, 67.9 tok/s, Peak VRAM: 1.29 GB)

---

3. Why This Model is Ideal for Phones & Edge Devices

MiniCPM5-2B EXL3 is purpose-built for extreme-efficiency deployments such as mobile phones (Android / iOS), single-board computers (Raspberry Pi 5), mini PCs, and thin-and-light laptops.

1. Zero Low-Memory-Killer (LMK) Risk on Phones

On modern smartphones with 6 GB or 8 GB RAM, running an unquantized 5 GB model or a bloated 4-bit format triggers the operating system's Low Memory Killer (LMK), crashing background applications or terminating the model process. At only 1.29 GB peak memory, MiniCPM5-2B EXL3 leaves over 75% of mobile system RAM untouched.

2. Over 3x Memory Bandwidth Reduction (Battery & Thermals)

LLM autoregressive decoding is strictly memory-bandwidth bound. Every single generated token requires streaming the entire model weights through the processor's memory bus:

• A 5.0 GB FP16 model transfers 5.0 GB of memory per token generated.
• MiniCPM5-2B EXL3 transfers only 1.61 GB per token. Cutting memory bandwidth transactions by 3.1x directly slashes SoC power draw, prevents thermal throttling during extended sessions, and preserves phone battery life.

3. Representational Depth at a 2B Parameter Size

Unlike shallow 2B models, MiniCPM5-2B features 42 transformer layers. This gives it structural depth comparable to much larger models, enabling coherent multi-step reasoning, grammar correction, and document parsing completely offline without cloud API latency or subscription fees.

4. Complete Offline Privacy

Runs 100% locally on-device. Sensitive personal notes, voice transcripts, private emails, and local agent tasks never leave the device.

---

4. Model Architecture Specifications

---

5. Storage & Per-Layer Allocation

---

6. VRAM Consumption & Memory Budget

Note: Includes CUDA runtime context overhead (~300–500 MB).

---

7. Hardware Compatibility & Running on Turing / Tesla T4

When attempting to build or run ExLlamaV3 on Turing hardware (such as Tesla T4, RTX 2060/2070/2080, or GTX 1660), upstream turboderp-org/exllamav3 fails during compilation due to Ampere-specific PTX assembly (mma.sync.aligned.m16n8k16 and cp.async) and 90 KB shared memory allocations.

Solution: Upstream PR #325 (sm75-dev)

To run on Turing (sm_75) GPUs, use the architectural adaptation branch from upstream PR #325 (rluisr/exllamav3:sm75-dev):

• Chained Dual MMA: Decomposes each m16n8k16 operation into two chained m16n8k8 instructions with identical register lane alignment.
• Synchronous Load Fallback: Replaces asynchronous cp.async with synchronous 16-byte register loads and stores.
• Clamped Shared Memory: Dynamically clamps shared memory allocation requests to 64 KB on Turing.

Setup Script for Tesla T4 & Cloud Environments

# 1. Install prerequisites
pip install -q tokenizers>=0.21.1 numpy>=2.1.0 scipy>=1.14.1 scikit-learn>=1.5.0 rich typing_extensions safetensors>=0.3.2 ninja pillow pyyaml marisa_trie pydantic llguidance>=1.7.0 huggingface_hub

# 2. Clone the Turing-adapted ExLlamaV3 branch
git clone --depth 1 -b sm75-dev https://github.com/rluisr/exllamav3.git /tmp/exllamav3

# 3. Build and install for sm_75
export TORCH_CUDA_ARCH_LIST="7.5"
export MAX_JOBS=4
pip install -v --no-build-isolation -e /tmp/exllamav3

---

8. Inference Usage

Python with ExLlamaV3

from huggingface_hub import snapshot_download
from exllamav3 import Cache, Config, Generator, Job, Model, Tokenizer
from exllamav3.generator.sampler import GreedySampler

# Download model weights
model_dir = snapshot_download("ewin-reg/MiniCPM5-2B-EXL3")

# Initialize model and 4k KV-cache
config = Config.from_directory(model_dir)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=4096)
model.load()

tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# ChatML prompt formatting
prompt = (
    "<|im_start|>user\n"
    "Explain the difference between vector quantization and scalar quantization in 2 bullet points.<|im_end|>\n"
    "<|im_start|>assistant\n"
    "<think>\n\n</think>\n"
)

job = Job(
    input_ids=tokenizer.encode(prompt, add_bos=False),
    max_new_tokens=128,
    sampler=GreedySampler(),
)
generator.enqueue(job)

# Stream response tokens
print("Response:")
while generator.num_remaining_jobs():
    for r in generator.iterate():
        if r.get("stage") == "streaming":
            print(r.get("text", ""), end="", flush=True)
print()

TabbyAPI Integration

Add the model to your TabbyAPI deployment:

# config.yml
model:
  model_dir: "/path/to/models/MiniCPM5-2B-EXL3"
  max_seq_len: 8192
  cache_mode: "FP16"  # Or Q4 for ultra-low memory
  gpu_split: "auto"

---

9. License & Citation

• Original Model: OpenBMB/MiniCPM5-2B licensed under Apache 2.0.
• Quantization Framework: ExLlamaV3 by turboderp.
• Turing SM75 Hardware Adaptations: Developed by rluisr (PR #325).