MiniCPM5-1B-int8-ov

MiniCPM5-1B — OpenVINO INT8 IR

An INT8 weight-compressed OpenVINO IR export of openbmb/MiniCPM5-1B, ready for CPU/GPU inference via Optimum Intel or OpenVINO GenAI.

Model details

• Base model: openbmb/MiniCPM5-1B (dense 1B LlamaForCausalLM architecture)
• Format: OpenVINO IR (.xml / .bin)
• Precision: INT8 asymmetric, per-channel weight compression (100% of layers)
• Features: hybrid reasoning mode (<think> / no-think) via enable_thinking in the chat template

Requirements

python3.12 -m venv ov-env
source ov-env/bin/activate

pip install --upgrade optimum-intel
pip install --upgrade openvino-genai   # optional, for lightweight deployment runtime

Note on transformers version: this model's tokenizer config ships in the newer Transformers v5 unified tokenizer format (tokenizer_class: TokenizersBackend). If you're running Transformers < 5.0 (e.g. 4.56.x), see Known Issues below before loading.

Usage — Optimum Intel

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "path/to/ov_minicpm5_1b_int8"   # or your published HF repo id
model = OVModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [{"role": "user", "content": "Explain quantum entanglement briefly."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,   # set True to enable <think> reasoning mode
)

inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=300)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Usage — OpenVINO GenAI (lightweight runtime)

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("path/to/ov_minicpm5_1b_int8", "CPU")
print(pipe.generate("Explain quantum entanglement briefly.", max_length=300))

Known issues

ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported

This model's tokenizer_config.json uses the Transformers v5 unified tokenizer backend format. If your environment is pinned to Transformers < 5.0 (a common constraint for tools like vLLM, older inference servers, or custom loaders), AutoTokenizer.from_pretrained() will fail with this error.

Fix, without upgrading Transformers: edit tokenizer_config.json in the model folder and change:

"tokenizer_class": "TokenizersBackend",

to:

"tokenizer_class": "PreTrainedTokenizerFast",

PreTrainedTokenizerFast is a generic fast-tokenizer class that has existed since early Transformers 4.x and loads directly from the bundled tokenizer.json, which already contains the complete vocabulary, merge rules, and tokenization logic. No functional difference in tokenization output — this only changes which Python class wraps the same underlying tokenizer data.

Alternative fix: upgrade to transformers>=5.0.0, which recognizes TokenizersBackend natively.

License

MiniCPM5-1B is released under Apache-2.0 by OpenBMB. See the original model card for full details, benchmarks, and usage terms.