MiniCPM5-2B-int4-symg128-ov

MiniCPM5-2B — OpenVINO int4 (symmetric, group size 128, AWQ + scale-estimation)

openbmb/MiniCPM5-2B — a 2.5B dense LlamaForCausalLM with a 131k context, hybrid thinking and native tool calling — converted to the OpenVINO™ IR format with weights compressed to INT4 by NNCF.

Quantization recipe

optimum-cli export openvino \
  --model openbmb/MiniCPM5-2B --task text-generation-with-past \
  --weight-format int4 --sym --group-size 128 --ratio 1.0 \
  --awq --scale-estimation --dataset wikitext2 \
  MiniCPM5-2B-int4-symg128-ov

• Symmetric g128, ratio 1.0, AWQ + scale-estimation on wikitext2 (the plain text-model path; no VLM calibration caveats apply here).
• Exported with optimum-intel main (2.2.0.dev0+f2455b9), transformers 5.10.2, NNCF 3.3.0, OpenVINO 2026.3.1. Symmetric int4 keeps it eligible for the Intel NPU.
• ACTIVATIONS_SCALE_FACTOR left at the baked 8.0 (hidden size 2048 does not need the bump that larger models do); verified coherent on long generations.

Runs on OpenVINO GenAI 2026.3 or newer.

Usage

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("MiniCPM5-2B-int4-symg128-ov", "GPU")
cfg = ov_genai.GenerationConfig()
cfg.max_new_tokens = 512
cfg.do_sample = True
cfg.temperature = 1.0      # vendor-recommended
cfg.top_p = 0.95
print(pipe.generate("Write a Python function that merges overlapping intervals.",
                    generation_config=cfg))

Two things a server in front of it must know:

• Thinking. The template is Qwen3-style (<think>, enable_thinking). With thinking on, sample — greedy think-mode loops ("The user says… So we need… The user says…" for tens of thousands of characters); at temperature 1.0 the same prompts close their reasoning in 2–8k characters. Budget ~8192 tokens.
• Tool calls are attribute XML with special-token markup. The template declares tools hermes-style but the model answers <function name="read_file"><param name="path">config.yaml</param></function> (CDATA-wrapped strings when they contain <, & or newlines), inline, with no <tool_call> wrapper. <function, </function>, <param and </param> are special tokens: decode with skip_special_tokens=False or the markup vanishes and only name="read_file"> name="path">config.yaml reaches your parser.

Benchmarks

This model is measured in core-ultra-llm-server — per-task pass/fail (code generation, editing, agentic tool use, analysis, autocomplete) on an Intel Core Ultra 155H iGPU, re-run whenever the engine or server changes. The leaderboard there is authoritative; this card carries no numbers so it cannot go stale. Its benchmark card (decoding, think budget) is cards/HarmenWessels__MiniCPM5-2B-int4-symg128-ov.yaml.