Downloads · 30 days
29
39% of all-time downloads
MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K is a text generation model from AXERA-TECH. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This version of MiniCPM5-2B-GPTQ-Int4 has been converted to run on the AX650 NPU using w4a16 quantization.
All-time downloads
74
Public
Repo size
3.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.axmodel2.7 GB · 84%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Sep 10, 2026
- Updated
- Sep 10, 2026
- SHA
- df71361f
Try a prompt
Base models
Model card
From the Hugging Face model README
MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
This version of MiniCPM5-2B-GPTQ-Int4 has been converted to run on the AX650 NPU using w4a16 quantization.
Compatible with Pulsar2 version: 5.0
Convert tools links:
For those who are interested in model conversion, you can try to export axmodel through the original repo :
Pulsar2 Link, How to Convert LLM from Huggingface to axmodel
Support Platform
- AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card
| Chips | ttft(128 tokens) | prefill | decode | CMM | Flash |
|---|---|---|---|---|---|
| AX650 | 196 ms | 653 tokens/s | 12 token/s | 3.3GiB | 3.8GiB |
The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.
How to use
安装 axllm
方式一：克隆仓库后执行安装脚本：
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh
方式二：一行命令安装（默认分支 axllm）：
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash
方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：
如果没有编译环境，请到：
https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm
下载 最新 CI 导出的可执行程序（axllm），然后：
chmod +x axllm
sudo mv axllm /usr/bin/axllm
模型下载（Hugging Face）
先创建模型目录并进入，然后下载到该目录：
mkdir -p AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
cd AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
hf download AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K --local-dir .
Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board
运行（CLI）
root@ax650:~# axllm run AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K/
11:12:03.898 INF Init:475 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
17% | ##### | 8 / 45 [6.94s<39.03s, 1.15 count/s] init 6 axmodel ok,remain_cmm(9453 MB)^C
20% | ###### | 9 / 45 [7.87s<39.34s, 1.14 count/s] init 7 axmodel ok,remain_cmm(9380 MB)^C
97% | ############################### | 44 / 45 [47.96s<49.05s, 0.92 count/s] init post axmodel ok,remain_cmm(6623 MB)
11:12:51.868 INF Init:798 | max_token_len : 2047
11:12:51.868 INF Init:801 | kv_cache_size : 256, kv_cache_num: 2047
11:12:51.868 INF init_groups_from_model:150 | prefill_token_num : 128
11:12:51.868 INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 2047
11:12:51.869 INF init_groups_from_model:398 | decode grp: 1, gid: 1, max_token_len : 4095
11:12:51.869 INF init_groups_from_model:398 | decode grp: 2, gid: 2, max_token_len : 6143
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 0, gid: 3, history_cap: 0, total_cap: 128, symbolic_cap: 1
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 1, gid: 4, history_cap: 128, total_cap: 256, symbolic_cap: 128
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 2, gid: 5, history_cap: 256, total_cap: 384, symbolic_cap: 256
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 3, gid: 6, history_cap: 384, total_cap: 512, symbolic_cap: 384
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 4, gid: 7, history_cap: 512, total_cap: 640, symbolic_cap: 512
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 5, gid: 8, history_cap: 640, total_cap: 768, symbolic_cap: 640
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 6, gid: 9, history_cap: 768, total_cap: 896, symbolic_cap: 768
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 7, gid: 10, history_cap: 896, total_cap: 1024, symbolic_cap: 896
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 8, gid: 11, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 9, gid: 12, history_cap: 1152, total_cap: 1280, symbolic_cap: 1152
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 10, gid: 13, history_cap: 1280, total_cap: 1408, symbolic_cap: 1280
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 11, gid: 14, history_cap: 1408, total_cap: 1536, symbolic_cap: 1408
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 12, gid: 15, history_cap: 1536, total_cap: 1664, symbolic_cap: 1536
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 13, gid: 16, history_cap: 1664, total_cap: 1792, symbolic_cap: 1664
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 14, gid: 17, history_cap: 1792, total_cap: 1920, symbolic_cap: 1792
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 15, gid: 18, history_cap: 1920, total_cap: 2048, symbolic_cap: 1920
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 16, gid: 19, history_cap: 2048, total_cap: 2176, symbolic_cap: 2048
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 17, gid: 20, history_cap: 2176, total_cap: 2304, symbolic_cap: 2176
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 18, gid: 21, history_cap: 2304, total_cap: 2432, symbolic_cap: 2304
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 19, gid: 22, history_cap: 2432, total_cap: 2560, symbolic_cap: 2432
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 20, gid: 23, history_cap: 2560, total_cap: 2688, symbolic_cap: 2560
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 21, gid: 24, history_cap: 2688, total_cap: 2816, symbolic_cap: 2688
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 22, gid: 25, history_cap: 2816, total_cap: 2944, symbolic_cap: 2816
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 23, gid: 26, history_cap: 2944, total_cap: 3072, symbolic_cap: 2944
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 24, gid: 27, history_cap: 3072, total_cap: 3200, symbolic_cap: 3072
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 25, gid: 28, history_cap: 3200, total_cap: 3328, symbolic_cap: 3200
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 26, gid: 29, history_cap: 3328, total_cap: 3456, symbolic_cap: 3328
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 27, gid: 30, history_cap: 3456, total_cap: 3584, symbolic_cap: 3456
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 28, gid: 31, history_cap: 3584, total_cap: 3712, symbolic_cap: 3584
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 29, gid: 32, history_cap: 3712, total_cap: 3840, symbolic_cap: 3712
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 30, gid: 33, history_cap: 3840, total_cap: 3968, symbolic_cap: 3840
11:12:51.869 INF init_groups_from_model:402 | prefill grp: 31, gid: 34, history_cap: 3968, total_cap: 4096, symbolic_cap: 3968
11:12:51.869 INF init_groups_from_model:409 | prefill_max_token_num : 4096
11:12:51.870 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ | 45 / 45 [47.97s<47.97s, 0.94 count/s] embed_selector init ok
11:12:51.883 INF load_config:554 | load config:
11:12:51.883 INF load_config:554 | {
11:12:51.883 INF load_config:554 | "enable_repetition_penalty": false,
11:12:51.883 INF load_config:554 | "enable_temperature": false,
11:12:51.883 INF load_config:554 | "enable_top_k_sampling": false,
11:12:51.883 INF load_config:554 | "enable_top_p_sampling": false,
11:12:51.883 INF load_config:554 | "penalty_window": 128,
11:12:51.883 INF load_config:554 | "repetition_penalty": 1.1,
11:12:51.883 INF load_config:554 | "temperature": 0.7,
11:12:51.883 INF load_config:554 | "top_k": 20,
11:12:51.883 INF load_config:554 | "top_p": 0.8
11:12:51.883 INF load_config:554 | }
11:12:51.883 INF Init:913 | LLM init ok
Commands:
/q, /exit 退出
/reset 重置 kvcache
/dd 删除一轮对话
/pp 打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >>
prompt >> 请计算 12345 * 6789，并给出计算过程。
11:14:04.261 INF SetKVCache:2027 | decode_grpid:0 prefill_grpid:3 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:33 prefer_symbolic_group:0
11:14:04.261 INF SetKVCache:2049 | current prefill_max_token_num:4096
11:14:04.514 INF SetKVCache:2065 | first run
11:14:04.578 INF Run:257 | input token num : 33, prefill_split_num : 1
11:14:04.578 INF Run:342 | prefill chunk p=0 history_len=0 grpid=3 kv_cache_num=0 input_tokens=33
11:14:04.777 INF Run:671 | ttft: 199.68 ms
<think>
我们被要求计算12345乘以6789，并给出计算过程。我们需要展示计算步骤。
</think>
我们将计算 12345 × 6789。
**计算过程：**
我们可以使用乘法分配律来简化计算，或者直接进行乘法运算。
**方法一：直接乘法**
12345 × 6789
我们可以将6789分解为6000 + 700 + 80 + 9，然后分别计算：
12345 × 6000 = 12345 × 6 × 1000 = 74070 × 1000 = 74,070,000
12345 × 700 = 12345 × 7 × 100 = 86,415 × 100 = 8,641,500
12345 × 80 = 12345 × 8 × 10 = 98,760 × 10 = 987,600
12345 × 9 = 111,105
现在将所有结果相加：
74,070,000 + 8,641,500 = 82,711,500
82,711,500 + 987,600 = 83,699,100
83,699,100 + 111,105 = 83,810,205
所以，12345 × 6789 = 83,810,205。
**方法二：使用竖式乘法**
我们可以按照竖式计算：
12345
× 6789
---------
逐步计算：
首先，12345 × 9 = 111,105
然后，12345 × 80 = 987,600
然后，12345 × 700 = 8,641,500
然后，12345 × 6000 = 74,070,000
将所有部分相加：
111,105 + 987,600 = 1,098,705
1,098,705 + 8,641,500 = 9,740,205
9,740,205 + 74,070,000 = 83,810,205
因此，结果相同。
**最终答案：**
12345 × 6789 = 83,810,205。
11:14:43.674 NTC Run:1104 | hit eos,decode avg 11.85 token/s
11:14:43.699 INF GetKVCache:2004 | precompute_len:495, remaining:3601 (tracked)
启动服务（OpenAI 兼容）
root@ax650:~# axllm serve AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
[I][ Init][ 138]: LLM init start
tokenizer_type = 1
96% | ███████████████████████████████ | 30 / 31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB)
[I][ Init][ 199]: max_token_len : 2047
[I][ Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047
[I][ Init][ 205]: prefill_token_num : 128
[I][ Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1
[I][ Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128
[I][ Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256
[I][ Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384
[I][ Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512
[I][ Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640
[I][ Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768
[I][ Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896
[I][ Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024
[I][ Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152
[I][ Init][ 214]: prefill_max_token_num : 1152
[I][ Init][ 27]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ | 31 / 31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok
[W][ Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
[I][ Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
[I][ Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
[I][ Init][ 672]: VisionModule deepstack enabled: layers=3
[I][ load_config][ 282]: load config:
{
"enable_repetition_penalty": false,
"enable_temperature": false,
"enable_top_k_sampling": false,
"enable_top_p_sampling": false,
"penalty_window": 20,
"repetition_penalty": 1.2,
"temperature": 0.9,
"top_k": 10,
"top_p": 0.8
}
[I][ Init][ 272]: LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K'...
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K
OpenAI 调用示例
from openai import OpenAI
API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K"
messages = [
{"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
{"role": "user", "content": "hello"},
]
client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
model=MODEL,
messages=messages,
)
print(completion.choices[0].message.content)
OpenAI 流式调用示例
from openai import OpenAI
API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/MiniCPM5-2B-GPTQ-Int4-AX650-C128-P4K-CTX6K"
messages = [
{"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
{"role": "user", "content": "hello"},
]
client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
model=MODEL,
messages=messages,
stream=True,
)
print("assistant:")
for ev in stream:
delta = getattr(ev.choices[0], "delta", None)
if delta and getattr(delta, "content", None):
print(delta.content, end="", flush=True)
print("
")
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Languages
- en, zh
- Created
- Sep 10, 2026
- Updated
- Sep 10, 2026