MOSS-Transcribe-Diarize-0.9B

MOSS-Transcribe-Diarize-0.9

This version of MOSS-Transcribe-Diarize-0.9 have been converted to run on the Axera NPU using w8a16 quantization.

Compatible with Pulsar2 version: 6.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

• https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

Support Platform

• AX650
  • AX650N DEMO Board
  • M4N-Dock(爱芯派Pro)
  • M.2 Accelerator card

Audio Process

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

PS: The model currently supports the input of up to 6 minutes of audio at a time.

模型下载（Hugging Face）

先创建模型目录并进入，然后下载到该目录：

mkdir -p AXERA-TECH/MOSS-Transcribe-Diarize-0.9
cd AXERA-TECH/MOSS-Transcribe-Diarize-0.9
hf download AXERA-TECH/MOSS-Transcribe-Diarize-0.9 --local-dir .

# structure of the downloaded files
tree -L 3
.
└── AXERA-TECH
    └── MOSS-Transcribe-Diarize-0.9
        ├── config.json
        ├── infer_moss_axengine.py
        ├── model.embed_tokens.weight.bfloat16.bin
        ├── model.embed_tokens.weight.float32.bin
        ├── model.embed_tokens.weight.npy
        ├── moss_openai_api.py
        ├── post_config.json
        ├── preprocessor_config.json
        ├── __pycache__
        ├── qwen3_p256_l0_together.axmodel
        ...
        ├── qwen3_p256_l9_together.axmodel
        ├── qwen3_post.axmodel
        ├── qwen3_tokenizer.txt
        ├── README.md
        ├── requirements.txt
        ├── tokenizer.json
        ├── utils
        ├── wav
        └── whisper_encoder_vq_adaptor.axmodel

3 directories, 42 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

(base) root@ax650:~/wangli/MOSS-Transcribe-Diarize-0.9B# python infer_moss_axengine.py ./wav/2speakers_example.wav
[INFO] Available providers:  ['AxEngineExecutionProvider', 'AXCLRTExecutionProvider']
[LLM] slice_len=256 prefill_max_len=8448 max_seq_len=10240
[Audio] loading /root/wangli/MOSS-Transcribe-Diarize-0.9B/wav/2speakers_example.wav
[Audio] duration=51.663s chunks=2 valid_audio_tokens=646 features=(2, 80, 3000) preprocess=16.421s
[AXEngine] loading audio model: /root/wangli/MOSS-Transcribe-Diarize-0.9B/whisper_encoder_vq_adaptor.axmodel
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.12.0s
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 6.0 48520c11
[Audio AXModel] embeddings=(2, 375, 1024) cost=1.153s
[Prompt] tokens=751 audio_span=665 audio_embeddings=646 build=0.009s
[AXEngine] loading LLM models through InferManager: /root/wangli/MOSS-Transcribe-Diarize-0.9B
Detected prefixes: ['qwen3'], chosen: qwen3, layers: 28
Init InferenceSession:   0%|                                                                                                                                                                                    | 0/28 [00:00<?, ?it/s][INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 6.0 48520c11
Init InferenceSession:   4%|██████▏                                                                                                                                                      
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 6.0 48520c11
Model loaded successfully!
[Generate] decoding until EOS or InferManager max_seq_len.
slice_indices: [0, 1, 2]
Slice prefill done: 0
Slice prefill done: 1
Slice prefill done: 2

[Generate] generated=473 cost=207.844s

[Transcript] segments=19
[0.25][5.28][S01]嗯，那么今天我们就简单的进行一下那个新生招聘的。
[5.32][8.21][S01]嗯，讨论吧，因为现在不是。
[8.24][10.21][S01]马上就新生到校嘛。
[10.23][13.72][S01]然后我们社团呢，也需要招聘一些新的社员。
[13.78][16.86][S01]然后就今天就大概就讨论一下。
[16.88][19.01][S01]嗯，怎么招聘的内容吧。
[19.13][21.51][S01]嗯，我们就首先想一下那个。
[21.53][23.54][S01]招聘的地点在哪里吧。
[24.14][27.41][S02]嗯，地点的话，我们现在可以有三个选择。
[27.46][30.16][S02]嗯，第一个的话，我们可以选择在。
[30.20][32.01][S02]操场，因为那儿。
[32.38][34.81][S02]嗯，学生流动量也挺大的。
[34.74][36.13][S01]操场的话。
[36.17][38.88][S01]这这段时间太热了，我怕。
[38.94][40.88][S01]那个人流量有点少。
[40.91][43.32][S02]嗯，那我们还可以有第二个选择呀。
[43.51][46.38][S02]嗯，我们可以在图书馆楼下。
[46.41][48.58][S02]那里有一块可以遮荫的地方。
[48.75][51.66][S01]哦，图书馆我觉得应该还可以吧。

Install ax-llm

cd /root
git clone --depth 1 -b feat/moss_transcribe_diarize https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
git submodule update --init --recursive
make build && cd build
cmake ..
make install -j6
mv ./axllm /usr/bin

Terminal run

(base) root@ax650:~# axllm run MOSS-Transcribe-Diarize-0.9B/
14:08:36.385 INF Init:468 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  30 /  31 [5.37s<5.55s, 5.58 count/s] init post axmodel ok,remain_cmm(7073 MB)
14:08:41.766 INF Init:791 | max_token_len : 10240
14:08:41.766 INF Init:794 | kv_cache_size : 1024, kv_cache_num: 10240
14:08:41.766 INF init_groups_from_model:150 | prefill_token_num : 256
14:08:41.766 INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 10240
14:08:41.766 INF init_groups_from_model:402 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 256, symbolic_cap: 1
...
14:08:41.766 INF init_groups_from_model:402 | prefill grp: 32, gid: 33, history_cap: 8192, total_cap: 8448, symbolic_cap: 8192
14:08:41.766 INF init_groups_from_model:409 | prefill_max_token_num : 8448
14:08:41.767 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  31 /  31 [5.38s<5.38s, 5.77 count/s] embed_selector init ok
14:08:42.105 INF init_whisper_audio_profile:161 | Whisper audio profile init ok: path=MOSS-Transcribe-Diarize-0.9B/whisper_encoder_vq_adaptor.axmodel duration=30.0s mel_frames=3000 tokens=375 out_dtype=fp32
14:08:42.105 INF Init:1077 | VisionModule init ok: type=MossTranscribeDiarizeVL, audio_pad=151671, audio_tokens_per_second=12.5000, time_marker_every_seconds=5, time_markers=1
14:08:42.106 INF load_config:554 | load config:
14:08:42.106 INF load_config:554 | {
14:08:42.106 INF load_config:554 |     "enable_repetition_penalty": false,
14:08:42.106 INF load_config:554 |     "enable_temperature": false,
14:08:42.106 INF load_config:554 |     "enable_top_k_sampling": false,
14:08:42.106 INF load_config:554 |     "enable_top_p_sampling": false,
14:08:42.106 INF load_config:554 |     "penalty_window": 20,
14:08:42.106 INF load_config:554 |     "repetition_penalty": 1.2,
14:08:42.106 INF load_config:554 |     "temperature": 0.9,
14:08:42.106 INF load_config:554 |     "top_k": 10,
14:08:42.106 INF load_config:554 |     "top_p": 0.8
14:08:42.106 INF load_config:554 | }
14:08:42.106 INF Init:909 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" or "video:<video_file>[:<fps>]" for video (default fps=2), "audio:<file>" for audio.
----------------------------------------
prompt >> 请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。
media >> audio:./wav/2speakers_example.wav
14:09:58.946 INF EncodeBlocks:403 | MossTranscribeDiarizeVL audio encoded: chunks=2 valid_audio_tokens=646
14:09:58.948 INF Prepare:2188 | MOSS audio placeholder expanded: audio_pads=646 span_tokens=665 total_prompt_tokens=751
14:09:58.948 INF SetKVCache:2064 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:751 prefer_symbolic_group:0
14:09:58.948 INF SetKVCache:2086 | current prefill_max_token_num:8448
14:10:01.017 INF SetKVCache:2102 | first run
14:10:01.063 INF Run:257 | input token num : 751, prefill_split_num : 3
14:10:01.063 INF Run:337 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=256
14:10:01.605 INF Run:337 | prefill chunk p=1 history_len=256 grpid=3 kv_cache_num=512 input_tokens=256
14:10:02.996 INF Run:337 | prefill chunk p=2 history_len=512 grpid=4 kv_cache_num=768 input_tokens=239
14:10:04.851 INF Run:664 | ttft: 3787.44 ms
[0.25][S01]嗯，那么今天我们就简单的进行一下那个新生招聘的。[5.28][5.32][S01]嗯，讨论吧，因为现在不是。[8.21][8.24][S01]马上就新生到校嘛。[10.21][10.23][S01]然后我们社团呢，也需要招聘一些新的社员。[13.72][13.78][S01]然后就今天就大概就讨论一下。[16.86][16.88][S01]嗯，怎么招聘的内容吧。[19.01][19.13][S01]嗯，我们就首先想一下那个。[21.51][21.53][S01]招新的地点在哪里吧。[23.54][24.14][S02]嗯，地点的话，我们现在可以有三个选择。[27.41][27.46][S02]嗯，第一个的话，我们可以选择在。[30.16][30.20][S02]操场，因为那儿。[32.01][32.38][S02]嗯，学生流动量也挺大的。[34.81][34.74][S01]操场的话。[36.13][36.17][S01]这这段时间太热了，我怕。[38.88][38.94][S01]那个人流量有点少。[40.88][40.91][S02]嗯，那我们还可以有第二个选择呀。[43.32][43.51][S02]嗯，我们可以在图书馆楼下。[46.38][46.41][S02]那里有一块可以遮阴的地方。[48.58][48.75][S01]哦，图书馆我觉得应该还可以吧。[51.66]

14:11:18.895 NTC Run:1090 | decode stop: reason=eos prompt_tokens=751 generated_tokens=473 output_max_token=-1 max_token_len=10240 dense_decode_start=751 decode_start=751 context_remaining=9016 avg 6.37 token/s
14:11:18.895 INF GetKVCache:2041 | precompute_len:1224, remaining:7224 (tracked)
prompt >> /q
14:11:27.121 INF CheckCmmBalance:79 | [cmm-sentry] Deinit: CMM balanced (baseline 9963 MB, now 9963 MB)

Opneai api serve by axllm

(base) root@ax650:~/wangli# axllm serve MOSS-Transcribe-Diarize-0.9B/
14:13:00.114 INF run_server_mode:2191 | server request/queue timeout: 1800000 ms (config.server_timeout_ms, overridable via --server_timeout_ms)
14:13:00.116 INF Init:468 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 96% | ##############################   |  30 /  31 [5.25s<5.42s, 5.72 count/s] init post axmodel ok,remain_cmm(7073 MB)
14:13:05.369 INF Init:791 | max_token_len : 10240
14:13:05.369 INF Init:794 | kv_cache_size : 1024, kv_cache_num: 10240
14:13:05.369 INF init_groups_from_model:150 | prefill_token_num : 256
14:13:05.369 INF init_groups_from_model:398 | decode grp: 0, gid: 0, max_token_len : 10240
14:13:05.369 INF init_groups_from_model:402 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 256, symbolic_cap: 1
...
14:13:05.370 INF init_groups_from_model:402 | prefill grp: 32, gid: 33, history_cap: 8192, total_cap: 8448, symbolic_cap: 8192
14:13:05.370 INF init_groups_from_model:409 | prefill_max_token_num : 8448
14:13:05.370 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  31 /  31 [5.25s<5.25s, 5.91 count/s] embed_selector init ok
14:13:05.654 INF init_whisper_audio_profile:161 | Whisper audio profile init ok: path=MOSS-Transcribe-Diarize-0.9B/whisper_encoder_vq_adaptor.axmodel duration=30.0s mel_frames=3000 tokens=375 out_dtype=fp32
14:13:05.654 INF Init:1077 | VisionModule init ok: type=MossTranscribeDiarizeVL, audio_pad=151671, audio_tokens_per_second=12.5000, time_marker_every_seconds=5, time_markers=1
14:13:05.654 INF load_config:554 | load config:
14:13:05.654 INF load_config:554 | {
14:13:05.654 INF load_config:554 |     "enable_repetition_penalty": false,
14:13:05.654 INF load_config:554 |     "enable_temperature": false,
14:13:05.654 INF load_config:554 |     "enable_top_k_sampling": false,
14:13:05.654 INF load_config:554 |     "enable_top_p_sampling": false,
14:13:05.654 INF load_config:554 |     "penalty_window": 20,
14:13:05.654 INF load_config:554 |     "repetition_penalty": 1.2,
14:13:05.654 INF load_config:554 |     "temperature": 0.9,
14:13:05.654 INF load_config:554 |     "top_k": 10,
14:13:05.654 INF load_config:554 |     "top_p": 0.8
14:13:05.654 INF load_config:554 | }
14:13:05.654 INF Init:909 | LLM init ok
Starting server on port 8000 with model 'MOSS-Transcribe-Diarize-0.9B'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
  GET  http://10.126.29.50:8000/health
  GET  http://10.126.29.50:8000/v1/models
  POST http://10.126.29.50:8000/v1/chat/completions
  GET  http://10.126.29.227:8000/health
  GET  http://10.126.29.227:8000/v1/models
  POST http://10.126.29.227:8000/v1/chat/completions
  GET  http://172.17.0.1:8000/health
  GET  http://172.17.0.1:8000/v1/models
  POST http://172.17.0.1:8000/v1/chat/completions
Aliases:
  GET  http://127.0.0.1:8000/models
  POST http://127.0.0.1:8000/chat/completions
  GET  http://10.126.29.50:8000/models
  POST http://10.126.29.50:8000/chat/completions
  GET  http://10.126.29.227:8000/models
  POST http://10.126.29.227:8000/chat/completions
  GET  http://172.17.0.1:8000/models
  POST http://172.17.0.1:8000/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: MOSS-Transcribe-Diarize-0.9B

Use axllm serve

python moss_openai_api.py ../wav/2speakers_example.wav --endpoint chat --prompt "请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。"
[0.25][S01]嗯，那么今天我们就简单的进行一下那个新生招聘的。[5.28][5.32][S01]嗯，讨论吧，因为现在不是。[8.21][8.24][S01]马上就新生到校嘛。[10.21][10.23][S01]然后我们社团呢，也需要招聘一些新的社员。[13.72][13.78][S01]然后就今天就大概就讨论一下。[16.86][16.88][S01]嗯，怎么招聘的内容吧。[19.01][19.13][S01]嗯，我们就首先想一下那个。[21.51][21.53][S01]招新的地点在哪里吧。[23.54][24.14][S02]嗯，地点的话，我们现在可以有三个选择。[27.41][27.46][S02]嗯，第一个的话，我们可以选择在。[30.16][30.20][S02]操场，因为那儿。[32.01][32.38][S02]嗯，学生流动量也挺大的。[34.81][34.74][S01]操场的话。[36.13][36.17][S01]这这段时间太热了，我怕。[38.88][38.94][S01]那个人流量有点少。[40.88][40.91][S02]嗯，那我们还可以有第二个选择呀。[43.32][43.51][S02]嗯，我们可以在图书馆楼下。[46.38][46.41][S02]那里有一块可以遮阴的地方。[48.58][48.75][S01]哦，图书馆我觉得应该还可以吧。[51.66]