gemma-3-270m-it

Gemma 3 270M IT on AXERA NPU

Ready-to-run deployment package for google/gemma-3-270m-it on AX650 / NPU3.

• This release packages the bf16 AXERA NPU text runtime.
• Compiled with Pulsar2 llm_build (prefill_len=128, kv_cache_len=1023).
• Includes the tokenizer/config files required at runtime.
• Includes the compiled Gemma 3 text .axmodel files and the packed token-embedding weights.
• Includes the validated bin/axllm binary for AX650 / NPU3 board-side deployment.
• Supports text-only chat through the axllm runtime.

Supported Platform

• AX650 / NPU3

Performance

All measurements below were taken on AX650 / NPU3 with the packaged runtime files. TTFT stands for time to first token. The packaged prefill window is 128 tokens; prompts longer than that are split into multiple prefill chunks, which is reflected in the last row.

• The numbers above are measured on-device with the axllm run interactive flow, not theoretical.
• A prompt that exceeds the 128-token prefill window is processed in multiple chunks, so its TTFT is higher than a single-chunk prompt (about one extra chunk of prefill). The long prompt above (170 tokens, two chunks) correctly extracted "next Friday" as the answer from the passage. Decode throughput is unaffected by prompt length.
• Decode throughput is the average reported by the runtime after the response reached eos; the small variation reflects the differing response lengths.

Startup Runtime Footprint

The flash total breaks down as 337 MiB for the 19 .axmodel files, 320 MiB for model.embed_tokens.weight.bfloat16.bin, and 16 MiB for tokenizer.txt.

Package Layout

.
├── README.md
├── config.json
├── post_config.json
├── tokenizer.txt
├── model.embed_tokens.weight.bfloat16.bin
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── gemma3_text_p128_l0_together.axmodel
├── ... (gemma3_text_p128_l1..l17_together.axmodel)
├── gemma3_text_post.axmodel
└── gemma_3_270m_it_tokenizer/
    ├── config.json
    ├── tokenizer.json
    ├── tokenizer.model
    ├── tokenizer_config.json
    ├── generation_config.json
    ├── special_tokens_map.json
    ├── added_tokens.json
    └── chat_template.jinja

This package uses a flat runtime layout: the compiled text runtime files, the packed embedding weights, and tokenizer.txt live at the repository root so axllm can load the model directory directly. The bin/ directory holds the validated AX650 axllm binary (axllm.version.json records the source branch, commit, and checksum it was built from). The original Hugging Face tokenizer/config files are kept in the gemma_3_270m_it_tokenizer/ subdirectory for reference. Model weights (safetensors) are not shipped in this package.

Direct Inference with axllm

Download the Model Package

Download the release package from Hugging Face:

mkdir -p AXERA-TECH/gemma-3-270m-it
cd AXERA-TECH/gemma-3-270m-it
hf download AXERA-TECH/gemma-3-270m-it --local-dir .

Install axllm

This package already includes a validated bin/axllm binary for AX650 / NPU3, so no build step is required — just make it executable:

chmod +x ./bin/axllm

If you prefer to build or fetch axllm yourself, use one of the following instead.

Option 1: clone the repository and run the installer:

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

Option 2: install with a one-line command (default branch: axllm):

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Option 3: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated axllm binary from GitHub Actions: https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm Then run:

chmod +x axllm
sudo mv axllm /usr/bin/axllm

Run on the Board

The package root is already arranged for axllm, so no extra runtime path arguments are required. The commands below use the packaged bin/axllm binary.

Note: the command below assumes you run it from the parent directory of AXERA-TECH/gemma-3-270m-it. If you are already inside the package directory, use ./bin/axllm run . instead.

$ ./AXERA-TECH/gemma-3-270m-it/bin/axllm run AXERA-TECH/gemma-3-270m-it

# output log example:
09:13:02.036 INF Init:1658 | LLM init start
09:13:02.037 INF Init:1689 | attention config: layers=18 sliding=15 full=3 linear=0 sliding_window=512 ref_full_layer_idx=0
tokenizer_type = 3
huggingface tokenizer mode = space_replace_bpe
100% | ################################ |  21 /  21 [5.45s<5.45s, 3.86 count/s] embed_selector init ok
09:13:07.483 INF Init:1839 | max_token_len : 1023
09:13:07.483 INF Init:1842 | kv_cache_size : 256, kv_cache_num: 1023
09:13:07.483 INF init_groups_from_model:1205 | prefill_token_num : 128
09:13:07.483 INF init_groups_from_model:1430 | prefill_max_token_num : 640
09:13:07.483 INF Init:1941 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >> What is the capital of France?
09:33:11.826 INF Run:3048 | ttft: 66.24 ms
The capital of France is Paris.

09:33:12.197 NTC Run:3448 | hit eos,decode avg 19.03 token/s
prompt >>

Interactive CLI

axllm run opens an interactive prompt. Type a message after prompt >> and the model streams a reply until it hits eos. Additional validated examples on AX650 / NPU3:

prompt >> Who wrote Romeo and Juliet?
The play "Romeo and Juliet" was written by William Shakespeare.

prompt >> Write a haiku about autumn.
Leaves turn gold and red,
Crisp air whispers through the trees,
Sunlight paints the world in hues.

prompt >> List the first five prime numbers.
The first five prime numbers are 2, 3, 5, 7, and 11.

Use /reset to clear the KV cache, /pp to print the conversation history, and /q or /exit to quit.

Serve with axllm

axllm serve exposes the same model through an OpenAI-compatible HTTP API.

Note: the command below assumes you run it from the parent directory of AXERA-TECH/gemma-3-270m-it. If you are already inside the package directory, use ./bin/axllm serve . --port 8000 instead.

$ ./AXERA-TECH/gemma-3-270m-it/bin/axllm serve AXERA-TECH/gemma-3-270m-it --port 8000
# output log example:
Starting server on port 8000 with model 'AXERA-TECH/gemma-3-270m-it'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/gemma-3-270m-it

Check the service health and list the available models:

$ curl http://127.0.0.1:8000/health
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}

$ curl http://127.0.0.1:8000/v1/models
{
  "data": [
    {
      "id": "AXERA-TECH/gemma-3-270m-it",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}

Send a chat completion request:

$ curl http://127.0.0.1:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "AXERA-TECH/gemma-3-270m-it",
      "messages": [{"role": "user", "content": "What is the capital of France?"}],
      "max_tokens": 64
    }'

# Example response:
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The capital of France is Paris.\n",
        "role": "assistant"
      }
    }
  ],
  "model": "AXERA-TECH/gemma-3-270m-it",
  "object": "chat.completion"
}

If you want a browser UI for the OpenAI-compatible service, use AXERA-TECH/lite_webui. Set the OpenAI base URL to http://<board-ip>:8000 and the model name to AXERA-TECH/gemma-3-270m-it.

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

• Original Hugging Face model: google/gemma-3-270m-it
• AXERA conversion and deployment workflow: AXERA-TECH/gemma-3-270m-it.axera

When rebuilding for AX650 / NPU3, compile with FLOAT_MATMUL_USE_CONV_EU=1 pulsar2 llm_build ... --chip AX650. This build used --hidden_state_type bf16 --prefill_len 128 --kv_cache_len 1023.

Discussion

• GitHub Issues
• QQ group: 139953715