Qwen3-4B-Instruct-2507-REFFT

Qwen3-4B-Instruct-2507-REFFT

Qwen3-4B-Instruct-2507 for the Hexagon NPU of the Qualcomm Dragonwing IQ-9075 (QCS9075), run by refft. W4A16: 4-bit weights, 16-bit activations.

Files

Qwen3-4B-Instruct-2507-w4a16.refft   # the model bundle: DSP module + weights (RFTB, ~3.1 GB)
refft-cli                    # the runtime (aarch64 Linux)
run_qwen3_w4a16.sh                    # runs it on the board
qwen3_prompt.py, tokenizer.json       # prompt text <-> token ids (Qwen3's tokenizer)
kernel/                               # the refft A1 kernel: Image, installer, patch, config

Requirements

• A Qualcomm IQ-9075 EVK (QCS9075) running Qualcomm Linux, as root; the model runs on CDSP1.

• python3 with the tokenizers package (for your own prompts; the benchmark prompt needs only python3).

• The refft A1 kernel (in kernel/): the runtime maps its DSP buffers only in 2 MiB SMMU blocks, which the A1 kernel's DMA heap provides; on another kernel the first allocation fails. It is the board's vendor kernel with one patch, installed beside the stock one (kernel/README.md):

  cd kernel && chmod +x install_a1_kernel.sh
  ./install_a1_kernel.sh --try && reboot     # boots it once
  ./install_a1_kernel.sh --default           # then keep it
  

Test it yourself

On the board:

hf download refinefuture-ai/Qwen3-4B-Instruct-2507-REFFT --local-dir qwen3-refft
cd qwen3-refft
chmod +x run_qwen3_w4a16.sh
./run_qwen3_w4a16.sh                                   # the benchmark prompt
./run_qwen3_w4a16.sh --prompt "Explain gravity in detail." --max_new_tokens 128

With no arguments it runs the benchmark prompt ("What is gravity? Keep the answer under ten words.") and checks the exact expected answer, "Gravity is a force that attracts objects toward each other." ([pipeline] PASS). Each run prints TTFT (prompt to first token) and TPS (decode tokens per second); --repeat 3 runs it three times in one load.

To call the runtime directly:

./refft-cli --bundle=Qwen3-4B-Instruct-2507-w4a16.refft \
  --prompt_ids=prompt.i32 --max_new_tokens=128 --eos=151645 --context=288 \
  --vector_workers=4 --dma_engines=0 --matrix_workers=1 --device_id=1

--prompt_ids takes little-endian int32 token ids (qwen3_prompt.py encode writes them); the DSP module is written once into --module_cache (default /tmp/refft_bundle) and loaded from there.

Performance

Measured on a Qualcomm IQ-9075 EVK, CDSP1 (Hexagon v73), against Genie (QAIRT 2.47.0) on the same board with the same prompt token ids, each system alone. prefill = prompt tokens, decode = generated tokens (12).

refft also runs past Genie's 4096-token context (decode 128): 8,192 tokens TTFT 16.9 s / 8.92 TPS, 32,640 tokens TTFT 226 s / 3.31 TPS.

Built from refft commit 48fb9603c (the DSP module and refft-cli).