llama-3.2-3b-instruct-executorch-qnn-htp

Llama-3.2-3B-Instruct — ExecuTorch QNN/HTP (SM8650 + SM8750)

Two ExecuTorch .pte artifacts of Llama-3.2-3B-Instruct compiled for the Qualcomm Hexagon NPU (HTP).

Not yet benchmarked. These are uploaded as a backup of a completed host export. No 160-request device measurement exists for either SoC yet, so this card contains no latency numbers. The local copies are being kept.

SoC mapping

• SM8650 = Qualcomm Snapdragon 8 Gen 3, Qualcomm Hexagon HTP v75 (the Redmi K70 Pro platform).
• SM8750 = Qualcomm Snapdragon 8 Elite, Qualcomm Hexagon HTP v79 (the Realme RMX5090 platform).

Artifacts

Both carry the FlatBuffer identifier ET12 at bytes 4–7. The SM8650 file was additionally parsed with the generated schema: identifier valid, 75 execution plans, 5 segments.

The filename hybrid_llama_qnn.pte is upstream's name for --model_mode hybrid; here the model genuinely is Llama.

Export parameters

Exact launcher: export_executorch_llama32_3b_host.sh (included in this repo).

Quantization / calibration

Note on the environment: lm_eval 0.4.5 is unusable in this host env, so the --calib_tasks wikitext route cannot work here; calibration is fed from the parquet file directly instead. This is why --calib_parquet* is used rather than the task-based flags.

Note on filenames seen during export

The SM8650 compile emitted temporary AIMET files whose names contain Qwen3_4B_Instruct_2507. That is a naming artifact of the shared quantization harness — both the invocation and the log identify llama3_2-3b_instruct with Llama3_3BQuantRecipe. The produced PTE is Llama-3.2-3B.

Reproduction

python executorch/examples/qualcomm/oss_scripts/llama/llama.py \
  --artifact <out> --build_folder <android-build> \
  --soc_model SM8750 --backend htp \
  --decoder_model llama3_2-3b_instruct \
  --checkpoint <Llama-3.2-3B-Instruct>/original/consolidated.00.pth \
  --params <Llama-3.2-3B-Instruct>/original/params.json \
  --tokenizer_model <Llama-3.2-3B-Instruct>/original/tokenizer.model \
  --model_mode hybrid --prefill_ar_len 128 \
  --max_seq_len 2048 --max_context_len 2048 --temperature 0 \
  --calib_parquet <wikitext-2-raw-v1-train.parquet> \
  --calib_parquet_column page --calib_parquet_limit 128 \
  --prompt "Reply with one word." --compile_only

The export peaks around 80 GiB host RSS and writes a ~19.9 GiB QDQ intermediate plus a ~34 GiB compiler temp tree; point TMPDIR at fast storage with room for both. Verify the result by checking bytes 4–7 for ET12 — not bytes 0–3, which is the FlatBuffer size prefix, not the identifier.

On device, ADSP_LIBRARY_PATH must include the runner directory alongside the platform DSP paths or QNN device creation fails with 14001. Launch under setsid nohup so the device process survives host-side adb teardown.

Scope

Intended for a serial-arrival, single-request latency comparison across on-device inference frameworks: batch size 1, no continuous batching, no speculative decoding. Accuracy was not evaluated.