Llama-3.2-3B-Instruct-RKLLM-v1.2.3

Llama-3.2-3B-Instruct — RKLLM v1.2.3 (w8a8, RK3588)

RKLLM conversion of meta-llama/Llama-3.2-3B-Instruct for Rockchip RK3588 NPU inference.

Converted with RKLLM Toolkit v1.2.3. This is a standard instruct model — it does not produce <think> reasoning blocks.

Key Details

Why This Model?

Llama 3.2 3B Instruct is Meta's latest compact instruction model. It brings a different architecture and training lineage to the RK3588 NPU lineup — strong at instruction following, coding, and general reasoning without the overhead of a thinking/reasoning mode.

At ~3B parameters it sits between the 1.7B and 4B Qwen3 models, offering a useful middle ground.

Hardware Tested

• Orange Pi 5 Plus — RK3588, 16GB RAM, Armbian Linux
• RKNPU driver 0.9.8
• RKLLM Runtime v1.2.3

Usage

With the official RKLLM API demo

# Clone the runtime
git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo

# Run (aarch64)
./build/rkllm_api_demo /path/to/Llama-3.2-3B-Instruct-rk3588-w8a8.rkllm 4096 8192

With the RKLLM API Server

Download and place in ~/models/:

mkdir -p ~/models/Llama-3.2-3B-Instruct
cd ~/models/Llama-3.2-3B-Instruct
git lfs install && git clone https://huggingface.co/GatekeeperZA/Llama-3.2-3B-Instruct-RKLLM-v1.2.3 .

The server auto-detects the model as instruct capability (no thinking). Use with GatekeeperZA/RKLLM-API-Server.

Conversion Script

from rkllm.api import RKLLM

llm = RKLLM()
llm.load_huggingface(model="meta-llama/Llama-3.2-3B-Instruct", device="cpu", dtype="float32")
llm.build(
    do_quantization=True,
    optimization_level=0,
    quantized_dtype="w8a8",
    quantized_algorithm="normal",
    target_platform="rk3588",
    num_npu_core=3,
    max_context=8192,
)
llm.export_rkllm("./Llama-3.2-3B-Instruct-rk3588-w8a8.rkllm")

WSL2 note: Requires ≥16GB WSL2 memory (memory=16GB in ~/.wslconfig). Write the output to a native Linux path (/home/user/) first, then copy to /mnt/ — writing directly to the Windows mount triggers OOM during the export phase.

File Listing

Compatibility Notes

• Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
• RKNPU driver: ≥ 0.9.6
• SoCs: RK3588 / RK3588S (3 NPU cores). Not compatible with RK3576 (2 cores) without reconversion.
• RAM: ~3.5GB loaded. Runs comfortably on 8GB+ boards.

Acknowledgements

• Meta / FAIR for the Llama 3.2 base model
• Rockchip / airockchip for the RKLLM toolkit and runtime
• Converted by GatekeeperZA