Llama-3.1-8B-vknn

Llama-3.1-8B-Instruct — VKNN .vxm (GPU, on-device)

Built with Llama. Llama-3.1-8B-Instruct compiled to VKNN's .vxm format for on-device GPU inference via the Vulkan backend.

The decoder ingests the prompt token-by-token, then decodes with the engine-resident KV cache; stop on <|eot_id|> (id 128009). Verified on device: greedy decode stays coherent with 0 CPU fallback — every node in the plan runs on the Vulkan backend. The peak GPU working set is ~6.4 GB, so an 8B fits on a phone-class GPU with ~7 GB free.

This is a GPU build — every tensor-compute op runs on the Vulkan backend. The int4 weights are imported directly from the ONNX Runtime MatMulNBits export (no re-quantization).

Usage

vknn_chat llama-3.1-8b-instruct-int4.vxm --backend vulkan --precision low --temp 0 --eos 128009

Feed whitespace-separated prompt token ids on stdin, formatted with the Llama-3 chat template (<|begin_of_text|><|start_header_id|>...); generated ids stream to stdout. See the VKNN LLM guide.

License

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Use of this derivative is subject to that license and Meta's Acceptable Use Policy.