SmolLM3-3B-int8-cw-ov

SmolLM3-3B — OpenVINO INT8 channel-wise (NPU-safe)

INT8 channel-wise symmetric OpenVINO export of HuggingFaceTB/SmolLM3-3B. The quality-first companion to the int4-cw build: channel-wise int4 is the lossiest int4 variant (the NPU can't compile group-quantized int4 — see the int4-cw card), so if you have the memory and can take ~half the decode speed, int8 keeps more of the model.

Export command

optimum-cli export openvino --model HuggingFaceTB/SmolLM3-3B \
  --weight-format int8 --sym --group-size -1 <output_dir>

Toolchain: OpenVINO 2026.1.0, optimum-intel 1.27.0, transformers 4.57.6. Verified on openvino-genai 2026.3.0 runtime.

Measured (Intel Core Ultra 9 285K, NPU driver 32.0.100.4778)

Note: int8-sym works on SmolLM3 but is NOT a general NPU recipe — on LFM2-family models it produces garbage output (and int8-asym runs correct but ~1 tok/s on NPU). Test per-architecture before assuming.

Usage

import openvino_genai as og
pipe = og.LLMPipeline("SmolLM3-3B-int8-cw-ov", "NPU", MAX_PROMPT_LEN=4096)
print(pipe.generate("Hello!", og.GenerationConfig(max_new_tokens=64)))

Or with NoLlama: python nollama.py --model-dir ~/models/SmolLM3-3B-int8-cw-ov.