SmolLM3-3B-int4-cw-ov

SmolLM3-3B — OpenVINO INT4 channel-wise (NPU-safe)

INT4 channel-wise symmetric OpenVINO export of HuggingFaceTB/SmolLM3-3B, built specifically so it compiles on Intel NPUs.

Why channel-wise? (read this if your int4 model crashes the NPU)

The default optimum-cli ... --weight-format int4 (group-quantized, gs128) produces IRs that crash the Intel NPU driver compiler:

[vpux-compiler] StopLocationVerifierPass Pass failed : Found N duplicated names
ZE_RESULT_ERROR_INVALID_NULL_POINTER

Channel-wise symmetric quantization avoids the bug entirely. This is the same layout Intel uses for its own *-int4-cw-ov NPU models.

Export command

optimum-cli export openvino --model HuggingFaceTB/SmolLM3-3B \
  --weight-format int4 --group-size -1 --sym --ratio 1.0 <output_dir>

Toolchain: OpenVINO 2026.1.0, optimum-intel 1.27.0, transformers 4.57.6. Verified on openvino-genai 2026.3.0 runtime.

Measured (Intel Core Ultra 9 285K, NPU driver 32.0.100.4778)

An int8 channel-wise build (better quality headroom, ~half the decode speed) is published alongside this one.

Usage

import openvino_genai as og
pipe = og.LLMPipeline("SmolLM3-3B-int4-cw-ov", "NPU", MAX_PROMPT_LEN=4096)
print(pipe.generate("Hello!", og.GenerationConfig(max_new_tokens=64)))

Or with NoLlama (OpenAI/Ollama-compatible server, NPU-first): drop the folder in ~/models and python nollama.py --model-dir ~/models/SmolLM3-3B-int4-cw-ov.