LFM2.5-2.6B-openvino-int8-npu

LFM2.5-2.6B - OpenVINO INT8 (Intel AI Boost / NPU)

OpenVINO IR export of LiquidAI/LFM2.5-2.6B quantized to INT8 for Intel AI Boost NPU.

The sibling INT4 export (mosesman/LFM2.5-2.6B-openvino-int4-npu) compiled on NPU but crashed on the first generate() with OpenVINO 2026.3 (invalid unordered_map key). This INT8 export generates correctly on NPU with OpenVINO 2026.4 nightly (verified 2026.4.0.dev20260818 on Core Ultra 7 256V).

Requirements

• OpenVINO / openvino-genai 2026.4+ (nightly until 2026.4 ships)
• NPU driver 32.0.100.3104 or newer

pip install --pre openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly

Quick start

import openvino_genai as ov_genai
from transformers import AutoTokenizer

model_id = "mosesman/LFM2.5-2.6B-openvino-int8-npu"
device = "NPU"  # or "GPU" / "CPU"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 17*19? Reply with only the number."}],
    tokenize=False,
    add_generation_prompt=True,
)

pipe = ov_genai.LLMPipeline(model_id, device)
config = ov_genai.GenerationConfig()
config.max_new_tokens = 128
config.do_sample = False  # NPU: greedy only

print(pipe.generate(prompt, config))

Notes

• First NPU compile can take ~1 minute; later loads use the driver/blob cache.
• On Lunar Lake, GPU decode is typically faster than NPU for this architecture (iGPU has higher AI TOPS). NPU is the better choice for power / leaving the GPU free.
• License: LFM 1.0 (same as the base model).