SmolLM2-360M-Instruct-ort-genai-int4-cpu

SmolLM2-360M-Instruct: ONNX Runtime GenAI, int4, CPU

An int4 ONNX export of HuggingFaceTB/SmolLM2-360M-Instruct for ONNX Runtime GenAI on the CPU execution provider.

All model weights and training are by the Hugging Face SmolLM team. This repo contains only a converted, quantized copy. It is not a new model and it was not fine-tuned.

Who it is for

Developers who want to try a small instruct model with ONNX Runtime GenAI on a plain CPU, without running the conversion themselves. It is a test artifact, not a model for production use.

Status and limits

• Experimental. No quality evaluation of this quantization is published. Expect some quality loss compared with the original fp16/bf16 model. Check outputs on your own task before you rely on it.
• The base model "primarily understands and generates content in English" and its output may be inaccurate or biased (base model card, Limitations).
• The context length set in genai_config.json is 8192.
• I made this conversion while building xllama, local LLM inference on Xbox Series S|X. xllama is an independent project, not affiliated with Microsoft or Hugging Face.
• This repo matches xllama's catalogue entry smollm2-360m-cpu-int4 in base model, int4 precision and ORT CPU runtime. It is not the file xllama ships, so xllama's throughput figures do not describe this repo.

Files

model.onnx + model.onnx.data (int4 weights), genai_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja (ChatML).

How to run

pip install onnxruntime-genai huggingface_hub
hf download gianlucamazza/SmolLM2-360M-Instruct-ort-genai-int4-cpu --local-dir smollm2-int4

import onnxruntime_genai as og

model = og.Model("smollm2-int4")
tokenizer = og.Tokenizer(model)
prompt = "<|im_start|>user\nWhat is the capital of Italy?<|im_end|>\n<|im_start|>assistant\n"

params = og.GeneratorParams(model)
params.set_search_options(max_length=256)
generator = og.Generator(model, params)
generator.append_tokens(tokenizer.encode(prompt))
stream = tokenizer.create_stream()
while not generator.is_done():
    generator.generate_next_token()
    print(stream.decode(generator.get_next_tokens()[0]), end="", flush=True)

License

Apache-2.0, the same as the base model. Keep the base model's attribution when you redistribute.

Credits

• Base model: SmolLM2 by Hugging Face (Allal et al., 2025, SmolLM2: When Smol Goes Big).
• Conversion: Gianluca Mazza (GitHub). I take LLM systems into production: state, recovery, eval, cost.