ministral-3-3b-instruct-genai-int4

Ministral 3 3B Instruct — ONNX Runtime GenAI, int4

This is a format conversion, not a fine-tune. Nothing was trained. The weights are mistralai/Ministral-3-3B-Instruct-2512, converted to an ONNX Runtime GenAI model directory and quantised to int4 so a desktop can load it without PyTorch. If you want the model, use the original; this exists because Amber Linux's broker loads genai directories and there was no such build to fetch.

Amber Linux ships the fine-tune of this model, ministral-3-3b-amberlin-tools, and this conversion is its untuned baseline: the "untuned" row of every comparison on that card is this build, on the same split and tool server.

Mistral publishes an ONNX export of this model, but in the transformers.js layout — decoder_model_merged, embed_tokens, no genai_config.json — which onnxruntime-genai does not load. That is the gap this fills.

What is in it

Built with the onnxruntime_genai.models.builder from the version this directory's genai_config.json records. Convert and run with the same genai version: a builder ahead of the runtime writes a config the runtime cannot load, and it says so in a way that reads like a broken model.

Both builds come from the base repo's own FP8 checkpoint, one command each:

python -m onnxruntime_genai.models.builder \
  -m mistralai/Ministral-3-3B-Instruct-2512 -o cuda -p int4 -e cuda \
  --extra_options prune_lm_head=true
python -m onnxruntime_genai.models.builder \
  -m mistralai/Ministral-3-3B-Instruct-2512 -o cpu -p int4 -e cpu \
  --extra_options prune_lm_head=true

prune_lm_head is the option that matters. Without it the graph returns logits for every prompt position, [batch, sequence, 131072], and the runtime holds an fp32 copy of them: about 512 KiB per prompt token on top of the KV cache, all of it discarded, since generation reads only the last position. Pruned, the graph returns [batch, 1, 131072]. The answers are byte-identical and the speed is the same; a turn's memory no longer grows with its prompt. On the CUDA build a tool turn's peak falls from 5,306 to 3,910 MiB at a 2,048-token context, and from 8,392 to 4,542 MiB at 16,384.

That version has no native upstream release, only Python wheels — so pip install onnxruntime-genai-cuda runs it, and for anything that is not Python, amberlin-runtime publishes a linux amd64 build of the pinned commit as Debian packages.

Two builds, one tokenizer

cpu/ and cuda/ each hold a complete graph, its weights and its own genai_config.json. The tokenizer, the tokenizer config and the chat template are byte-identical between them and sit at the root, so they are stored once.

Take the root files and one directory, and flatten it — genai loads a single directory and will not look into a subfolder:

hf download hyperquader/ministral-3-3b-instruct-genai-int4 --local-dir m \
  --include 'cuda/*' --include 'tokenizer*' --include 'chat_template.jinja'
mv m/cuda/* m/ && rmdir m/cuda

Repeat --include per pattern. Several patterns after one flag are accepted and the first is then silently ignored, which downloads a directory that looks complete and will not load.

A CUDA build does load on a CPU if provider_options is emptied, which makes the two look interchangeable. They are not: on the same machine that path runs at 21 tok/s against the CPU build's 33. The graphs are quantised differently, which is also why the CPU build is the larger of the two.

Measured here

Greedy decode, 128 tokens, best of three runs, onnxruntime-genai 0.16.0:

Each row is its own build, not one artefact measured twice.

Tool calling, without any training

A call is [TOOL_CALLS]name[ARGS]{…} — both markers are single special tokens, so the arguments are a plain JSON object and a call ends where that object balances.

Scored on 119 held-out prompts with the 13-tool catalogue in the prompt and generation constrained by a grammar built from the tool schemas. Calls are checked against the catalogue's schemas, not sent to a server:

Two things that number depends on, stated because they are easy to miss. The grammar constrains form, never judgement: it makes a malformed call and an out-of-enum argument impossible to emit, and it does not make the model choose correctly. And the catalogue is doing the work — offered no catalogue, this model takes the free-text branch on every prompt that wanted a call. A grammar is not a substitute for telling a model the tools exist.

The remaining failures are mostly arguments rather than routing, and a measurable part of them are disagreements with the reference answer rather than mistakes.

Licence and attribution

Apache 2.0, inherited from the base model. The weights are Mistral AI's work; this repository contributes the conversion and the measurements. Cite mistralai/Ministral-3-3B-Instruct-2512 for the model itself.