devstral-small-2-24b-instruct-2512

devstral-small-2-24b-instruct-2512

Derived from mistralai/Devstral-Small-2-24B-Instruct-2512. Weights revision 55c5b41e. This is that model's implementation on Tenstorrent hardware; see the original model card for training and evaluation details.

Devstral Small 2 is Mistral's 24B agentic coding model, tuned for software engineering work: exploring a repository, editing several files, and driving tools. This package serves its text decoder on Tenstorrent Blackhole through vLLM with tool calling enabled. It is an experimental community bring-up: text only, two devices minimum, and validated on a TT-QuietBox 2. Profiles p300, p150x4, p300x2.

Runs on p300 or p150x4 or p300x2 — see the serve profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Intended use

Direct use: Agentic coding over an OpenAI-compatible API — repository exploration, multi-file edits, and tool-driven workflows — for one user or a small batch on a single TT-QuietBox 2.

Out-of-scope use: Image or any non-text input: the Pixtral vision tower is not ported, so this serves the language model only. Also out of scope: a single-device deployment, which does not boot, and context beyond each profile's configured max_model_len, which was not validated.

Quickstart

uv tool install tenstorrent   # once — the Tenstorrent CLI, `tt`
tt model pull anirud/devstral-small-2-24b-instruct-2512
tt serve anirud/devstral-small-2-24b-instruct-2512

tt model pull (or tt-model pull --with-weights) downloads the Docker image and the mistralai/Devstral-Small-2-24B-Instruct-2512 weights at 55c5b41e98c2dbd21b0c8afffc540dcfc9eb5128 (into your HF cache; they are not in the image). tt serve (or tt-model serve) starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

Without tt-cli — tt-model alone does the whole job:

tt-model pull  anirud/devstral-small-2-24b-instruct-2512 --with-weights
tt-model serve anirud/devstral-small-2-24b-instruct-2512

Validated on a TT-QuietBox 2 (2 x Blackhole p300c, 4 devices), with 2, 4 and 4 devices opened for the p300, p150x4 and p300x2 profiles. Every profile was measured on this image; the figures are under Expected performance.

Serve profiles

One image serves every profile below; pick one with --profile.

Using it

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Devstral-Small-2-24B-Instruct-2512" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser mistral): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

A complete request, with the sampling recipe this checkpoint expects:

curl http://127.0.0.1:20000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistralai/Devstral-Small-2-24B-Instruct-2512",
    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],
    "temperature": 0.15
  }'

Tool calling is enabled with the Mistral parser. Send an OpenAI tools payload and the reply carries finish_reason: tool_calls with the arguments as JSON; return the result as a role: tool message to continue the exchange. This is not a reasoning model, so no reasoning parser is configured and no thinking block is emitted.

Expected performance

Evaluations

Run with lm_eval 0.4.10.dev0 (the Tenstorrent tstescoTT/lm-evaluation-harness evals-common fork) against this package's own serving stack on the p150x4 profile, sampling at the checkpoint's own recipe (temperature 0.15). No reference run was available, so these are absolute Tenstorrent numbers with no delta: the upstream card publishes only agentic results (SWE-Bench Verified 68.0%, SWE-Bench Multilingual 55.7%, Terminal Bench 2 22.5%), and lm_eval cannot run those harnesses.

Known quality finding

The two HumanEval rows are the same benchmark on the same build, 49 points apart, and the difference is protocol alone. Posed as a chat turn the model scores 88.4%. Asked to continue a bare function signature it scores 39.0% and sometimes answers in the wrong register — one completion returned # TODO: Implement this function. The instruct row is the fair number for an instruction-tuned model; the raw-completion row is the protocol HumanEval defines. Both are published because choosing one would be a presentation decision, not a measurement.

Against a CPU reference of this same checkpoint, short closed-form answers match exactly, while free-form code diverges token-for-token and stays functionally correct — a bfloat8_b device will not reproduce an fp32 host run verbatim in open-ended generation.

One defect was found and fixed during bring-up and is closed in this image: requests above 1,024 input tokens killed the engine, because empty multimodal keyword arguments reached a text-only prefill signature. Inputs from 128 to 8,192 tokens now pass a standing regression, and a live client session has driven it at 9,900-token prompts.

Sampling note for evaluators

The checkpoint's generation_config.json sets temperature 0.15 and the upstream card uses that value in every example; it is the recipe above. This package serves with sample_on_device_mode: all, so sampling runs on the device rather than the host.

Performance methodology

Closed-loop latency sweep of every profile, measured 2026-10-07 on a TT-QuietBox 2 (2 x Blackhole p300c, 4 devices) with 2, 4 and 4 devices opened for p300, p150x4 and p300x2, served from this image with tt-model serve and driven by vllm bench serve 0.26.0 inside the container. Prompts are random token ids of exactly ISL tokens; output is pinned to OSL tokens with ignore_eos, so every row holds the batch full for its whole length — these are sustained-rate numbers, not natural-conversation ones. One unrecorded warm-up request precedes each input length. N = 4 x users up to 4k input, 2 x users up to 16k, and users beyond. TTFT is the time to the first streamed token; TPOT = (E2EL - TTFT) / (OSL - 1). Values are medians over the row. A 300 s median-E2EL fence was configured to stop the ladder; no row reached it — the slowest measured median E2EL was 277.3 s, at 32,768 input and 32 users on p300x2. Characterization only, no configured targets. First boot compiles kernels and takes about 360 s; that is startup, not inference, and is excluded from every row below.

Column key: ISL/OSL = input/output tokens; Users = concurrent streams; N = prompts in the row; TTFT = time to first token; TPOT = time per output token; Decode = output tokens per second per user; Output = output tokens per second across all users; E2EL = end-to-end latency.

p150x4 — (1, 4) mesh, 65,536-token context; 20 of 20 planned rows measured

p300x2 — (1, 4) mesh, 65,536-token context; 30 of 30 planned rows measured

p300 — (1, 2) mesh, 32,768-token context; 12 of 12 planned rows measured

Limitations

The Pixtral vision tower is not ported. This package serves the text decoder only; image inputs are not supported and no multimodal endpoint is exposed.

There is no single-device profile, and two devices is the minimum. The weights and KV cache fit in one 32 GB device, but a prefill matmul overruns L1 by about 56 KB: tt_transformers picks the same MLP prefill grids on one device as on four, so one device's cores carry the whole MLP instead of a quarter of it. The configuration was attempted, did not boot, and is deliberately not declared.

max_model_len is a configured serving limit, not the model's. The upstream checkpoint advertises a 256k context; this package configures 32,768 tokens on p300 and 65,536 on p150x4 and p300x2. Longer contexts were not validated here.

All three declared profiles were measured on this image, and every row in the performance tables above completed. Accuracy evaluations were run on p150x4 only.

The image cannot be rebuilt from public source. The tt-metal branch it was built from lives on the build host and has not been pushed, so provenance below reports the commit as not published rather than linking it. The contents are shipped in code/ and are byte-identical to the model code inside the image.

tt-model info cannot resolve a local bundle directory — it accepts only a Hugging Face repo id — so inspecting a staged package before publishing is not possible with that command.

Risks and safety considerations

Two numerical paths in this port differ from a reference implementation and could change results without raising an error. The FP8 E4M3 checkpoint is dequantised to bfloat16 at load using its per-tensor scales, and the Llama-4 style query scaling is computed on the device. Both were checked against a host reference during bring-up — the query scale matched at every position tested, and the dequantisation was asserted against an independent implementation — but accumulated bfloat16 drift over long generations is not something those checks bound.

Free-form generation is not reproducible token-for-token against a host run of the same checkpoint, as described under Known quality finding. Treat exact-output comparisons against a CPU or GPU reference as expected to differ; compare behaviour, not tokens.

Licensing

The weights are Apache-2.0, as declared by the upstream repository; the upstream repository is not gated and ships no separate LICENSE file. The tt-metal model code in code/ is Apache-2.0. No non-commercial or redistribution restriction applies to either.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/anirud/devstral-small-2-24b-instruct-2512/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image: