SmolVLM2-500M-ExecuTorch

SmolVLM2-500M — ExecuTorch

smolvlm2_500m_video_instruct_xnnpack_8da8w.pte (518 MB)

• Source: HuggingFaceTB/SmolVLM2-500M-Video-Instruct — a SigLIP vision tower (768 wide, 12 layers) and a Llama decoder (960 wide, 32 layers)
• License: Apache-2.0
• Input: a 512×512 picture as [1, 3, 512, 512], and token ids for the words around it
• Output: logits over the 49,280-token vocabulary

One file, three entry points — the shape ExecuTorch's multimodal runner asks for:

A decoder that only takes token ids cannot be told about a picture. Splitting a vision-language model into a vision .pte and a text .pte runs aground there; MultimodalPrefiller::load asks one module for token_embeddings and text_model and uses vision_encoder if it finds it.

vision_encoder here takes a picture rather than a patch sequence, which is what that runner hands it — so this file can be driven by the runner as well as method by method.

Verification (Mac arm64, 2026-08-21)

The wrappers themselves are exact: run in eager without quantization they agree with the model at corr 1.000000, so what these numbers measure is the weights, not the wiring.

Why the decoder is int8 and not int4

The vision tower is int8 per output channel and the decoder is int8 over groups of 32. int4 on a decoder this size does not hold:

int4 keeps the first word and then drifts. Only the int8 build is here. Those two columns were measured before the embedding table was quantised, when this file was 657 MB.

Ask for one tile

The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph becomes seventeen of them — 1088 image tokens, against the 64 this graph produces and the 512 of context it was built with. do_image_splitting = False on the processor (or its image processor) gives the one tile that matches.

VL_CKPT=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
VL_PTE=smolvlm2_500m_video_instruct_xnnpack_8da8w.pte \
python convert/run_vl.py <image> "What is in this picture?"

The driver builds the prompt with the model's own processor and writes the picture's rows wherever the processor put an image token, which is what the model does internally.

It invents text on signs

Asked about a London street, this model answers with a shop name that is not there. That is the model and not the conversion: eager, unquantized, invents a different one from the same photograph. If reading signs matters, LFM2.5-VL reads them (450M).

Conversion

convert/export_vl_bundle.py. Two things needed re-authoring:

• The position embeddings are chosen with torch.bucketize, so that a picture filling part of the grid still lands on the right ones. There is no bucketize kernel in the runtime, and a full square grid has one answer anyway: patch i takes position i. The export checks that against the model's own code rather than assuming it.
• The cache has to live inside text_model. ExecuTorch copies a mutable buffer into each method that names it (Program::load_mutable_subsegment_into writes into the method's own memory), so a prefill method and a decode method would each get their own and neither would see the other's writes. One graph serves both. StaticCache does not survive lowering, and neither does a cache held by any object outside the module tree — its tensors get lifted a second time as constants, and run_decompositions then returns a function where a GraphModule was expected.

The number that decides whether it runs on a phone

CONTEXT, the upper bound on the dynamic sequence dimension. The memory planner sizes its arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and iOS kills the process with signal 9 before the first method has finished loading. One picture is 64 rows here, so 512 leaves room for a long question and a long answer.

(conversion scripts: executorch-models · iOS sample: executorch-samples)

The token embedding table

The table is an nn.Embedding, and the quantisation filter in convert/export_vl_bundle.py was written as isinstance(module, nn.Linear), so it went out in fp32 — 189 MB of a 657 MB file. tie_word_embeddings does not save it either: quantize_ swaps lm_head.weight for a new tensor and the embedding keeps pointing at the old fp32 one. (LFM2.5-VL ties and SmolVLM2 does not; both leaked.)

It is now int8 over groups of 64 along each row, written out by hand because torchao's IntxWeightOnlyConfig reaches an nn.Embedding but will not lower (Missing out variants: {'torchao::dequantize_affine'}). On this model the first-step logits improve slightly — corr 0.99536 against 0.99406 for the fp32 table — while the twelfth greedy token differs where the fp32-table build matched all twelve. The file drops from 657 MB to 518 MB. Both numbers are here rather than the flattering one.

One scale per row is not enough for a narrow table: on SmolVLM2-256M, whose rows are 576 long, per-row scales moved the first-step logits from corr 0.99197 to 0.97811 and flipped the top-1 token, so that model keeps its fp32 table. Groups of 64 are what make this one free.