Qwen2-VL-2B

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Qwen2-VL-2B-Instruct — LiteRT-LM (on-device Vision-Language Model)

Qwen/Qwen2-VL-2B-Instruct converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime — the first Qwen2-VL-family VLM in this format.

Qwen2-VL is Alibaba's widely-used general-purpose vision-language model: a dynamic-resolution ViT vision encoder feeds the Qwen2-1.5B language decoder, giving strong general VQA, description, and document/OCR reading in a phone-sized package. This bundle runs the whole thing through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully on-device.

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/Qwen2-VL-2B Qwen2-VL-2B.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Two cats are sleeping on a pink couch with two remote controls on the couch.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

Device-verified on a Pixel 8a (Google AI Edge Gallery, vision on GPU + decoder on CPU) and on the desktop LiteRT-LM runtime (macOS CPU):

• General description / VQA (photo, on-device): accurate and detailed grounding — an Ansel-Adams-style landscape → "a striking black-and-white photograph … an expansive river winds through a lush, forested valley … to the left … a steep, rocky hillside with rocky cliffs … low hills and mountains, covered in snow, emphasizing the stark nature of the wilderness".
• Document OCR (Read all the text in this image., macOS): perfect transcription of a synthetic report page — every figure, the e-mail address and the phone number.
• Counting / reading tables works (e.g. "there are 5 products listed"); ranking table cells across rows is the one weak spot — see the M-RoPE note below.
• Vision tower: static-rewrite vs the reference implementation corr 1.0 (fp32), 0.90 at int8, zero FLEX/CUSTOM ops (GPU-clean); the int8 vision is functionally accurate on both VQA and OCR above.
• Decoder: extracted as a standalone Qwen2 model — bit-exact fp32 logits vs the original; int4 is the same blockwise-32 + OCTAV recipe used across the shipped Qwen3/Qwen2 LLMs.

M-RoPE note & one known limitation. Qwen2-VL's decoder uses 3-D M-RoPE. The LiteRT-LM fast_vlm runtime supplies plain sequential (1-D) positions — for text tokens this is mathematically identical, and it preserves description, general VQA, counting, and OCR with no visible loss (A/B-verified against true M-RoPE). The one casualty is cross-cell comparison reasoning over 2-D structures — e.g. "which row of this table has the highest value": the 1-D positions flatten the table's 2-D layout, so the model can read every cell correctly (OCR is perfect) but may pick the wrong cell when asked to compare across rows/columns. This is inherent to the runtime (it has no M-RoPE), not the base model or the quantization — the fp32 base with true M-RoPE answers such questions correctly. Use it for reading and describing tables; don't rely on it to rank cells.

General-purpose VLM. Unlike an OCR-specialist, Qwen2-VL answers open questions about an image (describe, count, read, reason) as well as extracting text. Ask it anything about the picture — with the 2-D-table-ranking caveat above.

One image per chat. Like the other fast_vlm bundles, send each image in a fresh conversation — a second image in the same chat degrades (context bleed from the first turn was observed on CPU).

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-25.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in main; the runtime accepts that split.

The gate prompt carries no image, so this covers engine creation and the text path. The vision path on the GPU is not measured here.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm). Load Qwen2-VL-2B.litertlm with the vision tower enabled (Modality.textImage), attach a photo, and ask a question.

Vision-only bundle (no audio tower): bring the engine up with the vision modality only — requesting .all fails at session creation on bundles without an audio section.

Run on Android — Google AI Edge Gallery

Install a recent Google AI Edge Gallery, download Qwen2-VL-2B.litertlm, import it (tap +, enable "Support image"), attach an image and ask.

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,672,672,3]→[1,2304,1280]) + VISION_ADAPTER ([1,2304,1280]→[1,576,1536]) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
• Static rewrite of the dynamic-res vision tower: Qwen2-VL's ViT is native-resolution (packed patches, 2-D rope, cu_seqlens) and does not torch.export. The static graph fixes 672×672 and uses full attention (single image = one sequence) with precomputed 2-D rope.
• Conv3d → Conv2d fold. The patch-embed is a Conv3d (temporal_patch_size=2); for a single image the processor duplicates it into the 2 temporal frames, so the Conv3d over 2 identical slices equals a Conv2d with the summed temporal kernel w[:,:,0]+w[:,:,1] — GPU-safe, no Conv3d op.
• No GATHER_ND (this is what lets it run on the phone GPU). The obvious way to reorder raster patches into the merger's 2×2-block order is a gather — but that emits a GATHER_ND op, which the mobile GPU delegate cannot compile, so the vision executor fails to create and the whole engine won't load. Instead, patches stay in raster order through the encoder (full attention is permutation-equivariant, so ordering is irrelevant as long as each patch carries its own 2-D rope), and the 2×2 merge is done in the adapter with 4 strided slices + concat (f[:,0::2,0::2] … f[:,1::2,1::2]), all ops ≤4D. Static-rewrite corr vs the reference stays 1.0.
• Decoder: the Qwen2-1.5B text model inside the VLM is re-hosted as a standalone Qwen2ForCausalLM (state-dict 1:1, bit-exact logits, lm_head tied to embeddings) and exported with the standard litert-torch path, cache 4096.

2026-08-28 — start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token — so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: You are a helpful assistant.. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in Qwen2-VL-2B.litertlm now emits the block exactly once when no system message is given. In Qwen2-VL-2B.litertlm, the block is not emitted when you pass a system message. The restored block adds 11 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer in Qwen2-VL-2B.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (é, ñ, ü, °, ·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and any character without a whole-character vocabulary entry (emoji, most of Latin Extended-A) became the token the conversion had reused as UNK — the end-of-text token for this vocabulary. Qwen2-VL-2B.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 — chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)

The chat template in Qwen2-VL-2B.litertlm matched history turns only on role 'model', so a conversation created through the Conversation API with the standard role 'assistant' (for example create_conversation(messages=[...]) restoring a chat) silently dropped every assistant turn from the rendered prompt — the model saw consecutive user turns with its own replies gone. The template's role conditions now also accept 'assistant'. The engine's own send_message flow was never affected.

Template-only change: every section of the bundle except the metadata is byte-identical to the previous file (verified by per-section sha256), and single-turn renders, token ids, prefill counts and greedy answers are unchanged (verified on the LiteRT-LM runtime), so the numbers on this card still describe this file — only the file's own sha256 differs. Re-download only matters if you inject conversation history with role 'assistant'. Details: google-ai-edge/LiteRT-LM#3417 (https://github.com/google-ai-edge/LiteRT-LM/issues/3417).