InternVL3-2B

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 34.4 tok/s · prefill 573 tok/s · TTFT 380 ms · all 1243 ops delegated (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 43.7 tok/s · prefill 208 tok/s · TTFT 1.17 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/internvl3-2b/CARD.md

InternVL3-2B — LiteRT-LM (on-device Vision-Language Model)

OpenGVLab/InternVL3-2B converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models, and the same runtime that runs litert-community/FastVLM-0.5B).

InternVL3-2B is a compact vision-language model: an InternViT vision encoder + pixel-shuffle + MLP projector feeding a Qwen2.5-1.5B language decoder. This bundle runs it through LiteRT-LM's fast_vlm multimodal path — give it an image and a question, get a grounded answer, fully on-device.

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/InternVL3-2B InternVL3-2B.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Two cats are sleeping on a pink blanket.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

The iPhone rows are carried over from this repository's own earlier on-device note, where the model loads in about 3–4 s; the run logs are not retained here, so the run count and prompt are not known. Single-image VQA is what the GPU row covers — see the multi-image limitation below.

The macOS GPU backend is not usable for this bundle on litert-lm 0.15.0. Asked to describe an image with --backend gpu, it returns a run of ! characters instead of a caption. litert-lm benchmark still reports GPU numbers for it — 2,031 tok/s prefill and 183 tok/s decode — but they measure a path that produces no usable text, so no GPU row is quoted here. Use the CPU backend on the desktop. This is a desktop-runtime observation and says nothing about the iPhone or Android GPU paths described elsewhere on this card.

Multi-turn text works: you can ask follow-up questions about the same image.

The image is described accurately and in detail. The vision tower converts bit-faithfully to the reference (float CPU-parity corr ≈ 1.0); int8 vision weights keep grounding quality.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-25.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in main; the runtime accepts that split.

The gate prompt carries no image, so this covers engine creation and the text path. The vision path on the GPU is not measured here.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

⚠️ Known limitation — one image per conversation on the GPU backend

Single-image VQA — the primary use case — works great on GPU (~45 tok/s on iPhone 17 Pro). But on the GPU (Metal) backend, a second image in the same conversation truncates the answer — ask about one image per chat (start a new conversation for a different image).

This is GPU-delegate-specific, not a model/bundle issue: on the CPU backend, multi-image works perfectly (verified). The same GPU truncation reproduces with Apple's litert-community/FastVLM-0.5B, so it is general to the runtime's GPU fast_vlm path, not specific to this model. (Ruled out as causes: max_num_images — CPU works with it set to 1; and the vision encoder's 5D reshape — a 4D-clean rebuild still truncates on GPU.) For reliable multi-image, run on the CPU backend.

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load InternVL3-2B.litertlm with the image (vision) tower enabled (modalities [.vision]), attach a photo, and ask a question.

Note for app integrators: this is a vision-only bundle (no audio tower). Bring up the engine with the vision modality only (Modality.textImage / [.vision]) — requesting the audio tower (.all) on a bundle with no audio section fails at session creation.

Run on Android — Google AI Edge Gallery

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

Run this model with image input in the official Google AI Edge Gallery app — no custom app needed (the bundle carries the tokenizer, chat template, and image preprocessing config):

1. Push the bundle onto the phone (or download it there directly from this repo): adb push InternVL3-2B.litertlm /sdcard/Download/
2. Open the Gallery app, tap the + icon (bottom-right) and pick InternVL3-2B.litertlm in the file picker.
3. In the Import Model dialog, check "Support image" (required for image input), pick GPU (fast) or CPU, then tap Import.
4. Open the Ask Image task, choose the imported model, attach a photo, and ask.

Tip: on the GPU backend use one image per conversation (a known GPU-delegate trait of fast_vlm models); pick CPU if you want multiple images in one chat.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/InternVL3-2B InternVL3-2B.litertlm internvl3-2b
litert-lm run internvl3-2b     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,448,448,3]→[1,256,4096]) + VISION_ADAPTER ([1,256,4096]→[1,256,1536]) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
• The vision encoder bakes InternVL's ImageNet normalization and the NCHW transpose into the graph (the runtime feeds a [0,1] NHWC image).
• The InternViT attention is rewritten to be 4D-clean (qkv split before the head reshape, avoiding the 5D reshape(B,N,3,H,d) that GPU delegates reject) — numerically identical (corr ≈ 1.0), but it keeps the vision encoder almost entirely on the GPU delegate.
• Decoder exported with externalized embedder; InternVL's dynamic-NTK rope_scaling is stripped to base RoPE (valid since the export cache ≤ the base context window).

2026-08-28 — start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token — so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: 你是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in InternVL3-2B.litertlm now emits the block exactly once when no system message is given. In InternVL3-2B.litertlm, the block is not emitted when you pass a system message. The restored block adds 37 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer in InternVL3-2B.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (é, ñ, ü, °, ·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and any character without a whole-character vocabulary entry (emoji, most of Latin Extended-A) became the token the conversion had reused as UNK — the end-of-text token for this vocabulary. InternVL3-2B.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 — chat template fix: history turns with role 'assistant' (metadata-only, weights unchanged)

The chat template in InternVL3-2B.litertlm matched history turns only on role 'model', so a conversation created through the Conversation API with the standard role 'assistant' (for example create_conversation(messages=[...]) restoring a chat) silently dropped every assistant turn from the rendered prompt — the model saw consecutive user turns with its own replies gone. The template's role conditions now also accept 'assistant'. The engine's own send_message flow was never affected.

Template-only change: every section of the bundle except the metadata is byte-identical to the previous file (verified by per-section sha256), and single-turn renders, token ids, prefill counts and greedy answers are unchanged (verified on the LiteRT-LM runtime), so the numbers on this card still describe this file — only the file's own sha256 differs. Re-download only matters if you inject conversation history with role 'assistant'. Details: google-ai-edge/LiteRT-LM#3417 (https://github.com/google-ai-edge/LiteRT-LM/issues/3417).