LFM2.5-1.2B-Thinking

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, lfm2.5-1.2b-thinking-int4): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 9.2 tok/s · prefill 55 tok/s · TTFT 4.78 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-1.2b-thinking-int4/CARD.md

Measured on device (edge-compat, lfm2.5-1.2b-thinking-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 35.5 tok/s · prefill 357 tok/s · TTFT 600 ms (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 7.1 tok/s · prefill 75 tok/s · TTFT 3.56 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-1.2b-thinking-int8/CARD.md

LFM2.5-1.2B-Thinking — LiteRT-LM

LiquidAI/LFM2.5-1.2B-Thinking converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.15 / a recent AI Edge Gallery). Sibling of litert-community/LFM2.5-1.2B-Instruct.

Update (2026-08-04): the .litertlm files were updated in place to add the ExecutorMetadata section that litert-lm ≥ 0.15 requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with missing some output TensorBuffers). Weights and graph are byte-identical to the original release, and the files ran on litert-lm 0.14 until the 2026-10-13 re-pack, whose lfm2 model type needs 0.15.

LFM2.5-1.2B-Thinking is the reasoning variant of Liquid AI's hybrid conv-attention flagship: it works problems inside <think>…</think> before answering. The bundle declares the thought channel in its metadata, so LiteRT-LM ≥ 0.14 streams the reasoning on a separate thought channel — your app can show or hide it natively, and past thinking is stripped from multi-turn context automatically.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Accuracy

GSM8K (greedy, 0-shot CoT, max-tokens 2048 — a thinking model needs the budget, n=100, same harness for all rows): PyTorch bf16 81% · LiteRT int8 77% (−4pt) · int4 72% (−9pt). Both files pass an 8-question sanity gate (7/8, zero degenerate); the reasoning stream arrives on the thought channel and the final answer follows cleanly after </think>.

Usage

litert-lm run ./LFM2.5-1.2B-Thinking_int8.litertlm --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?"

Give it a generous token budget (≥2048) — a reasoning model truncated mid-thought produces no final answer. The reasoning arrives on the thought channel; the final answer arrives on the main text channel after </think>.

GPU — the _int4_gpu file

Pixel 8a (Tensor G3), litert_lm_main built from the litert-lm v0.16.0 release tag, 263-token prompt, generation capped at 256, 3 runs per backend:

On phone-class hardware the GPU's win is prefill and time-to-first-token (3–6× both); decode is bandwidth-bound and roughly a wash, so long prompts gain far more than long answers. The CPU rows spread because the phone throttles across a run series; the GPU rows repeat to within 2%.

Apple M4 Max (litert-lm benchmark --cache no, litert-lm 0.16.0, -p 256 -d 256 --runs 3, otherwise idle machine; both backends were verified to generate correct text before these numbers were quoted):

On a desktop GPU the gain is across the board — roughly 11× prefill and 4× decode against the same file on CPU.

GPU — the _int8_gpu file

_int4_gpu above is the small, fast GPU file. _int8_gpu is the accurate one: the same weights as the int8 file — post-hoc int8 on the linears and embedding, convs left float — re-exported on the same litert-torch 0.9.3 lineage so a GPU delegate will take it. Before it existed there was no way to run this repo's int8 weights on a GPU at all; the int8 file reaches 536 of 579 operations and the runtime refuses the partial split.

Galaxy S26 (SM8850, Adreno), litert-lm v0.16.0, 205-token prompt, generation gated separately from the benchmark:

The two columns were measured a day apart in separate sessions, not side by side. Both are the first of two runs, which is the fair comparison available: this phone slows as it heats, and the two files generate different amounts of text in a benchmark pass, so their second runs are not exposed to the same heat. The spread is shown so you can see it. The decode figures in particular are indicative rather than controlled — the generation lengths differ.

They are a pair, not an upgrade. int4 decodes faster and downloads 41% smaller; int8 answers better. Prefill is close. Pick int4 when tokens per second or download size is the constraint, int8 when accuracy is.

Memory is the surprise. Measured the same way on the same phone — process high-water mark during a generation run — the 1.24 GB int8 file peaked at 530 MB and the 736 MB int4 file at 609 MB. Both figures move a little between runs, and the gap survives that: every reading taken of the int8 file — logged and unlogged — sits below every reading taken of the int4 file. The larger file is the lighter one in use, and neither is anywhere near the several-gigabyte figure a "the GPU holds weights in fp32, so budget about four times the file size" rule of thumb predicts. If you have been sizing GPU headroom that way, measure instead.

Apple M4 Max, litert-lm 0.16.0, -p 256 -d 256 --runs 3 --cache no, each reading taken after a 300 s idle period:

Decode moves with the KV budget here just as it does on CPU — 1024 is the faster setting and a good chat default; the file allows up to 4096. Compare the right rows: the _int4_gpu table earlier in this card was taken at the default 4096 budget, so the like-for-like pair is this file's 4096 row against it, not the 1024 row.

Reproducing these GPU figures. This Mac's GPU slows under back-to-back benchmarking much faster than you would guess. Four runs of one file in a row — the sibling LFM2.5-1.2B-JP_int8_gpu, not this one —, at the same effective settings, read 3712 → 3712 → 3529 → 2324 tok/s prefill: flat for two, then a 37% fall, with decode falling further over the same four runs (249 → 247 → 224 → 140). Recovery needs a real pause: 180 s of idle returned prefill to 3712, 300 s to 3946 and 900 s to 3949, so 180 s still leaves you about 6% low. Those recovery figures are from that same sibling file; this file's own rested readings are the two in the table above. The two rows above were each taken after 300 s, one reading at a time, which is what you need to reproduce them. The _int4_gpu Mac table earlier in this card predates this finding — same --runs 3 protocol, but with no record of what the machine had been doing beforehand — so treat a head-to-head between the two tables as rough rather than exact.

Quality is unchanged by the re-export. Measured head to head against the published int8 file through one harness on one day on the macOS CPU backend, GSM8K n=100 greedy: 77% for both. Asked the same prompt on the CPU backend with greedy decoding, the two files return byte-identical text — 10,081 bytes, the same reasoning, the same final answer — the re-export changes the graph, not the weights.

GPU — iPhone 17 Pro (iOS Metal)

The engine-creation failure previously tracked in LiteRT-LM#3129 turned out to be a context-sizing issue on the integration side, not a runtime bug. This section is about the _int4_gpu file; the _int8_gpu file has not been tested on iOS. The Metal delegate compiles its kernels against the KV cache that maxNumTokens pre-allocates; a value that does not match the file's exported plan makes engine creation fail with a shader-compile error. On iOS Metal, set maxNumTokens to 1024 (Swift EngineConfig(maxNumTokens:), C API litert_lm_engine_settings_set_max_num_tokens). For a thinking model, budget the context accordingly — the thinking phase alone can use several hundred tokens.

iPhone 17 Pro, litert-lm v0.16.0 xcframework, maxNumTokens 1024, single run, device charging: decode 69.7 tok/s, peak ~680 MB. All Metal kernels compile; the model thinks for a few seconds (TTFT 6.5 s including the thinking phase on our prompt) and then produces a coherent final answer.

Run on Android

Install a recent Google AI Edge Gallery, import this repo (or adb push a file and use local import: menu → Models → “+” → From local model file), select the CPU backend, set max tokens high (2048–4096), and chat. GPU acceleration needs one of the _gpu files — _int4_gpu or _int8_gpu — and litert-lm ≥ 0.16.0; the Gallery bundles its own runtime, which may lag that version — if its GPU toggle fails there, use CPU.

In a Flutter app, flutter_edge_ai_litertlm (1.14.0) installs this repo's bundle through FlutterEdgeAi.resolveHuggingFace('litert-community/LFM2.5-1.2B-Thinking'), which reads the repo's litertlm_manifest.json and picks the file and backend for the device; in React Native, react-native-litert-lm (0.8.0) does the same through resolveFromManifest(). Neither route is device-verified by this card; each package's README states its tested platforms.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, CPU backend, -p 256 -d 256 --runs 3 (the tool averages three iterations), warm-up run discarded, otherwise idle machine. Decode on this family depends strongly on the KV budget, so both settings are listed:

Set --max-num-tokens to the smallest value your use case needs — 1024 is a good chat default, and the file allows up to 4096. At 1024 the int4 file decodes fastest; at 4096 the two variants converge.

The _int8_gpu file carries the same weights as the int8 file, and on the CPU backend it measures like it. Run one after the other, -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024: _int8_gpu 413.7 tok/s prefill / 86.8 tok/s decode against int8 412.9 / 86.8. Decode is within 1%. Prefill matches to within 0.2%. No separate CPU table is given for the GPU file — use the numbers in this paragraph, which were taken in one sitting; its GPU figures are in the _int8_gpu section.

A caveat about the table above, and it is about the table rather than about any file: those rows were measured by an earlier session, and the same int8 file re-measured with the command just quoted reads 412.9 tok/s prefill against the 1536 shown, and 86.8 tok/s decode against 98.8. What differs between the two measurements has not been run down, so treat the prefill column above as not currently reproducible and the decode column as roughly right.

Use the CPU backend for the int8 and int4 files — neither can create a GPU engine. Both come from the pre-0.9.2 ShortConv export generation, whose prefill graph still carries INT64 ADD/CAST inside Lfm2ShortConv, plus GATHER_ND and a GREATER_EQUAL with const inputs. The GPU delegate takes 536 of the 579 operations and leaves 43 on the CPU, and the runtime then refuses the partial split: Hint fully delegated to single delegate is set, but the graph is not fully delegated. Re-exporting from the post-0.9.2 lineage removes those INT64 ops — that is what the _gpu files are. _int4_gpu runs fully delegated on Android OpenCL, macOS and iOS Metal (with maxNumTokens 1024; see the iPhone section); _int8_gpu carries these same int8 weights and runs fully delegated on Android OpenCL and macOS.

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery) int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is about 1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices. Those Android figures are single ship-gate runs, not medians. First device load compiles the graph and can take about a minute; later loads are instant.

Known limitation — multi-turn conversations on litert-lm >= 0.15

On the current runtime a conversation's context retains previous turns' <think> reasoning (history is appended, not re-rendered), while the model was trained with past reasoning stripped from its context. Single-turn quality is unaffected, but across several turns in one conversation the model can drift — a later answer may repeat an earlier turn's reply instead of addressing the new question (observed at turn 3 of a 3-turn probe, litert-lm 0.15.0 CPU, both variants). Practical guidance: start a fresh conversation per task (each on a new Engine; see "One Engine per conversation" below), and size --max-num-tokens so thinking turns complete — a reply truncated mid-<think> derails the turns after it.

One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's short-convolution state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). Until 2026-10-12, this card advised keeping the Engine loaded across conversations, which is the condition this note describes. In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (CPU backend, also for the _gpu files; litert-lm 0.18.0 on a Mac), none of the four files could be measured, in either of their two runs (a probe is run a second time only when its first run does not pass). The probe caps each answer at 200 tokens, and this model answers after its reasoning (uncapped, the _int4 file's answer to 17 times 23 took 521 tokens). Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

Tool calls on litert-lm 0.18.0. Since the 2026-10-13 re-pack (metadata only: llm_model_type lfm2, and the chat template now renders tool calls and tool results), the runtime parses the LFM2 tool call and the tool result reaches the model; the earlier files returned the call as message text (<|tool_call_start|>[get_weather(location="Paris")]<|tool_call_end|>). The lfm2 model type needs litert-lm ≥ 0.15.0; the tool-call parser is 0.18.0's — on 0.15–0.17 the call still arrives as text (read from the runtime source, not run here). For _int4_gpu and _int8_gpu, the GPU files: the 2026-10-13 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous files and have not been re-run — on 0.18.0 the Mac GPU digit check passed for _int8_gpu and ended in cap_hit for _int4_gpu, as it did with the previous file.

Thinking cannot be switched off. On litert-lm 0.18.0 ThinkingConfig(enable_thinking=False) has no effect on any of the four files: the model still reasons in the thought channel; the _int4 files fill a 400-token cap with it and give no answer text, the _int8 files finished inside the cap and answered (measured 2026-10-12). Set thinking_token_budget (300 answered 17 × 23 under a 700-token cap) and a cap above it.

Conversion notes

Converted with released litert-torch 0.9.1 with the same exporter fix as the Instruct sibling: the stock LFM2 short-conv block saves its conv state from padded prefill columns, corrupting the first generated token of nearly every reply; the fix derives the chunk's valid length from the attention mask in-graph and gathers the state from the last valid columns (verified token-identical to an exact per-token reference loop). Multi-length prefill signatures (1–1024). Quantization on this finetune: post-hoc int8 on linears + embedding with the convs left float, and post-hoc int4 on linears only. Post-hoc conv quantization breaks generation. Quantizing convs at export time instead costs this tune nothing measurable — 76 against 77 at n=100, inside the noise — so either recipe would have done here; the int8 and int4 files both leave the convs float, matching the JP sibling, where export-time conv-int8 does cost 9 points.

Why there is a separate _int4_gpu file. The exporter patch above reads the chunk's valid length with index_select and a mask sum, which lower to GATHER_ND and INT64 ops, and GPU delegates reject both — the CPU-only files take 536 of 579 operations on the GPU and the runtime then refuses the partial split. litert-torch 0.9.2 fixed the underlying conv-state bug upstream using valid-token masking and a one-hot matmul state select, which emits neither op. _int4_gpu is that export: litert-torch 0.9.3 + litert-converter 0.3.1, the same int4 recipe and the same weights as the int4 file, plus the executor-metadata section litert-lm ≥ 0.15 needs. It delegates fully on Android OpenCL — 501/501 and 519/519 nodes, zero rejected ops — and runs on the macOS GPU backend. Quality is unchanged: run head to head against the published int4 file on identical prompts, decoding and scoring, both score 38/50 on GSM8K (CPU, n=50), and the GPU file passes an 8-question sanity gate on both backends. The CPU-only files are left exactly as they are rather than replaced, so nothing changes for anyone already using them.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under the LFM Open License v1.0 (see LICENSE, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.