Falcon-H1-1.5B-Deep-Instruct

Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 6.0 tok/s · prefill 121 tok/s · TTFT 1.92 s · all 10451 ops delegated (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 12.0 tok/s · prefill 103 tok/s · TTFT 2.14 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 2.0 tok/s · prefill 25 tok/s · TTFT 10.90 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/falcon-h1-1.5b-deep-instruct-int8/CARD.md

Falcon-H1-1.5B-Deep-Instruct — LiteRT-LM

tiiuae/Falcon-H1-1.5B-Deep-Instruct converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. Sibling of litert-community/Falcon-H1-0.5B-Instruct, litert-community/Falcon-H1-1.5B-Instruct and litert-community/Falcon-H1-3B-Instruct — same conversion, same patch.

Falcon-H1 is TII's fully-hybrid design: every layer runs a grouped-query attention branch and a Mamba2 selective-scan branch in parallel on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state. The Deep variant trades width for depth — 66 layers at hidden size 1280 — so the bundle carries 264 state buffers (132 conv/SSM + 132 KV), the most of any Falcon-H1 size.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Correctness

• Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 48 decode positions — top-1 and top-5 identical at every position, mean per-position logit correlation 1.0000, mean KL ≈ 0.
• 8-question sanity gate: 8/8 on every lane — GPU and CPU, litert-lm 0.15.0 and 0.16.0, no degeneration.
• Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length) — CPU fills 12–51 and GPU fills 12–31 all clean.
• iPhone 17 Pro (Metal): the 8-item composite probe answers 8/8 on GPU and 8/8 on CPU. The two backends produce the same answers word for word, apart from one function word ("8 times 7 equals 56" on GPU, "is 56" on CPU).
• One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file answered all three questions in full sentences on the same Engine although that conversation's system prompt asked for one word, against one-word answers on the fresh Engine, in both of its runs (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue). For example, it answered the capital question with "The capital of France is Paris." on the same Engine. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

Usage

litert-lm run ./Falcon-H1-1.5B-Deep-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Falcon-H1-1.5B-Deep-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

On device (cold start, single runs, 146-token composite prompt, quality harness):

Honest notes:

• GPU runs with fp32 activations (declared in the bundle) — expect a corresponding memory multiple over CPU.
• Depth costs decode: 66 layers means 66 sequential attention+scan blocks per token. Decode lands below the shallower 3B sibling (65.3 tok/s on the same machine and settings) despite this model being roughly half its size, while prefill — which batches over the sequence — comes out ahead (1134 vs 979 tok/s).
• On GPU the runtime logs TopK requires src tensor C dimension to be divisible by 4 (vocab 65537). It is family-normal on every Falcon-H1 size — a fallback path, not a failure; generation and the quality gates are unaffected.
• The full prefill ladder loads on a 12 GB phone, all 264 state buffers included: GPU peak is 4.89 GB, 2.8× the file. Load takes about a minute on GPU (8 s on CPU) — the Metal kernels for every signature are built at engine creation.

Galaxy S26 — GPU and CPU

The published bundle runs on the Android GPU backend: LiteRT takes the whole graph and the model generates.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

Speed on the same handset with one binary and one 205-token prompt file; only --backend changes (litert_lm_advanced_main from litert-lm 0.16.0, --benchmark, --sampler_backend=cpu). The GPU row is the gate run of 2026-08-24 (one run); the CPU rows were taken 2026-09-05, two cold runs (compile and weight caches deleted between runs) with the phone cooled below 42 °C before each run.

The CPU decodes 1.5× faster and is ready in 6 s where the GPU spends 89 s building its kernels; peak memory is level. The GPU wins prefill. This repo's litertlm_manifest.json recommends cpu for flagship-class Android on that pair. Pick gpu for long prompts with short answers.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm falcon_h1_work/):

• Composite hybrid cache layer: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
• Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
• Falcon-specific wiring: the µP multiplier vector (mup_vector, a non-persistent model-level buffer) and ssm_in_multiplier are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
• Prefill-pad guard: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
• Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License and changes

Distributed under the Falcon LLM License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.