Nanbeige4.2-3B-LiteRT

Nanbeige4.2-3B LiteRT-LM

Unofficial LiteRT-LM conversion of Nanbeige/Nanbeige4.2-3B, pinned to source revision 451ed48c3273ecef7ea8faaa43c31ce529763bb1.

The bundle preserves Nanbeige's exact two-loop execution over 22 shared physical decoder layers, including midpoint RMSNorm and 44 logical KV-cache slots. It was exported with the official LiteRT Torch and LiteRT-LM tooling.

Artifact

Bundle details:

• LiteRT-LM container version: 1.5.0
• quantization: dynamic weight INT8, activation FP32
• maximum cache length: 4,096 or 65,536 tokens, as named
• signatures: prefill_128 and decode
• tokenizer: packaged SentencePiece model and upstream Jinja chat template
• model section type: tf_lite_prefill_decode
• logical cache: 44 layers / 88 K and V tensors

The 4K file is the mobile-oriented default. The 64K file uses a dynamic cache dimension in all 88 K/V inputs for both signatures. Its 1,024-token export shape is only a trace/example size: prefill_128 keeps the 128-token compute chunk fixed while the cache and attention-mask widths resize dynamically. The package metadata caps the runtime at 65,536 tokens.

At the full cap, FP32 K/V state alone is approximately 23.6 GB (22 GiB), before model weights, output-cache tensors, and runtime workspace. Most mobile devices cannot allocate that. Select a context limit appropriate for the target device; the 64K artifact is intended for high-memory runtimes.

Toolchain and reproduction

The tested conversion used:

• litert-torch-nightly==0.10.0.dev20260722
• ai-edge-litert-nightly==2.2.0.dev20260721
• ai-edge-quantizer-nightly==0.9.0.dev20260722
• litert-lm-builder==0.14.0
• LiteRT-LM source revision e72a9a2ea3bb8e33b6faff6002744c55472e3600

modeling_nanbeige-litert.patch contains the compatibility adapter for the custom Transformers model. export_nanbeige_litert.py contains the exact 4,096/128 export recipe. export_nanbeige_litert_64k.py adds dynamic prefill and decode cache axes, the 65,536-token metadata cap, and the 44-slot cache factory.

python export_nanbeige_litert.py
python export_nanbeige_litert_64k.py
litert-lm-peek \
  --litertlm_file Nanbeige4.2-3B-dynamic-wi8-afp32-65536.litertlm

Validation

The package was inspected successfully with litert-lm-peek. The extracted quantized graph was then executed with LiteRT built-in CPU kernels on an AMD Ryzen 9 7950X3D:

• prefill_128 completed in 201.05 seconds.
• Four consecutive cache-fed decode calls completed in 26.58–31.65 seconds.
• Every decode produced finite logits with shape [1, 1, 166144].
• All 88 cache inputs and all 88 cache outputs were present.

The included validate_nanbeige_litert.py reproduces this test.

The 64K package was additionally validated beyond its 1,024-token trace window with validate_nanbeige_litert_64k.py:

• LiteRT-LM metadata reports max_num_tokens: 65536.
• Both prefill_128 and decode report dynamic cache/mask axes.
• All 88 K/V inputs are dynamic in each signature.
• A built-in-kernel decode with a 1,025-token cache completed in 515.92 seconds.
• Logits were finite with shape [1, 1, 166144] and range [-12.451786, 19.707262].
• Sampled first/last cache outputs were finite with shape [1, 8, 1025, 128].

The full 65,536-token allocation was not executed locally because the cache state plus duplicated input/output tensors and runtime workspace exceed the practical memory budget of the test machine. The beyond-trace execution, dynamic signatures, and embedded 65,536 metadata were all verified.

Current nightly limitations

• XNNPACK in the tested nightly produced non-finite results. On the 64K artifact, a 1,025-token decode executed in 645.76 seconds but returned NaN logits/cache values; the built-in LiteRT CPU kernels returned finite results. Select a known-good delegate/runtime on target hardware.
• The layer-19 key-cache output and its second-loop counterpart each reported 2,048 non-finite diagnostic entries after prefill. They did not propagate into logits across four consecutive causally masked decode handoffs. This adapter and artifact should be treated as experimental until Nanbeige is supported upstream and the cache-output anomaly is resolved.

This card reports those findings deliberately; it does not claim upstream LiteRT-LM certification.

License

The base model is Apache-2.0 licensed. Review the upstream model card for its full terms and intended-use information.