granite-4.0-350m-litertlm

granite-4.0-350m-litertlm

ibm-granite/granite-4.0-350m converted to the LiteRT-LM .litertlm format for on-device inference with LiteRT-LM (Android, iOS, desktop, embedded).

• Base model: ibm-granite/granite-4.0-350m
• License: apache-2.0 (inherited from the base model)
• Format: .litertlm (granite-4.0-350m_q8_ekv1280.litertlm)
• Quantization: 8-bit dynamic — weights int8, activations fp32 (dynamic_wi8_afp32)
• KV-cache / max context: 1280 tokens

Usage

# CLI
litert-lm run granite-4.0-350m_q8_ekv1280.litertlm --prompt "What is the capital of France?"

The bundle embeds the base model's tokenizer and chat template, so prompts are templated for you. It also serves over an OpenAI-compatible endpoint:

litert-lm serve granite-4.0-350m_q8_ekv1280.litertlm --host 127.0.0.1 --port 9379
# then POST /v1/chat/completions with "model": "granite-4.0-350m_q8_ekv1280"

Conversion recipe

Converted with Google's litert-torch generic HF export (litert-torch==0.9.1, generative.export_hf): the checkpoint's own transformers modeling code is torch.exported (prefill + decode, static KV cache), quantized with the AI Edge Quantizer, and bundled with the HF tokenizer + chat template — no per-family builder.

A lossless dense reinterpretation is applied first (all-attention GraniteMoeHybridForCausalLM → dense GraniteForCausalLM), because the hybrid wrapper is not torch.export-compatible against a static cache (see litert-torch#933).

Validation / Parity

This conversion ships recorded parity evidence against the Hugging Face reference (generated by the workbench parity gate; see parity_report.json).

Weight-remap parity (argmax-identical logits, torch↔torch)

Granite Nano ships under the GraniteMoeHybridForCausalLM class but is all-attention; it is losslessly reinterpreted as a dense GraniteForCausalLM before export. Logits of the original vs. the reinterpreted model over the pinned prompts:

• argmax match: 100.0% over 5 prompts — greedy decoding is unaffected by the remap
• max abs logit diff: 7.41e-05 (within 0.001; residual is fp round-off from splitting the fused SwiGLU)

Runtime parity (HF reference vs .litertlm, greedy)

Greedy decoding (temp 0 / top-k 1) through each model's own chat template, over 5 pinned prompts:

• exact text match: 20.0%
• reference contained in device output: 20.0%

Honest caveat: litert-lm exposes no logits and the export is quantized to int8, so runtime parity is text/token parity, not logit parity. It is token-exact on short prompts; longer generations may diverge due to int8 + greedy argmax sensitivity — expected, not a conversion bug.

Limitations

• Quantized (q8): on-device outputs may differ slightly from the fp32 reference, especially on longer greedy generations.
• CPU backend validated; other backends untested here.

Training data

Format conversion only — no additional training data was used. Training-data composition and governance are inherited from the base model, ibm-granite/granite-4.0-350m (see its model card).

All personally identifiable information has been scrubbed from the training data.

Provenance

• Base: /workspace/models/ibm-granite_granite-4.0-350m
• Toolchain: litert-torch==0.9.1
• sha256: c8e9a29493f62b7c44461fb36980987c4c1454c75e95f57ba0539a8edc9dce76
• Converted and parity-validated with the edge-workbench pipeline.

Acknowledgments

Base model by its original authors (see the base model card). Conversion tooling: Google AI Edge (LiteRT-LM, litert-torch).