Qwen3-0.6B-int4

Qwen3-0.6B — INT4 · LiteRT-LM

On-device INT4 builds of Qwen/Qwen3-0.6B in the .litertlm format for Google's LiteRT-LM runtime (Android / iOS / desktop / embedded).

At ~332 MB each (≈7× smaller than the fp32 source), these load fast and keep a small memory footprint on phones.

Two flavours — pick your thinking behaviour

Qwen3 is a hybrid reasoning model. Because LiteRT-LM bakes the prompt format into the file, the thinking mode is fixed per file, so this repo ships both:

• The standard file embeds Qwen3's Jinja chat template (faithful ChatML; if your runtime can pass enable_thinking, it stays controllable).
• The no-think file prefills a closed empty <think></think> into the assistant turn, so the model answers directly. This was built for a wearable health-assistant where a phone streams short replies to a watch.

Build details

Converted with litert-torch (the generative API, formerly ai-edge-torch) from the original PyTorch checkpoint.

Multiple prefill signatures matter for latency: with a single large signature every prompt is padded to it. The exponential set above lets a short prompt run a tiny prefill instead of always paying the 1024-token cost.

Prompt format

Standard Qwen3 ChatML. The no-think file pre-closes thinking on the assistant side:

<|im_start|>user
{user message}<|im_end|>
<|im_start|>assistant
<think>

</think>

{answer}<|im_end|>

Usage (LiteRT-LM, Android / Kotlin)

val engine = Engine(
    EngineConfig(
        modelPath = "/path/to/qwen3_0.6b_nothink_q4_block32_ekv1280.litertlm",
        backend = Backend.CPU(),
        cacheDir = context.cacheDir.absolutePath,
    )
).apply { initialize() }

val conversation = engine.createConversation(
    ConversationConfig(
        systemInstruction = Contents.of("You are a helpful, concise assistant."),
        samplerConfig = SamplerConfig(temperature = 0.8, topK = 40, topP = 0.9),
    )
)

conversation.sendMessageAsync("How did I sleep last night?")
    .collect { msg -> /* stream text */ }

Stop tokens and the prompt format are embedded in each file — no extra runtime configuration required. With the standard (thinking-on) file you may want to strip the leading <think>…</think> block before display.

Performance (on device, measured)

Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL) against CPU (XNNPACK) on the same device, one fixed 205-token prompt text (202 tokens under the standard file's tokenizer, 206 under the nothink file's), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, every file/backend pair was generation-checked with a real prompt (for the nothink file a simple arithmetic prompt — its GPU and CPU outputs were identical, so the GPU runs the model faithfully); the GPU rows are full delegation (both files: decode 1437/1437 ops and every prefill signature fully on LITERT_CL).

What the table says:

• The GPU's win is the prompt side: prefill 4.5–6.3× faster than the same phone's CPU, and time-to-first-token 0.17 s against 0.7–0.9 s. Decode is a wash (44–50 against 47 tok/s on the nothink file) — the usual shape for sub-1B models, where the CPU keeps up with generation but not with prompt processing.
• Peak RSS is ~25% lower on the GPU (730–770 against 980–1040 MB). GPU engine init is the one cost the table hides: 2.7–5.6 s per process against 0.9–1.3 s on CPU.

Reproduce

import json
from litert_torch.generative.examples.qwen import qwen3
from litert_torch.generative.utilities import converter, export_config as ec_lib
from litert_torch.generative.layers import kv_cache as kv_utils

CKPT = "Qwen3-0.6B"  # local HF snapshot
ec = ec_lib.ExportConfig()
ec.kvcache_layout = kv_utils.KV_LAYOUT_TRANSPOSED
ec.mask_as_input = True

common = dict(
    output_path=".",
    prefill_seq_len=[8, 64, 128, 256, 512, 1024],
    kv_cache_max_len=1280,
    quantize="dynamic_int4_block32",
    export_config=ec,
    output_format="litertlm",
    hf_tokenizer_model_path=f"{CKPT}/tokenizer.json",
    stop_token_ids=[151645, 151643],
)

# Standard (thinking on): bake the Qwen3 Jinja template
tpl = json.load(open(f"{CKPT}/tokenizer_config.json"))["chat_template"]
converter.convert_to_litert(
    qwen3.build_0_6b_model(CKPT), output_name_prefix="qwen3_0.6b",
    jinja_prompt_template=tpl, **common)

# No-think: prefill a closed empty think block in the assistant turn
converter.convert_to_litert(
    qwen3.build_0_6b_model(CKPT), output_name_prefix="qwen3_0.6b_nothink",
    user_prompt_prefix="<|im_start|>user\n", user_prompt_suffix="<|im_end|>\n",
    model_prompt_prefix="<|im_start|>assistant\n<think>\n\n</think>\n\n",
    model_prompt_suffix="<|im_end|>\n", **common)

Training data, provenance & privacy

This is a format/precision conversion only — no fine-tuning or additional training was performed. Weights and behaviour are inherited directly from Qwen/Qwen3-0.6B; see the Qwen3 technical report for that model's pretraining data.

No new training data was introduced by this contribution, so there is no additional dataset to summarise. Because no data was collected or used here, there is no PII involved in this submission — the conversion operates purely on the public base-model weights and tokenizer.

Intended use & limitations

• Use: offline assistants on phones / edge devices — chat, summaries, simple Q&A over provided context.
• No-think file: not for complex multi-step reasoning or math (thinking is off). Use the standard file or a larger Qwen3 for that.
• Quantization: INT4 introduces some quality loss vs the fp32/INT8 source.
• Inherits the capabilities, biases, and limitations of Qwen/Qwen3-0.6B.

License

Apache 2.0, inherited from Qwen/Qwen3-0.6B.

Acknowledgements

• Qwen team for Qwen3-0.6B
• Google AI Edge for LiteRT-LM and litert-torch