TokForge-Qwen2.5-3B-CoreML-ANE-INT8

Published by TokForge for on-device inference.

TokForge

• Website: https://tokforge.ai
• Discord: https://discord.gg/Acv3CBtfVm
• Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
• iOS TestFlight: https://testflight.apple.com/join/jnufjzRr

Runs on-device in the TokForge app.

TokForge-Qwen2.5-3B-CoreML-ANE-INT8

Qwen/Qwen2.5-3B-Instruct → CoreML stateful INT8, targeted at the Apple Neural Engine (ANE) on iOS 18+.

This is a stateful CoreML conversion of Qwen2.5-3B-Instruct with an in-model KV-cache (MLState), per-block INT8 weight quantization, and a fixed context window of 2048 tokens. Built for the TokForge iOS app's CoreML/ANE inference track (a flagship "bigger model in the same RAM" path alongside llama.cpp/MNN).

What it is

• Architecture: Qwen2.5-3B-Instruct: 36 layers, 16 attention heads, 2 KV heads (GQA), head_dim 128, hidden 2048, vocab 151936.
• Format: CoreML .mlmodelc (precompiled), stateful KV-cache via MLState.
• Quantization: weights per_block(32) linear_symmetric INT8 (decompressed to fp16 on the ANE at compute time, saves RAM/bandwidth, not compute).
• Context window: fixed 2048 tokens (static-shape KV buffers).
• Sizes: fp16 mlpackage 5889 MB → INT8 mlpackage 3129 MB.

Requirements

• iOS 18+ / macOS 15+ (MLState stateful models + per-block INT8 are iOS18-only).
• Apple Neural Engine (A17 Pro / A18 / A19 / M-series). Load with MLComputeUnits.cpuAndNeuralEngine. Verify ANE dispatch at runtime; CoreML can silently fall back to GPU/CPU if ANEF specialization fails.
• The HuggingFace tokenizer (tokenizer.json + tokenizer_config.json included here), e.g. via swift-transformers.

I/O signature (the bridge contract)

Stateful model. 36 layers → 72 KV-cache states (key_cache_<i> / value_cache_<i>, i = 0..35).

Inputs:

• input_ids: {"dtype": "INT32", "shape": [1, 16], "flex": "shapeRange", "flex_detail": [[1, 1], [1, 2048]]}
• causal_mask: {"dtype": "FLOAT16", "shape": [1, 1, 16, 2048], "flex": "shapeRange", "flex_detail": [[1, 1], [1, 1], [1, 2048], [2048, 2048]]}
• position_ids: {"dtype": "INT32", "shape": [1, 16], "flex": "shapeRange", "flex_detail": [[1, 1], [1, 2048]]}
• cache_offset: {"dtype": "INT32", "shape": [1], "flex": null, "flex_detail": null}

Outputs:

• logits: {"dtype": "FLOAT16", "shape": [], "flex": null, "flex_detail": null}

States (MLState): 72 entries, each shape [1, 2, 2048, 128] fp16:

• key_cache_0 … key_cache_35
• value_cache_0 … value_cache_35

Driving it from Swift

• input_ids [1, S] int32: token ids for this step (S = prefill length, or 1 for decode).
• causal_mask [1, 1, S, 2048] fp16: additive mask (0 = attend, -1e4 = masked); column j valid iff slot j has been written and j ≤ that row's absolute position.
• position_ids [1, S] int32: absolute RoPE positions of the S tokens.
• cache_offset [1] int32: the begin slot in the KV window; the S new K/V rows are written in place to [cache_offset : cache_offset + S] (contiguous static-length slice).
• The model writes K/V into the key_cache_*/value_cache_* MLState buffers in place, attends over the full 2048-slot window (validity enforced by causal_mask), and returns logits. Runtime output shape is [1, S, 151936] (the spec lists logits with an unbound shape because of the flexible seq dim; at runtime it is [1, S, 151936]). Sample from logits[:, -1, :].

The seq dim is a flexible RangeDim(1..2048). Inputs input_ids / position_ids are [1, RangeDim], causal_mask is [1, 1, RangeDim, 2048], cache_offset is fixed [1].

Verified on the M4 (host)

• Loads on CPU_AND_NE after a slow first load (ANEF device-specialization: ship the precompiled .mlmodelc + rely on the on-device ANEF cache so only first load is slow).
• Prefill S=16 runs and returns finite logits [1, 16, 151936].
• Known caveat (S=1 decode on the macOS host), updated after a follow-up investigation: Single-token (S=1) stateful decode does not execute on the M4 build host (macOS 26.3, coremltools 9.0): every compute unit (CPU_ONLY, CPU_AND_GPU, CPU_AND_NE, ALL) fails with E5RT: Error(s) occurred executing a BNNS Op ("file a radar on BasicNeuralNetworkSubroutines"). This was root-caused to a host-side E5RT/BNNS execution limitation for small-sequence stateful slice-update graphs, NOT a defect in this artifact:
  • It is independent of the shape spec. A re-export with EnumeratedShapes including S=1 ({1,16,64,128}) was built and tested; it does NOT fix S=1, and is strictly worse on this host: enumerated stateful models fail the execution-plan build entirely (error code: -14, make_state() → "not loaded with the Core ML Framework"; cf. coremltools issues #2548 / #2271 and Apple's AsymmetricalEnumeratedShapesException). So the published artifact intentionally keeps the RangeDim(1..2048) spec.
  • It is independent of quantization and of this model: a minimal fixed-shape S=1 stateful toy model reproduces the same E5RT BNNS error.
  • Prefill works: S=16 returns finite logits [1,16,151936] on CPU_AND_NE. Why this is very likely host-only: Apple's own shipping CoreML LLMs (WWDC24 Mistral-7B, the on-device Llama-3.1 post) use exactly this RangeDim(1..max) + stateful KV-cache recipe and do decode at S=1 on real devices. The on-device A-series ANE runtime differs from this Mac's E5RT/BNNS host path. Therefore S=1 decode must be verified on a real iOS-18 device (a planned on-device measurement), not on the Mac. If a device also fails S=1, the fallback is a fixed-width decode window (pad to the working S and mask), but that is not expected to be necessary.

Conversion recipe (reproducible)

• coremltools 9.0, torch 2.8, transformers 4.44.2 on Apple Silicon (M4).
• HF model loaded fp16 with attn_implementation="eager" (FlashAttention swapped for SDPA: FA can't convert).
• Stateful wrapper: per-layer register_buffer KV tensors [1, 2, 2048, 128], new K/V scattered with a static-window index_copy (Apple SliceUpdateKeyValueCache style). This avoids the coremltools-9.0 rank0_expand_dims_swap / "Cannot delete op 'end_1'" graph-pass bug that a naive dynamic end-index write-slice triggers (which blocks quantizing the stateful model).
• ct.convert(..., states=[StateType...], minimum_deployment_target=ct.target.iOS18, compute_units=ct.ComputeUnit.CPU_AND_NE, compute_precision=FLOAT16).
• linear_quantize_weights(OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_block", block_size=32)) (per-block requires iOS18). NOT global kmeans palettization; that throws inf on Qwen's weight outliers (Qwen activations run wider than usual).
• Compiled to .mlmodelc with xcrun coremlcompiler compile.

Conversion script: convert_qwen_coreml.py (in this repo / on the build host).

Attribution & license

• Base model Qwen2.5-3B-Instruct © Alibaba Cloud / Qwen team, under the Qwen Research License (non-commercial) (https://huggingface.co/Qwen/Qwen2.5-3B-Instruct/blob/main/LICENSE). This conversion inherits that license.
• Conversion via Apple coremltools (BSD-3). Stateful KV-cache mirrors Apple's SliceUpdateKeyValueCache reference and the ANEMLL / john-rocky CoreML-LLM patterns.
• Packaged by TokForge for on-device iOS inference.

Credits

• Qwen2.5-3B-Instruct by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Qwen Research License (non-commercial). This license allows non-commercial use only (research or evaluation); commercial use needs a separate license from Alibaba Cloud.
• Converted to Core ML with coremltools by Apple (BSD-3-Clause).