Downloads · 30 days
2
6% of all-time downloads
canary-qwen-2.5b-coreml-static
Muesli-HQ/canary-qwen-2.5b-coreml-static
canary-qwen-2.5b-coreml-static is a machine learning model from Muesli-HQ. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Static-shape prefill/decode CoreML models for the decoder component of nvidia/canary-qwen-2.5b, targeting Apple Silicon (ANE/GPU) inference in dictation apps.
All-time downloads
32
Public
Repo size
5.6 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin5.6 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Mar 27, 2026
- Updated
- Mar 27, 2026
- SHA
- 0aac6ba6
Base models
Model card
From the Hugging Face model README
Canary-Qwen 2.5B — Static CoreML Decoder (FP32)
Static-shape prefill/decode CoreML models for the decoder component of nvidia/canary-qwen-2.5b, targeting Apple Silicon (ANE/GPU) inference in dictation apps.
How this differs from the dynamic-prompt versions
coreml-int8 (dynamic, shipped) | coreml-static (this repo) | |
|---|---|---|
| Decoder graph | Flexible query length via RangeDim | Fixed-shape prefill (164 tokens) + single-token decode |
| Prompt handling | Dynamic prompt length, CoreML specializes at runtime | Baked-in for Muesli/Swift prompt shape (33 text + 126 audio + 5 text = 164) |
| Cache update | Slice assignment with dynamic end-step | One-hot mask scatter (CoreML-friendly, no dynamic indexing) |
| Quantization | INT8 (W8A8) | FP32 (not yet quantized) |
| Cold start | Slower — CoreML must specialize the flexible graph on first run | Faster — shapes are fixed, no specialization needed |
| Use case | General-purpose, any prompt shape | Dictation-optimized for a known prompt/audio regime |
Why static graphs?
The flexible RangeDim decoder caused CoreML cold-start specialization pain and
earlier lowering failures. Static prefill+decode graphs are much more
CoreML-friendly for a fixed dictation prompt/audio regime. The trade-off is that
if the Swift prompt structure changes, the models need to be re-traced.
Contents
canary_prefill_static.mlpackage/ — Fixed-shape prefill (164 tokens)
canary_decode_static.mlpackage/ — Fixed-shape single-token decode with KV cache
Architecture
- Base model: Qwen/Qwen3-1.7B (decoder only — encoder is separate)
- LoRA: nvidia/canary-qwen-2.5b LoRA merged at scale 256/128
- Layers: 28 transformer layers, hidden=2048, GQA with 8 KV heads, head_dim=128
- Max sequence length: 256
- Static prefill length: 164 (Muesli/Swift prompt shape)
- KV cache state dtype: float16
- LM head: separate artifact (not fused — fused path was slower in benchmarks)
- RoPE theta: 1,000,000
Prefill inputs
| Name | Shape | Type |
|---|---|---|
hidden_states | [1, 164, 2048] | float32 |
Decode inputs
| Name | Shape | Type |
|---|---|---|
hidden_states | [1, 1, 2048] | float32 |
cache_update_mask | [1, 256] | float32 (one-hot at current position) |
cache_valid_mask | [1, 256] | float32 (1.0 for valid cached positions) |
Provenance
- Traced on Modal (CPU-only, torch 2.6.0) from
nvidia/canary-qwen-2.5bsafetensors - Converted to CoreML on macOS with
coremltools 9.0 - Trace run:
20260327-020004-static-prefill-164-sliceassign - Cache update uses slice-assignment pattern (earlier
.copy_()pattern failed CoreML conversion)
Related repos
phequals/canary-qwen-2.5b-coreml-int8— shipped dynamic-prompt INT8 versionphequals/canary-qwen-2.5b-coreml-fp16— earlier dynamic-prompt FP16 version
- License
- apache-2.0
- Created
- Mar 27, 2026
- Updated
- Mar 27, 2026