qwen3-1.7b-coreml-int8

Qwen3-1.7B Base → Core ML FP16 / W8 / W4

<!-- polished-overview:start -->

Overview

qwen3-1.7b-coreml-int8 is a Core ML conversion intended for deployment on Apple platforms, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

What is included

• *.bin (3 files)
• Additional configuration, tokenizer, processor, or shard files (25 visible artifacts total)

Quick start

Core ML

Use the compiled Core ML assets from an Apple-platform application or a current Core ML tooling stack. Check the repository's input/output metadata before wiring the model into a production pipeline.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

<!-- polished-overview:end -->

Reproducible conversion of Qwen/Qwen3-1.7B-Base into stateful Core ML packages for iOS 18+, iPadOS 18+, and macOS 15+.

W8 is the recommended deployment package. Block-32 W4 saves substantially more storage, but the corrected bilingual evaluation found much higher repetition and no continuations exactly matching FP16.

Published artifacts

The converter produces:

coreml_out/
├── Qwen3-1.7B-Base-TopK64-FP16.mlpackage
├── Qwen3-1.7B-Base-TopK64-W8.mlpackage
├── Qwen3-1.7B-Base-TopK64-W4.mlpackage
├── Tokenizer/
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   └── ...
└── conversion_manifest.json

The quantized variants are weight-only:

• W8 stores eligible weights as symmetric per-channel INT8.
• W4 stores eligible weights as symmetric INT4 in blocks of 32 values along the input axis.
• Core ML decompresses those weights for floating-point execution. These are not W8A8 or W4A4 models.
• Linear/gather outputs and the KV cache use FP16 storage.
• Remaining activation arithmetic stays FP32 to prevent the non-finite decode failure observed during development.
• Top-k runs inside the graph, avoiding transfer of all 151,936 logits to Swift for every generated token.

Model configuration

KV-cache memory

Key and value states use FP16:

For an iPhone 15 Pro, start with 1,024 tokens when memory and thermals matter most, or 2,048 when a longer context is required. Always validate the final configuration on a physical target device.

Conversion on macOS

Stateful conversion and inference require macOS. Create an isolated Python 3.12 environment:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -r requirements.txt

Generate all three variants:

python convert_qwen3_1_7b_coreml_int8.py \
  --model-id Qwen/Qwen3-1.7B-Base \
  --context-size 1024 \
  --top-k 64 \
  --source-dtype float32 \
  --output-dir coreml_out

The safe mixed-precision trace requires a float32 source. A float16 source is rejected because it can lower attention matmuls to FP16 before the mixed policy is applied and can produce NaN logits during stateful decode.

To remove the FP16 intermediate only after W8 and W4 save successfully:

python convert_qwen3_1_7b_coreml_int8.py \
  --context-size 1024 \
  --top-k 64 \
  --source-dtype float32 \
  --remove-fp16 \
  --output-dir coreml_out \
  --overwrite

Use --top-k 0 to export full-logit packages instead.

High-vocabulary token-ID repair

Packages created before the July 21 repair could alias channels above 65,535 inside the FP16 Core ML LM head:

• Token 80182 ( partes) appeared as 14646 (endment).
• Token 70136 ( primero) appeared as 4600 ( Error).

The values were plausible, which made the failure look like punctuation, EOS, Spanish tokenization, or TTS corruption. The IDs were actually the correct IDs modulo 65,536.

The repaired converter applies the 151,936-row LM head in 32,768-row chunks, computes local top-k candidates, adds vocabulary offsets in FP32, and casts once to public INT32 indices. Published packages declare:

lm_head_strategy=chunked_32768_fp32_ids

The macOS regression in tests/test_topk_token_ids.py covers IDs 80182, 70136, 150000, 65535, and 65536.

PyTorch parity gate

Validate packages individually:

python validate_coreml.py \
  --mlpackage coreml_out/Qwen3-1.7B-Base-TopK64-FP16.mlpackage \
  --context-size 1024 \
  --prompt "Para resolver un problema difícil, yo"

Corrected results for that prompt:

FP16 matched PyTorch for all 20 checked continuation tokens. W8 preserved both top-1 choices. W4 stayed finite and returned valid IDs, but diverged on the first stateful decode step.

Corrected bilingual Qwen3-TTS → ASR benchmark

Each package generated five English and five Spanish continuations. Qwen3-TTS MLX synthesized only the generated continuation, and Parakeet ASR transcribed it at 127.0.0.1:5093/v1/audio/transcriptions.

Language routing was explicit and isolated:

NeuTTS, Chatterbox, Supertonic, and cloud fallbacks were disabled. All 30 rows completed without generation, TTS, or ASR infrastructure failure.

Friendly comparison

WER is an audio-carrier metric here: it measures whether TTS plus ASR preserved text already generated by the language model. It does not measure semantic quality, relevance, repetition, or whether the model stayed in the expected language.

W4's low carrier WER does not make it the best language model. ASR can transcribe repetitive text perfectly. W8 remains the recommended quantized package because it is closest to FP16 and has low repetition.

64-token raw repetition ablation

The 20-token speech benchmark can end before a repetition loop becomes obvious. A follow-up therefore generated 64 raw greedy tokens for the same ten English and Spanish prompts, with no TTS or ASR in the measurement path.

Protecting the embedding and all five repaired LM-head chunks at W8 reduced mean repetition by 42.7% relative to uniform W4, but did not make W4 competitive with W8. Keeping the same vocabulary-facing weights in FP16 cost essentially as much as the complete W8 package and still produced more failures.

The experiment also exposed two implementation requirements for mixed-bit conversion: the selector must recognize the chunked LM head, and Core ML op_name_configs must use consuming operation names rather than constant names. It also exposed a metric blind spot: word trigrams can score a long repeated numeric token as zero repetition. Any future gate must add token-level, character-level, and maximum-identical-run checks.

Raw FP16/W8/uniform-W4/W4-VocabW8 rows are published in validation/w4_vocab_w8_repetition_20260721.json. The stronger vocabulary-FP16 run is reported as an exploratory aggregate because its raw rows were not added to the repository.

Decision: W8 is the default package. Uniform W4 remains an ablation, and mixed W4 remains experimental. TTS/ASR correlation is intentionally not used as an acceptance gate for a candidate that already fails raw-text generation.

The formerly corrupted Spanish prompt now produces this with FP16 and W8:

lo divido en partes más pequeñas y sencillas. ¿Cómo se llama este método de

Both audio rows scored WER 0.0.

Why some benchmark samples end mid-sentence

The comparison intentionally used --max-new-tokens 20. Some language-model continuations therefore ended in fragments before TTS received them. This is a source-generation limit, not audio truncation.

For genuinely long input, the local Qwen3-TTS wrapper now packs complete sentences into at most 25 words per chunk (configurable from 20 to 30) and joins them with a 350 ms sentence pause. A 59-word Spanish follow-up produced four chunks of 13, 13, 17, and 16 words with the Spanish reference; Parakeet recovered 59/59 words with WER 0.0, including the final phrase.

Run the comparison with:

python asr_correlate_models.py \
  --int4-package coreml_out/Qwen3-1.7B-Base-TopK64-W4.mlpackage \
  --int8-package coreml_out/Qwen3-1.7B-Base-TopK64-W8.mlpackage \
  --fp16-package coreml_out/Qwen3-1.7B-Base-TopK64-FP16.mlpackage \
  --tokenizer coreml_out/Tokenizer \
  --tts-script "$HOME/.config/opencode/tts.sh" \
  --tts-engine qwen3-mlx \
  --endpoint http://127.0.0.1:5093/v1/audio/transcriptions \
  --asr-model parakeet-tdt-0.6b-v3 \
  --context-size 1024 \
  --max-new-tokens 20 \
  --warmup-tokens 24 \
  --output-dir asr_correlation

Verify the published rows and summary:

python verify_validation_artifacts.py \
  --results validation/current_20260721/qwen3_coreml_qwen3tts_asr_results.json \
  --summary validation/current_20260721/qwen3_coreml_qwen3tts_asr_summary.json

iOS integration

Add the W8 or W4 .mlpackage to Xcode. swift/QwenCoreMLRunner.swift:

• creates a new MLState for every independent generation;
• reuses that state during decode;
• builds prefill and decode masks;
• rejects non-finite output;
• supports top-k and full-logit packages;
• serializes predictions sharing the same state.

The runner receives token IDs. Integrate Tokenizer/ through Hugging Face swift-transformers or your own tokenization layer.

Runtime rules

1. Create a new state for each independent generation.
2. Do not run concurrent predictions against the same state.
3. Prefill uses mask shape [1, 1, promptLength, promptLength].
4. Decode uses mask shape [1, 1, 1, visibleLength].
5. Visible prefill positions are 0; the upper triangle is -65504.
6. Decode uses an all-zero mask.
7. Prompt plus generated tokens must not exceed the exported context.

Recommendation

• Keep FP16 as the scientific and quality reference.
• Deploy W8 by default.
• Use W4 only when storage is the dominant constraint and increased divergence and repetition are acceptable.

See PROCESS_AND_FINDINGS.md for the full chronology, QUANTIZATION_STORIES.md for the engineering narrative, and VALIDATION.md for detailed gates and caveats.