TTS-Core-AI

TTS Core AI Lab — converted for iOS / iPhone

This repository contains Core AI .aimodel assets converted for an arm64 iOS application running on physical iPhone hardware. It is the model-asset companion to the TTSCoreAILab iOS app.

Deployment target: iOS 27.0+ on arm64 iPhone hardware.

Benchmark device: iPhone 17 Pro. This identifies the device used for the current RTF benchmark; it is not a guarantee that every included profile has passed end-to-end validation on every iPhone.

What was converted

Kokoro subgraphs are exported through torch.export and converted with coreai_torch. Supertonic ONNX graphs are recovered at fixed shapes, converted through onnx2torch and torch.export, then converted with coreai_torch. The resulting assets use the versioned AICode layout required by the iOS 27 device compiler.

These are deployment assets, not general-purpose PyTorch, ONNX, Core ML, web, Android, macOS, or iOS Simulator models. Use them only with the matching TTSCoreAILab app and its Core AI runtime integration.

Device and profile notes

• The app targets arm64 iPhone hardware with deployment target iOS 27.0.
• The iPhone 17 Pro is the current benchmark device.
• Kokoro packages include 64x64, 128x256, and 256x512 routed profiles, plus legacy/reference profiles.
• Supertonic packages include 64x32, 128x64, 128x128, and 128x256 fixed-shape profiles.
• Asset presence does not imply physical-device validation. Validate the exact app revision, profile, prompt, and device before shipping or profiling.

App integration

The iOS project downloads the two directories below into its repository root:

• TTSCoreAILab/Models
• TTSCoreAILab/SupertonicResources

From the app repository, run:

./scripts/download-assets.sh

---

Project overview

Local iOS workbench for incrementally porting and profiling the split Kokoro pipeline in Swift.

The app integrates the official KittenTTS Swift package and Supertonic 3 ONNX pipeline, and provides four modes:

• KittenTTS: ONNX Runtime CPU, eight voices, configurable speed.
• Kokoro: the existing 11-stage split Core AI pipeline.
• Supertonic 3: four Core AI models, ten voices, configurable flow steps, 44.1 kHz.
• Compare: sequentially generates all three models for the same text, preserves PCM results for independent playback, and displays wall time, inference time, RTF, Core AI coverage, and speedup.

KittenTTS downloads its Nano model and phonemizer assets on first use. Kokoro and Supertonic 3 use Core AI model assets hosted separately on Hugging Face: https://huggingface.co/augustZheng/TTS-Core-AI.

Model Assets

The Xcode project expects these asset directories:

• TTSCoreAILab/Models
• TTSCoreAILab/SupertonicResources

They are not committed to GitHub because the full asset bundle is about 4.2 GB and contains files larger than GitHub's regular 100 MB file limit.

Download them from Hugging Face before building:

./scripts/download-assets.sh

The script uses uv to run the Hugging Face CLI without creating a project virtual environment.

The current stage runs a fixed Hi. text-to-acoustic-feature chain:

fixed token ids + attention mask
-> ALBERT
-> BERT projection
-> fixed-length duration encoder
-> duration head
-> Swift duration/alignment calculation
-> text encoder CNN + LSTM
-> Swift ASR alignment
-> F0/noise shared LSTM + residual blocks

Bundled assets:

• kokoro_bert_eager_64.aimodel
• kokoro_bert_projection.aimodel
• kokoro_duration_encoder_no_pack_64_intmask.aimodel
• kokoro_duration_head.aimodel
• kokoro_text_encoder_conv_64_intmask.aimodel
• kokoro_text_encoder_lstm.aimodel
• kokoro_f0n_shared_lstm_64.aimodel
• kokoro_f0n_blocks_64.aimodel

How to Try

1. Open TTSCoreAILab.xcodeproj in Xcode.
2. Select the same physical iPhone target that passed the original smoke test.
3. Run the TTSCoreAILab app target.
4. Tap Run Acoustic Stage.

Expected reference result from the Python CoreAI runtime:

• shapes: [1,64,768], [1,512,64], [1,64,640], [1,64,50]
• valid-token durations: [18, 2, 5, 9, 6, 1]
• raw predicted frames: 41
• fixed target frames: 64
• aligned ASR: [1, 512, 64]
• F0: [1, 128]
• noise: [1, 128]

Python CoreAI reference mean-absolute values:

• shared features: 0.35923
• F0: 72.4452
• noise: 6.59444
• text conv: 0.02399
• text hidden: 0.16430
• aligned ASR: 0.18732

The current build now also includes:

• kokoro_decoder_pre.aimodel
• kokoro_generator_core.aimodel
• kokoro_istft.aimodel
• fixed Hi. decoder style and harmonic-source fixtures
• Swift PCM-to-WAV playback through AVAudioPlayer

The current UI mirrors the browser Local Lab:

• editable text and af_heart voice selection;
• selectable bundled Core AI budgets (64 / 64, 128 / 256, and 256 / 512), with automatic profile escalation for longer segments;
• optional automatic segmentation;
• total time, Core AI inference time, audio duration, and RTF;
• per-stage Core AI timing bars;
• generated audio playback controls and a detailed run log.

The PyTorch reference and native Kokoro comparison controls are visible but disabled because those runtimes are not bundled in the iOS app.

The bundled profiles increase the text/frame budget from 64 / 64 to 128 / 256 and 256 / 512; automatic mode chooses the smallest profile that fits a segment. Supertonic ships its validated 64 / 32 baseline assets under SupertonicResources/coreai-assets-baseline.

Tap Generate Locally. A successful run should play generated audio and report:

• generator input: [1, 512, 128]

• spec/phase: [1, 22, 7681]

• audio: [1, 1, 38400]

• audio mean absolute value: approximately 0.01756

• audio min/max: approximately -0.2076 / 0.3555

• RTF Result

The current benchmark on iPhone 17 Pro.

RTF (Real-Time Factor) measures how long it takes to generate one second of audio.

• RTF < 1.0 → Faster than real time (suitable for interactive TTS)

• Lower is better

The benchmark below compares the same prompt on the same device across different runtimes.

<p align="center"> <img src="docs/images/rtf-result.png" width="900"> </p>