magenta-realtime-2-iphone

Magenta RealTime 2 on the iPhone, Without the GPU

Ten minutes of live 48 kHz stereo music generation on an iPhone 15 Pro Max, at 1.03× real time, with zero underruns and no app GPU work.

This is Google DeepMind's Magenta RealTime 2 (mrt2_small, 230M parameters), cut into three Core ML graphs and placed on the engine each one suits. On the iPhone 15 Pro Max (A17 Pro), the temporal transformer and the codec decoder run on the Neural Engine and the depth transformer runs on the CPU: 11.2 + 9.8 + 2.3 ms per 40 ms frame (iteration-normalized p50 over a 610 s run).

What that claim does and does not cover:

• Buffered, not live. The run keeps up to about 21 s of audio queued, and it resets the temporal state every 10 s. Without the reset, the MLX reference itself emits samples at or above full scale within 600 s.
• Not on the iPhone 12 Pro (A14). It reaches 0.90× real time and runs out of buffered audio after 294 s.
• The FP16 decoder is approximate. On a held-out window it scores 38.8 dB against the FP32 decoder on the A17 Pro's Neural Engine, and it is wrong on the rare windows whose activations still exceed FP16 range after its rescale (1 of 600 by host scan).

Exporters, validation harness, method docs: github.com/mattmireles/magenta-realtime-2-iphone Papers: Surgical Inference (conversion and admission findings, Section 5) and Throughput Is Not Liveness (end-to-end runtime study), both linked from the GitHub repo.

What's in the download

All models are mlprogram with a minimum deployment target of iOS 18 / macOS 15. Every package here was regenerated from the public exporters, and its weight.bin matches the certified package byte for byte (checksums below).

Why three graphs

prompt text ──(compiled once, on a Mac)──▶ prompt vector
               │
               ▼
┌──────────────────────────────────┐
│  TEMPORAL  (stateless step)      │ ◀── ANE, fp16
│  Next 40 ms frame, host K/V ring │
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DEPTH  (in-graph rollout)       │ ◀── CPU, fp16
│  Samples 12 RVQ levels           │     host Gumbel noise
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  RVQ GATHER  (host)              │ ◀── CPU
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  DECODER  (SpectroStream)        │ ◀── ANE fp16 + rescale (A17 Pro)
│  Every 12 frames, 25-frame window│     CPU fp32 (A14)
└──────────────┬───────────────────┘
               ▼
┌──────────────────────────────────┐
│  iSTFT + OVERLAP-ADD  (host)     │ ◀── CPU
└──────────────┬───────────────────┘
               ▼
   48 kHz stereo audio

Three findings decided the cuts (details in Surgical Inference, Section 5):

1. In-graph state mutation, not attention, is the ANE compile cliff. Multi-frame ct.StateType graphs fail ANECCompile() with error −14. The same math as a stateless one-frame step, with caches as ordinary inputs and updates as ordinary outputs, compiles fully to the ANE.
2. Finite is not correct. The decoder's activations exceed FP16 range. Without the exact rescale, FP16 is wrong in either layout: the CPU and GPU return NaN, and the ANE returns finite output at about 0 dB SNR. With the rescale it is usable on the A17 Pro and still wrong on the A14's ANE, which is why the A14 uses the FP32 decoder.
3. Weight bytes set the floor. Each weight traversal costs at least weight bytes ÷ DRAM bandwidth. Rolling all 12 depth levels into one graph still reads the layer weights once per level; the gain came from FP16's halved bytes.

Sampling entropy (RNG, Gumbel noise) and the codebook gather stay on the host, so seeds reproduce and token-level parity is testable. CFG is baked in at conditioning time.

Tensor contracts

Everything is static-shaped. sampled_codes are codebook-local (0–1023); the unique id is 6 + level*1024 + code. temporal_feedback is the next frame's temporal_inputs row. The decoder runs once every 12 frames over a 25-frame window that keeps 12 frames of causal history; without that history its output is wrong. Full I/O schemas are in metadata/.

Usage (Swift sketch)

import CoreML

let temporal = try MLModel(contentsOf: temporalURL, configuration: aneConfig)  // .cpuAndNeuralEngine
let depth    = try MLModel(contentsOf: depthF16URL, configuration: cpuConfig)  // .cpuOnly
let decoder  = try MLModel(contentsOf: decoderURL,  configuration: decConfig)  // A17 Pro: f16 on .cpuAndNeuralEngine; A14: fp32
var ring = HostKVRing(layers: 12, window: 41)   // 48 fp16 buffers, host-owned

while generating {                              // must average >= 25 frames/s
  let t = try temporal.prediction(from: ring.inputs(
            temporalInputs: prevFeedback, source: conditioning))
  ring.append(updates: t)                       // host-owned ring write
  let d = try depth.prediction(from: ["temporal_frame": t.frame,
                                      "gumbel_noise": gumbel(seed),
                                      "inverse_temperature": invTemp])
  prevFeedback = d.temporalFeedback
  // gather 12 codebook rows on the host; every 12 frames run the decoder over
  // a 25-frame window (12 frames of history), then iSTFT + overlap-add.
}

Check placement with MLComputePlan at session start; a successful prediction on .cpuAndNeuralEngine does not prove the ANE ran it.

Verify the numbers

Every number on this card has a receipt in validation/ or in the GitHub repo's validation/results/. The device results (placement, sustain runs, decoder accuracy) are in the GitHub repo's docs/validation-receipts.md and the Surgical Inference paper. Quick local check:

pip install coremltools numpy torch
git clone https://github.com/mattmireles/magenta-realtime-2-iphone && cd magenta-realtime-2-iphone
hf download mattmireles/magenta-realtime-2-iphone --local-dir hf

# Depth rollout, FP32: zero token mismatches required
PYTHONPATH=exporters python validation/validate_depth_body_rollout.py \
    --model-path hf/MRT2DepthBodyRollout.f32.mlpackage --float32

Checksums (sha256 of Data/com.apple.CoreML/weights/weight.bin)

The temporal weights are shared with the 2-frame carry export; the package here is the one-frame streaming graph (--frames 1 --streaming), which is the one that runs on the ANE.

Superseded (superseded/)

Export metadata (and, for the temporal graphs, validation receipts) for earlier artifacts, kept as negative-result evidence. The packages themselves are not hosted; the superseded exporters reproduce them.

Conditioning

The temporal body cross-attends to a 256-dim compiled prompt (source_encoded). The exporter (export_conditioning.py) compiles a text prompt through MusicCoCa and the MRT2 conditioning encoder on a Mac, deterministically, with CFG baked in at reference strength (3.0, 1.0, 1.0).

We ship one conditioning vector, "smooth electronic", as a test vector, not a preset. Treat conditioning quality as experimental.

Requirements

• iOS 18+ / macOS 15+.
• FP16 decoder: A17 Pro-class Neural Engine with .cpuAndNeuralEngine. On the CPU or GPU it is usable on typical content but returns non-finite values on the hottest windows; on the A14's Neural Engine it is wrong. Use the FP32 decoder elsewhere.
• Measured on iPhone 15 Pro Max (A17 Pro) and iPhone 12 Pro (A14). Only the A17 Pro sustains real time.
• coremltools ≥ 8 to re-export.

License and attribution

The model weights here are content-preserving derivatives (precision and memory-layout transforms only; no fine-tuning, pruning or distillation) of google/magenta-realtime-2 by Google DeepMind, redistributed under the same CC-BY-4.0 license. Use of the model remains subject to the upstream model's terms; you are responsible for what you generate.

Conversion code, validation harnesses and docs are Apache-2.0 in the GitHub repo.

Credits

• Google DeepMind, the Magenta team, for Magenta RealTime 2, SpectroStream and MusicCoCa, and for releasing on-device weights under a license that makes this work possible (magenta/magenta-realtime).
• Apple's coremltools team.
• Port by Matt Mireles. Related: kokoro-coreml.