sweet-search-coreml-cascade

sweet-search CoreML variant cascade

A cascade of 12 CoreML .mlpackage variants used by sweet-search for fast code-search inference on Apple Silicon M3+ (Apple Neural Engine).

What this is

• 6 NomicBERT embedding variants (embed/nomic_bert_b{B}_s{S}_fp16.mlpackage.tar.gz) covering the stair-step shape set the sweet-search indexer's cache-aware bucketer actually produces in practice:

  batch	seq	rationale
  64	96	hardCap × short seq
  64	192	hardCap × short-med
  32	384	tokenBudget regime
  16	512	long, cache-bound start
  4	1024	long tail
  1	2048	extreme tail / oversize docs

• 6 ModernBERT late-interaction variants (li/li_modernbert_b{B}_s{S}_fp16.mlpackage.tar.gz):

  batch	seq	rationale
  128	48	upperCap × very short
  128	128	upperCap × short
  64	256	medium
  16	512	long, cache-bound start
  4	1024	long tail
  1	2048	LI max length

How it's used

sweet-search init on an Apple Silicon M3+ Mac detects the hardware via core/infrastructure/hardware-capability.js::detectHardwareCapability() and downloads these tarballs alongside the other models. Each tarball is verified against a SHA256 checksum in core/infrastructure/coreml-cascade.json and atomically extracted into ~/.cache/sweet-search/models/coreml-cascade/.

Non-eligible hardware (Intel Mac, Linux, Windows, M1/M2) never downloads the cascade — the capability check filters these entries out before fetch.

Source

All variants are produced by scripts/spike-coreml/trace_cascade.py in the sweet-search repo, which:

1. Loads the source safetensors (nomic-ai/CodeRankEmbed and lightonai/LateOn-Code)
2. Wraps each model with mean-pool + L2-normalize (embed) or projection + per-token normalize (LI) as pure PyTorch
3. Traces at each (batch, seq) pair via torch.jit.trace
4. Converts to CoreML mlprogram format via coremltools.convert with compute_precision=FLOAT16, compute_units=ALL

Compile targets compute_units=CPU_AND_NE at load time in the Rust addon — the CoreML GPU path miscompiles attention/MLP/norm for both models and produces silently-wrong output (see scripts/spike-coreml/test_coreml_*_parity.py in the sweet-search repo for the broken GPU cosine measurements).

Rebuilding

git clone https://github.com/panonitorg/sweet-search
cd sweet-search
# Set up Python venv with torch + coremltools (one-time)
python3 -m venv scripts/spike-coreml/.venv
source scripts/spike-coreml/.venv/bin/activate
pip install torch coremltools safetensors packaging numpy huggingface_hub
# Build cascade directly into the managed cache
node scripts/build-coreml-cascade.js

After re-tracing, recomputing SHA256 checksums, and updating core/infrastructure/coreml-cascade.json, upload to this repo via huggingface_hub.upload_file.

License

Apache-2.0 — same as the upstream models.