Nanbeige4.1-3B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->

Measured decode — iPhone 17 Pro: 16.9 tok/s · Mac (M4 Max): 105 tok/s (DeviceMark row nanbeige-3b, int8hu bundle · data)

<!-- gen-cards:devicemark end -->

Nanbeige4.1-3B (text decoder) — Core AI

[!NOTE] Update 2026-07-17: gpu-pipelined-b2/ adds the nanbeige4_1_3b_decode_int8hu_block32_sym bundle re-exported with coreai-core 1.0.0b2 (loadable on the OS 27 beta 3 toolchain; the original b1 tree is retained). This is the exact artifact measured and iPhone-gated on DeviceMark (PB nat 24/24 · oracle 24/24, iOS 24A5380h).

Plain-Llama dense decoder (Nanbeige LLM Lab): 32 layers, GQA 20 q / 4 kv heads, head_dim 128, hidden 2560, SwiGLU intermediate 10496, vocab 166144 (untied lm_head), RMSNorm eps 1e-5, RoPE θ=70M, context 262144 — no QK-norm, no qkv/mlp bias (the textbook Llama shape; model_type: "llama", 3.93B total / ~3B non-embedding backbone). Source: Nanbeige/Nanbeige4.1-3B (Apache-2.0). A reasoning / agentic model whose first-party card claims it beats Qwen3-4B and rivals Qwen3-32B / Qwen3-30B-A3B (LiveCodeBench-Pro-Easy 81.4 vs 40.2, AIME 2026-I 87.4, GPQA 83.8) — a 32B-class reasoner at 3.93B, running on an iPhone.

⬇️ Converted .aimodel bundle (ready to run): mlboydaisuke/Nanbeige4.1-3B-CoreAI — gpu-pipelined/nanbeige4_1_3b_decode_int8hu_block32_sym_s1/ (full LanguageBundle incl. tokenizer).

The first plain-Llama model on the pipelined-engine fast path: it reuses qwen3.py MINUS the q/k-norm (qwen3 already has a bias-free fused QKV), so the body is the existing overlay with one norm removed — see models/macos/llama.py. Pure-attention, KV-only state (no conv / recurrent), so it needs no engine patch beyond the base stack.

<!-- gen-cards:use-it begin id=nanbeige4.1-3b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->

Use it

New to Core AI? Start with CoreAIKit 0.7.3. Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

Measured decode — iPhone 17 Pro: 16.9 tok/s · Mac (M4 Max): 105 tok/s (DeviceMark row nanbeige-3b, int8hu bundle · data)

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let tldr = try await CoreAI.summarize(text, options: .model("nanbeige4.1-3b"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone --branch 0.7.3 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Nanbeige4.1 3B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run -c release chat-cli --model nanbeige4.1-3b --prompt "What can you do, offline?"

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let id = "nanbeige4.1-3b"
let chat: ChatSession
if id == "qwen3-0.6b" {
    // Freeze the release starter; other selections retain the live catalog's
    // model-specific dispatch (including paired Gemma bundles).
    guard let model = ModelCatalog.builtin.entry(id: id)?.modelID else {
        throw CoreAIKitError.modelNotAvailableOnPlatform(id: id)
    }
    chat = try await ChatSession(model: model)
} else {
    chat = try await ChatSession(catalog: "nanbeige4.1-3b")
}
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

When Apple's FoundationModels built-in model isn't enough, keep your session code and swap the model — one line. CoreAIKit's KitLanguageModel plugs supported chat bundles into the system LanguageModelSession. Tool calling depends on the model's dialect; guided generation also requires a compatible sequential engine. Check the linked support matrix before using either capability.

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.3) → product CoreAIKit
• Info.plist: none needed
• Entitlements (iOS): com.apple.developer.kernel.increased-memory-limit
• First run downloads the model — ~4,600 MB (Mac) / ~4,600 MB (iPhone) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

<!-- gen-cards:use-it end -->

Measured (macOS + iOS 27 beta, release builds, p=128 g=256, COREAI_CHUNK_THRESHOLD=1)

• Loads on iPhone 17 Pro: cold GPU specialization engine ready 53.5 s, device free 51 GB, no jetsam / no std::bad_alloc — the largest bundle we have run on the pipelined bench (4.58 GB payload).
• --static-ids is REQUIRED for the device. The generic dynamic-input_ids export is fast on the Mac but on the iPhone pipelined engine (chunkThreshold=1, every step S=1) it pays a per-step input_ids re-specialization that is pathological on a 4.3 GB model (~37 s/step cold; the 900 s probe never finished the first 24-token run). Fixing input_ids at [1,1] (the qwen3.5 loop-free device pattern; --static-ids → _s1 bundle) eliminates it — chunkThreshold=1 feeds S=1 anyway, so no prefill loss — and the device numerics complete 24/24.
• The untied 166144-vocab head is ~0.85 GB; quantize it absmax per-block-32 int8 (--head-sym, plain symmetric). symmetric_with_clipping craters big-vocab heads (the documented qwen lever).

int4: NO-GO — int8 is this reasoning model's floor

int4hu (body int4 per-block-32 + int8 head) is 2.9 GB and 169 tok/s on the Mac, and its raw single-token greedy still returns "Paris" — but multi-token reasoning CRATERS: the same "17 sheep, all but 9 run away" trick collapses to a wrong "17" with a repetition loop and Chinese drift. The single-token probe is misleading for a reasoning model — you must check multi-token. This is the non-QAT-int4 structural cliff (same wall as qwen3.5 / LFM2.5; needs QAT). Palettized (k-means) int4 does not rescue it either — for non-QAT weights the cliff is the scheme-independent property, and on the GPU-pipelined path the LUT dequant is slower than linear besides. int8hu ships.

ANE: right architecture class, wrong size

Plain-dense is the one class that could ride the ANE (where the LUT-friendly palettized weights run native-fast, unlike on the GPU). But the ANE sweet spot is the ~0.6–1B rung (tied head): a 0.6B fully-palettized model rides the ANE blazing. At 3.93B + a 166144 untied head Nanbeige overruns the ANE working set, so it ships GPU-pipelined like the rest of the dense line. The ANE-blazing target is a 1B plain-dense model, not this one.

Numerics gating

• Parity ladder (fp32 eager vs native HF LlamaForCausalLM oracle, no trust_remote_code): teacher-forced top-1 24/24, cosine 1.000000, max-abs-logit Δ = 0 (_smoke/test_nanbeige_parity.py, USE_HF_IMPL=true).
• Engine gate: raw-token greedy on the int8hu bundle reproduces the fp32 oracle's first token ("Paris"); reasoning output coherent and correct.
• Device gate: iPhone greedy sequences 24/24 token-identical to the Mac reference on both fixed prompts (_smoke/gen_nanbeige_device_ref_tokens.py). Reasoning models drift on a bare prompt after the answer — the first token is the anchor (Paris 9965 / Tokyo 20150) and the full 24 still matched here.

Convert it yourself

cd coreai-models   # with the plain-Llama overlay (models/macos/llama.py) in place
# device ship (REQUIRED static [1,1] for fast iPhone decode):
.venv/bin/python ../coreai-models-community/conversion/export_nanbeige41_decode_pipelined.py \
    int8hu --head-sym --static-ids
COREAI_CHUNK_THRESHOLD=1 ./.build/out/Products/Release/llm-benchmark \
    --model exports/nanbeige4_1_3b_decode_int8hu_block32_sym_s1 -p 128 -g 256 -n 3

Run contract: COREAI_CHUNK_THRESHOLD=1 before engine creation; the bundle's input_ids is static [1,1], so every prefill token is fed as an S=1 step (never call engine.warmup() — warm with a 1-token generate; llm-runner needs --warmup exact --warmup-length 1).

License

Model weights and conversion code: Apache-2.0 (Nanbeige LLM Lab upstream; the conversion code in this repo is BSD-3-Clause). Redistribution retains the upstream notices.

<!-- funnel:v1 -->

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->