Gemma-4-12B-CoreAI

Gemma 4 12B — Core AI (.aimodel)

google/gemma-4-12B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for Apple silicon by visible-cx. These are derivative artifacts: Google's QAT-trained weights rounded onto the int4 (or int8) grid and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or MLX. Gemma 4 12B is a dense model — no Per-Layer Embeddings and no gather-table sidecar, unlike the E2B and E4B bundles in this org.

The repo holds two generations of build. The three dense2in* folders are the original 2-input dense exports. tiered/ is the current one: a single bundle carrying seven pinned context capacities from 4,096 to 262,144, gated bit-for-bit against each other, for 4.54 MB more than a single-capacity bundle.

tiered/ is STAGED, not qualified. No tier has ever been run on a Mac, the multifunction bundle has never been loaded by the engine, and its compiled blob is a forecast. Every per-rung requirement below inherits that forecast. See The seven-rung ladder.

Known issue on the dense2in* bundles — memory growth on long generations. The Core AI runtime retains one compiled specialization per sequence-length signature, and these decode graphs make sequence length load-bearing, so a long free-form generation retains roughly 80 MB of GPU allocations per generated token until the process exits. Measured on the sibling E2B/E4B exports (~81 MB/token at 3.3k of context, guided and unguided alike). The tiered/ line does not have this defect — shape-stability was measured directly on its wkv_stable predecessor: 6,962 forward passes, one shape signature, stable=true on every one, with total process-footprint growth of 5.0 MB across the whole run.

v5/ — ring extent R = 2W: single-pass speculative verify (2026-09-02)

v5/gpu-pipelined/gemma4_12b_qat_decode_int4linsym_hsc_r2048_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k (main.mlirb SHA-256 bd67ee6c89d07f79…), sidecar v5/ios-frontend/gemma4_12b_tied_vocab/head.f16 (byte-identical to v4's, fb21807f…).

What changed, and only this. The sliding-attention ring holds 2,048 slots instead of 1,024 (capacity_tier_notes.ring_slots = 2048, sliding_window still 1024). Every weight, every rung, the head sidecar, prefill and the verify signature are v4's. The ring state per rung grows from 335 MB to 671 MB (localKeyCache/localValueCache [40,1,8,2048,256] fp16); the global cache and everything else are unchanged.

Why. With R = W (v4), a speculative verify block had to run twice — once with accepted_count = 0 to adjudicate, once to commit the accepted prefix — because a rejected draft's row, once written, would be admitted by later queries. With R = 2W the slot a rejected row occupies has an implied position (p = q − ((q − j) mod R)) that fails the window test (q − p) < W for every live query until the slot is legitimately overwritten. So the host may pass accepted_count = K on every call and continue from p + n: one verify pass per block. The two-pass protocol remains valid.

Evidence (torch level, 6-layer core on the conversion box, fp32).

Not yet verified on these bytes: an end-to-end greedy-stream check through the compiled verify_c65536 on a 16 GB Mac (three attempts were floor-killed by memory on a loaded machine, not by the bundle). The app pin remains v4 until the Swift engine wires the verify entrypoint; nothing routes to v5 today.

Contents

Each folder holds <name>.aimodel/ (main.mlirb, main.hash, asset metadata.json), a bundle-level metadata.json, and tokenizer/ (tokenizer.json 32,169,626 B, tokenizer_config.json 3,090 B, chat_template.jinja 18,683 B, generation_config.json). With the shared tokenizer the staged tiered/ bundle is 6.27 GiB on disk.

dense2in-pf64's weights are bit-identical to dense2in's; the multifunction build adds an S=64 prefill entrypoint for under a megabyte, because the exporter deduplicates weights across entrypoints.

Stop token: every bundle declares eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits. A host that stops on the raw upstream <eos> instead will overrun every reply.

Manifest context is metadata only — except on tiered/. On the dense2in* bundles --max-ctx sets language.max_context_length and nothing else, so the numbers above can be raised or lowered without a re-export (whether the result fits in memory is a separate question). On tiered/ the capacity is a real, pinned graph extent per entrypoint, and language.max_context_length reports the top rung — see the warning under Selecting a rung.

Provenance

"QAT-unquantized" means QAT-trained, stored full width; the int4 rounding happens at export onto the grid training already targeted. That is why int4 is the primary recipe rather than a quality compromise — the int8 variant is an anchor, not the default.

tiered/ — the seven-rung ladder

One bundle, seven main entrypoints, each with its own pinned KV capacity:

4,096 / 8,192 / 16,384 / 32,768 / 65,536 / 131,072 / 262,144

The top rung is the checkpoint's own text_config.max_position_embeddings, so the ladder spans the architecture end to end.

It is nearly free, and that was measured before it was built. A pinned capacity costs no compiled bytes: at c4096 and c16384 the same graph's main.mlirb differs by 26 bytes on a 6.7 GB file, because capacity is a runtime state allocation and nothing else.

Six extra capacities — including two that quadruple and octuple the previous top rung — cost 4.54 MB, 0.068% of the bundle, and dedup gets marginally better with more entrypoints: the four-rung build's cost law predicted 6,705,179,393 B for seven and the file came in 50,267 B under, 0.0007%.

The one idea that makes it work: the tiers are the same quantized module, copy.copy-ed so each carries its own capacity — a fresh __dict__ over the same _modules and _parameters, so every weight tensor is the identical object. Quantizing per tier would produce numerically identical but distinct tensors, and distinct tensors are what dedup cannot merge.

Build-time memory is the one place seven rungs are not free. Peak export RSS was 98.10 GB against 67.29 GB at four rungs and 67.28 GB at one. It is a build cost, not an artifact cost, but a machine with less than ~128 GB should not attempt a seven-rung export.

The cross-tier gates — bitwise, not "same SNR class"

Bottom rung against each higher rung, on identical weights, one S=1 decode step at a time, fp32, window 128, on the length-bounded Metal kernel's registered torch reference:

max|Δlogit| is exactly 0.0 at every step of every pair, so the verdict is bitwise identity rather than a tolerance. Wraparound is exercised by construction in all of them (92 ring writes land on a used slot at 220 tokens against a 128 window), and each rung carries exactly one shape signature on each side — the property the whole design rests on, measured per rung rather than assumed.

The top pair is the one that mattered: c4096 against c262144 is a 64× capacity ratio and 4,227,858,432 B of KV the small side never allocates, and it is bit-for-bit identical anyway.

What these gates do not touch: they exercise the bounded kernel's torch reference, not its MSL (there is no Metal on the export box), and no tier has been run on device.

The per-tier cost, and what a 16 GB Mac needs to hold it

pinned KV is exact arithmetic from the state shapes: a global pair [8,1,1,C,512] at fp16 is 16,384 B per token of capacity, and the sliding ring [40,1,8,1024,256] is 0.3125 GiB and identical in all seven rungs — it does not scale with capacity at all.

All GiB. The forecast is blob + pinned KV(C) + R, where the tiered blob is 6.241 GiB and R = 0.419 GiB is the non-KV resident remainder, obtained by decomposing the measured 7.22 GiB steady plateau at c16384. The c16384 row therefore reproduces a measurement by construction, and every other row is that residual carried across.

The right-hand column is a window class, not a yes/no. Comparing each requirement against a machine's median afternoon would assert that the only window a machine ever offers is its median one, and this project's own record says otherwise. What the 16 GB reference machine has actually been measured to hold:

So c4k/c8k/c16k are the ordinary case for a 16 GB Mac left alone for a few minutes; c32k is plausible on a quiet machine and not something to promise; c64k is reachable and not routine; and c128k/c256k are honestly out of reach on 16 GB. The 32 GB column is "untested", not "yes" — this project has no 32 GB machine, reclaimable is not a fixed fraction of RAM, and writing "yes" there would be a guess wearing a measurement's clothes.

There is no separate prefill-workspace adder. The prompt-length term was measured directly — +0.018 GiB per 1,000 prompt tokens, within 18% of the KV-per-token arithmetic the table already charges, and 20× below the naive extrapolation that had been feared. So the requirements above stand as published, including c131,072 at 10.22 GiB. With int8 KV the top rung would fall to 9.07 GiB; that lever is real, arithmetic and not yet built.

64k is the rung the evidence asks for

A retrieval-side audit of eleven real report questions against the app's own graph, run at each rung with no model in the loop:

65,536 is where the ladder changes state. Eight of the eleven questions want 39,786–40,021 tokens; a 64k window offers 61,184, so all eight become graph-bounded — their entire scoped universe fits, and no larger window improves those answers. 128k and 256k add nothing for them at 1.00 and 3.00 GiB more KV. On evidence per gibibyte, 64k dominates 128k. The other three questions want ~1,052,400 tokens and are unsatisfiable at every rung; what 256k buys them is that their need is finally computed against the whole pool instead of a truncated one.

Selecting a rung

1. Do not read the bundle's language.max_context_length to size a prompt. On a tiered bundle it is the top rung (262,144). A session that acquired main_c4096 and budgeted against 262,144 would fail silently: the engine's stop is min(requested, max(0, maxContextLength − input.count)), so it returns 0, yields nil before producing anything, and the stream finishes empty. This is the one place a tiered bundle can go wrong quietly.
2. metadata.json carries language.capacity_tiers[] — per rung function, context_window_tokens, global_kv_bytes, ring_kv_bytes, total_kv_bytes — plus capacity_tier_notes.global_kv_bytes_per_token, so a host can price every rung without introspecting the graph.
3. If that field is ever absent, the rungs are still recoverable from function_map's main role because each entrypoint is named for its capacity — but the KV cost is not in the name, and a host must not invent one. An unpriced ladder should be climbed from the cheapest rung, not the largest.
4. A downgrade is a new session. Re-acquiring rebuilds the cache and therefore discards the KV, so "between-batch downgrade" is honest and "mid-session downgrade" is not.

tiered/ is decode-only: its function_map is {"main": [...]} with no prefill function, so prompts are prefilled one token at a time through the same main.

Requirements

• Apple silicon Mac, Core AI runtime. Mac-only; this does not fit an iPhone or iPad.

• Engine contract: 2 inputs — input_ids [1,1], position_ids [1,S] → logits, no per-step mask. That is the contract both engines accept, and the logits-capable sequential engine is what grammar-constrained decoding needs.

• dense2in* states: keyCache / valueCache Float16, 48 × 1 × 8 × ? × 512, dynamic sequence dim → GrowingKVCache (initial 256, doubling). KV cost 786,432 bytes per token — 3.22 GB at 4096, 12.9 GB at 16384. This dominates the dense2in* envelope.

• tiered/ states: a global pair [8,1,1,C,512] at the selected rung plus a capacity-invariant sliding ring [40,1,8,1024,256]. See the per-tier table.

• dense2in* memory envelope:

  	int4	int8
  Weights (main.mlirb)	8.71 GB	14.67 GB
  + KV at 4096	~11.9 GB	~17.9 GB
  + KV at 16384	~21.6 GB	~27.6 GB

  Against a 16 GB Mac's Metal recommendedMaxWorkingSetSize of ≈ 10.7 GB, the int4 bundle's affordable context is roughly 2,500 tokens, and the int8 bundle does not fit that machine at all. Minimum practical machine memory for dense2in*: 32 GB.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Measurements

No bundle in this repo has been measured on Apple silicon. What follows is measured on a sibling build — gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_c16384, the single-capacity predecessor of tiered/, which is not published here — on a 16 GB M2 Pro under a watchdog. It is included because it is the only Gemma 4 12B evidence this project owns and because the tiered/ line inherits its graph.

Three things come out of it, and they point in different directions:

• Shape-stability held, under memory pressure severe enough to kill the run, and process footprint grew 5.0 MB across 6,962 passes. The wired memory is held driver-side, not in the process's own footprint. Respecialisation and leaks are both excluded.
• The 12B does not fit a 16 GB Mac at c16384. A steady 9.94 GiB wired against a machine whose ordinary reclaimable is 4.5–7.3 GiB is not a "wait for a quieter moment" case.
• Most of the 5.37 tok/s is thrash, not architecture. A dense 12B against an 8B-A1B activating ~1.5B parameters is perhaps a 6–8× compute ratio; the observed prefill ratio was 32×, on a machine that swapped 1.52 GiB and held reclaimable under 2.6 GiB for eighteen minutes. An unpressured estimate is prefill in the tens of tok/s — an inference from the ratio, not a measurement.

The plateau law is per-architecture, and this is what established that. Wired ÷ compiled blob: LFM2.5 8B-A1B MoE 1.092 peak (flat in prompt length); Gemma 4 E2B 1.101 against blob + PLE tables; Gemma 4 12B dense 1.157 steady / 1.348 peak, and not flat in prompt length. A single constant applied to every bundle under-charges the dense 12B by 0.33 GiB at its plateau and 1.51 GiB at its peak.

The c4096 re-export was run, and it bought 0.1875 GiB — the whole prize, because the sliding ring is five-sixths of the KV at c4096 and does not scale with capacity. Steady demand 7.22 → 7.03 GiB, requirement 8.28 GiB. It works, it is free, and it does not rescue the 12B on a 16 GB machine.

For context on the dense2in* bundles specifically, the only figure that exists is upstream's, on hardware not used here: approximately 33 tok/s decode on an M4 Max for the int4 12B. Read published Gemma 4 Q4 figures of 40–50 tok/s with care — those use Gemma 4's MTP speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode.

Usage

Swift Package Manager, via CoreAIKit — a community package, not affiliated with Apple, requiring macOS 27 beta:

.package(url: "https://github.com/john-rocky/coreai-kit", branch: "main")
// target dependency: .product(name: "CoreAIKit", package: "coreai-kit")

ModelID addresses a bundle as repo + path + revision; the subtree downloads from the Hub on first use and is cached afterwards:

import CoreAIKit

let model = ModelID(
    "visible-cx/Gemma-4-12B-CoreAI",
    path: "dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64")

var config = ChatSession.Configuration()
config.engineVariant = .sequential   // required for guided / grammar-constrained decoding
let chat = try await ChatSession(model: model, configuration: config)

Unlike the E2B/E4B bundles in this org, these are plain 2-input graphs: no PLE table sidecar, no static input buffers, and either engine will load them.

tiered/ needs a host that can name which of its seven main entrypoints to load and can size prompts against the selected rung rather than the bundle scalar. ChatSession takes one function name for the role today; driving a specific rung means loading the bundle directly and passing that entrypoint's name.

Digests

Taken independently on both ends of the transfer that moved it, and matching. Core AI .aimodel bundles are not byte-reproducible — the exporter is not deterministic even against itself — so integrity rests on digests of the exact published bytes rather than on a rebuild.

Status

No numerics gate has been run on device for any bundle in this repo. The producer fingerprint proves the toolchain, not the numerics.

License

Google publishes the upstream QAT checkpoint under Apache-2.0 with a license_link to the Gemma 4 license, and this repo mirrors that declaration. Use is governed by those terms and by the Gemma Prohibited Use Policy; the obligations travel with any redistribution of these bundles. The contribution here is the conversion and the qualification evidence, not the weights.

tiered-v3/ — an fp16 vocab path, seven rungs, prefill and verify

The Quality-tier build, and the vocab path carries no int4 at all.

The embedding is fp16 and bit-exact to the checkpoint. Measured on the real tensor: the fp16 cast of the bf16 table is lossless — worst |delta| 0.000e+00 — because bf16's 8 mantissa bits fit inside fp16's 11 and the table maxes at 1.3906, nowhere near fp16's range limits. Not an approximation of the checkpoint's values; the values themselves.

That replaces a 4-bit embedding whose cost had never been priced against the checkpoint. When it was, both candidates came out around 15–17 dB end-to-end (the previously published _e4 build measures 14.794 dB / 254-of-256 top-1), which is why this revision spends the bytes.

The head is fp16 too, because Gemma ties embedding and head: leaving the head int4 would keep a blockwise_shift_scale reconstructing a full [262144, 3840] fp16 table for the matmul. With both fp16 no full-table reconstruction remains anywhere on the vocab path.

Entrypoints. Seven capacity rungs (4k, 8k, 16k, 32k, 64k, 128k, 256k), each with a last-row-only prefill_c<N> at S=64, plus verify_c262144 at S=8 returning FULL [1, 8, 262144] logits — the shape a prompt-lookup or speculative loop needs to check eight drafted tokens at once. Every entrypoint has zero symbolic extents.

Not yet run on a Mac. No token generated, no decode speed claimed.

tiered-v3/gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k — 8,190,509,056 bytes

gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.hash            32 B  2d9b6fde48da0ec177fadaf5f90c53bf86d5dcb5cbaa35397713fa89a005e50b
gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.mlirb 8,158,295,378 B  57bf5d507ca70e001c58bf72ea20308b232cb56835bfe3e5fcde13ac017cb34d
gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/metadata.json           105 B  93893e9d87ff83172d1ca84eec61e939233e4956b3a677b49f873463c0e2499b
metadata.json                                                4,174 B  c782edb705134877ac58bac96c9ffe7e8010b24dbac3c929ed91cb4073a55fda
tier_report.json                                            17,968 B  b33ebdeb508d561805930a85f34acc2148dff2d91f20cab6e1ff2e2619229c19
tokenizer/chat_template.jinja                               18,683 B  ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4
tokenizer/tokenizer.json                                32,169,626 B  cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
tokenizer/tokenizer_config.json                              3,090 B  c3fd34e321798e184e53894aaa1d8c4b37d08daf1fb1826dbceb3dcda0c62d06