maple-preview-speedy-colibri-int2

maple-preview-speedy-colibri-int2

int2 container for colibrì, a Rust MoE inference engine for single-box streaming-expert inference.

This repository exists so the model can be downloaded and run directly, with no conversion step. Maple is small enough that converting it yourself is easy too — coli convert /path/to/deepgrove/maple-preview maple-preview-container takes minutes, not hours. Downloading this container saves the upstream 40.4 GB BF16 checkpoint fetch as well as the conversion.

• Upstream: deepgrove/maple-preview
• Base model: deepgrove/maple-preview
• Architecture: 20B-A1B, 24 layers, GQA (16Q/4KV, head_dim 128, partial rope 64), per-layer QK-norm, 256 experts top-8 with a softmax router and no shared expert, clamped SwiGLU (limit 7.0), 3:1 sliding-window(512)/full attention interleave with NoPE on the 6 global layers
• Expert format: int2 (ternary), bit-exact — see below

Use

The engine is github.com/GriffinPilz/SpeedyColibri. With it cloned and built, one command downloads this container and one serves it. Build from main — the bf16 IO tier this container uses landed in #63:

scripts/fetch.sh maple-preview
scripts/serve.sh maple-preview 8080          # OpenAI-compatible API

serve.sh fetches the container itself when it is missing, so the first line is optional — it is there for when you want the download to happen at a time you chose. Both are idempotent: over a complete directory they verify it, over an interrupted one they finish it.

For a one-shot generation without the server, coli gen maple-preview-container "<prompt tokens>".

Pulling it by hand works too, but the directory name is what the registry resolves, so use exactly this one (under your COLI_MODELS_ROOT, default ~/models) or serve.sh will not find it:

huggingface-cli download Kanposer/maple-preview-speedy-colibri-int2 --local-dir maple-preview-container

Compression is free here — literally

Maple is the one model in this fleet where the container costs nothing in quality:

Nothing is rounded, clipped, or requantized. The container is a repack.

Measured on one DGX Spark (GB10, 121.7 GiB), 2026-08-08

Single sequence, greedy, 512-token prompt. Median of repeated runs on one build, gated on token identity — a "faster" number that changed the output fails loudly instead of being reported as a win.

The serving figure is a 32-token REQUEST rate, so it divides by the whole HTTP round trip and a cost paid once lands on all 32 tokens: 109.0 is 19 ms + 8.57 ms x 32 tok, where the 8.57 ms is the same engine the decode row measures.

It got there in two steps, both of which were fixed costs rather than a slower engine. It read 70.5 until the server's accept loop stopped sleeping 100 ms between connections — a nap that is harmless on a model spending ~1100 ms per token and was 15-24% of the wall clock on this one. It read 91.1 until the expert path stopped refusing to group a prefill: its grouped int2 kernel required one row per expert, so a short prompt fell back to ~30-40 launch triples per layer across 24 layers. Grouping those cut the per-request fixed cost from 52 ms to 19.

Read the decode pair carefully. colibrì's decode timer brackets forward() and stops before the lm_head matmul. On this model that omission is not a rounding error — the head is 2.5 ms of an 8.9 ms token — so the honest end-to-end figure is 112.1 tok/s, and the 155.8 is what the forward pass alone sustains. Both are printed rather than only the flattering one.

That decode figure is a median of EIGHT. It read 116.3 until four powered suites put it at <!--hist--> 112.1 — three warm suites agreeing at 112.80/112.13/110.62, one cold suite at 85.47 — and the 116.3 was a single median-of-3, the top draw of a spread whose other samples that day <!--hist--> were 113.12 and 107.15. Decode is the noisiest figure on this card.

Prefill moved most of all, 160.2 -> 230.0 tok/s, from the same expert-grouping change. The gate that blocked it carried a comment saying grouping does not pay on long prefill; sweeping prompt length with the path forced on says grouped wins at every length tried, from 2.07x at 4 tokens to 1.75x at 512.

Maple is in a different regime from the larger containers in this collection. At 5.3 GB of weights on a 121 GB box, nothing streams and nothing evicts: expert-load measures 1 ms. Read these numbers as "what this engine does when the model fits", not as a speedup over models that are bound by reading experts off NVMe.

Method, caveats and the full fleet table: SpeedyColibri README.

Verified against the reference implementation

Maple's own modeling_maple.py was run unmodified — only fa3.py was swapped for an SDPA-backed equivalent, since the published one hard-requires a FlashAttention build — and compared two ways:

• Residual stream, layer by layer: all 23 comparable layers agree within 0.06–3%, including all six NoPE/global layers, which a wrong router, window or rope would not produce.
• Teacher forcing over 32 positions: top-1 agrees 31/32. The single disagreement is a 0.125-logit near-tie inside the reference's own top-2 (19.750 vs 19.625) — roughly two ULPs of the bf16 it computes in, where colibrì runs f32.

What was changed

Weights were repacked into colibrì's container layout: routed experts into coalesced per-expert spans, ternary tensors into 2-bit codes, dense tensors into bf16. No fine-tuning, distillation, or other modification of model behaviour was performed.

Licence and attribution

This is a derivative of deepgrove/maple-preview and is distributed under the upstream licence, MIT.

All rights in the original model remain with its authors. Please read and comply with the upstream licence before downloading or using these weights.