maple-preview-gguf

⚠️ STATUS — READ THIS BEFORE DOWNLOADING

All three packs generate text on GPU: maple-f16.gguf (~385 tok/s, H200), maple-q4_k_m.gguf (GPU-verified), and maple-tq2_0.gguf (~97 tok/s on an RTX 4000 Ada; previously CPU-only)**, with the Maple-enabled llama.cpp fork at github.com/stamsam/llama.cpp (branch prism, rev 9ee03ee) — all layers GPU-offloaded.

Mainline llama.cpp CANNOT run any of these files. The maple architecture exists only in the stamsam/llama.cpp fork (the PrismML upstream does not have it), and the fork's ternary format is not interchangeable with mainline types.

Development artifacts of an ongoing port, shared openly.

---

Overview

This repository hosts GGUF conversions of deepgrove/maple-preview — a 20B-A1B ternary-weight Mixture-of-Experts reasoning LLM by DeepGrove (2026, MIT license) — plus the original BF16 checkpoint (in bf16-checkpoint/). The GGUF pipeline was built on the PrismML llama.cpp fork; the complete Maple runtime (converter + inference graph) lives in the stamsam/llama.cpp fork (branch prism).

Model summary

Hosted files

Tensor counts and shapes were verified by post-write inspection (GGUFReader): 291 tensors, architecture key maple, per-tensor types confirmed (f16: F16/F32; tq2_0: 168x type-35 + 2x Q4_0 + 121x F32; q4_k_m: 145x Q4_K + 25x Q6_K + 121x F32).

Status

The Maple compute graph is implemented and verified in github.com/stamsam/llama.cpp (branch prism). All three packs generate coherent text (tested: "What is 2+2?" → "The answer is 4."): maple-f16.gguf and maple-q4_k_m.gguf with all layers offloaded to an NVIDIA H200, and maple-tq2_0.gguf on an RTX 4000 Ada at ~97 tok/s (CPU: ~5.5 tok/s) and on a Jetson Xavier at ~14.8 tok/s.

What the graph implements (matching the MLX reference deepgrove/maple-preview-2bit-mlx):

• Hybrid attention — 3:1 SWA-512:GA: sliding-window layers (window 512, partial 64/128-dim RoPE, theta 10000) with full-attention layers every 4th (il % 4 == 3) that carry no RoPE (n_rot = 0).
• Flash-head QK — per-head RMSNorm (q_norm/k_norm) on the reshaped Q/K before RoPE; k_proj at 512 dims (4 KV heads x 128).
• MoE — 256 experts, top-8, moe_intermediate 512, clamp-7 SwiGLU (silu(min(gate, +7)) * clip(up, -7, +7)), fp32 softmax + renormalized routing.
• KV — sliding-window KV cache (window 512) for the SWA layers.

Known limitations:

• Ternary GPU support is new — tq2_0 (GGML type 35) got its CUDA kernels (dequant, get_rows, mmvq vec_dot) at fork rev 9ee03ee; big-batch matmuls use the dequant+gemm path (mmq kernels not yet ported).
• Fork-only — the maple arch lives in stamsam/llama.cpp (branch prism); mainline llama.cpp and the PrismML upstream fork cannot load these files.
• Early validation — verified for coherent generation on simple prompts; no benchmarks, perplexity, or systematic evals yet.

For reference implementations of the model itself:

• Apple Silicon: deepgrove/maple-preview-2bit-mlx (official MLX runtime)
• Elsewhere: the original PyTorch checkpoint (deepgrove/maple-preview) — or the bf16-checkpoint/ copy in this repo

Usage

These files only work with stamsam/llama.cpp (branch prism); mainline llama.cpp cannot load them.

# GPU (f16 or Q4_K_M packs)
llama-cli -m maple-q4_k_m.gguf -p "Your prompt here" -n 512

# Server (OpenAI-compatible API)
llama-server -m maple-q4_k_m.gguf --port 8080

maple-tq2_0.gguf runs the same way but executes on the CPU backend until CUDA kernels for the ternary type land. No Maple-specific CLI flags are required.

Quantization recipe

All steps run CPU-side on an NVIDIA H200 host: the converter and quantizer are CPU processes — no GPU compute is involved, and while CUDA is available to torch, it is unused in these steps.

1. Convert: convert_hf_to_gguf.py (PrismML fork, with the Maple converter in this fork) on the deepgrove/maple-preview checkpoint → maple-f16.gguf (dense reference).
2. Quantize: llama-quantize on maple-f16.gguf:
   • maple-tq2_0.gguf — per-tensor type mix mirroring the official maple-preview-2bit-mlx tiering:
     • fork ternary tq2_0 (type 35, ~2.06 bpw) for attn_q/attn_k/attn_v/attn_output and all ffn_*_exps;
     • Q4_0 for token_embd and output (lm_head);
     • F32/F16 for routers and norms.
   • maple-q4_k_m.gguf — uniform Q4_K_M over the entire model.

The maple architecture is registered in the fork's arch table with a dedicated llama_model_maple class (hparams, tensor map, and full compute graph). Note: an earlier attempt used the fork's type-42 Q2_0 layout, which the quantizer can write but no inference kernel can read — it was superseded by the type-35 tq2_0 format used in maple-tq2_0.gguf.

Caveats

• Fork-only formats, no interop: the fork's ternary tq2_0 (GGML type 35) and its Q2_0 variant (type 42) are custom layouts — mainline llama.cpp has different Q2_0/Q2_K types under the same names and cannot read these files. There is no compatibility layer.
• CPU-only ternary: maple-tq2_0.gguf has no CUDA kernels yet — expect ~5.5 tok/s CPU throughput on a 96-core host, not H200 speed.
• Q4 on ternary weights is wasteful: Maple's experts and q/o projections are already ternary in storage; a uniform Q4_K_M re-quantizes them to 4 bits with little precision benefit. maple-q4_k_m.gguf is a quantizer sanity check / compatibility data point, not the recommended format — the tiered maple-tq2_0.gguf mirrors the intended storage layout.
• Not the MLX format: the tiering philosophy matches the official MLX pack (2-bit ternary for projections+experts at 16 codes/u32, 4-bit for lm_head+embeddings at 8 codes/u32, BF16 for router+norms), but the binary formats differ. Files are not interchangeable between the two runtimes.
• Development artifacts, not a release: trial/experimental work on a custom fork; formats and tensor layouts may change without notice as the port progresses.

Development log

• Converted deepgrove/maple-preview (BF16 checkpoint, 20.214 B params / 18,651 tensors) → maple-f16.gguf via the fork's converter (CPU, H200 host).
• Quantized → maple-tq2_0.gguf (tiered ternary pack: 168x type-35, 2x Q4_0, 121x F32) and maple-q4_k_m.gguf (uniform Q4_K_M).
• Post-write verification: tensor counts and shapes inspected against the source checkpoint (2026-08-04).
• Graph implemented + verified (2026-08-04): hybrid SWA/GA attention, flash-head QK, 256-expert clamp-7 SwiGLU MoE, fp32 renorm routing — f16 and Q4_K_M packs generate coherent text on an H200 (~385 tok/s at f16, all layers GPU-offloaded).
• Ternary GPU kernels (rev 9ee03ee): CUDA dequant + get_rows + mmvq vec_dot for tq2_0 — verified on an RTX 4000 Ada at ~97 tok/s decode / ~111 tok/s prefill (CPU: 5.5 tok/s).
• Jetson Xavier verified (2026-08-04): maple-tq2_0.gguf runs CPU-only on the Xavier (aarch64, 8 cores, 14 GB RAM) at 14.8 tok/s decode / 37.5 tok/s prefill, coherent output.
• Next phase: quality benchmarks vs the MLX reference, CUDA build for the Xavier (sm_72) once its toolkit is set up.

Credits

• Model: DeepGrove — deepgrove/maple-preview (MIT)
• Official on-device runtime: deepgrove/maple-preview-2bit-mlx
• Base tooling: PrismML llama.cpp fork
• Maple runtime fork + conversion + packaging: stamsam/llama.cpp — MIT