dragon.c

dragon.c: A From-Scratch C Inference Engine for Meituan LongCat-Flash-Thinking (560B MoE, FP8) on a 31 GB Consumer Laptop

bespoke ontology — field engineering record, 2026-07-05

📦 Source code available on GitHub: https://github.com/bespokeontology/dragon.c

Built and validated in one night on a Dell laptop (RTX 4070 8GB, 31 GB RAM, Ubuntu 24.04), plus a GPU tier the following session: 50× total speedup, ~6 s/token and falling.

---

Abstract

We present dragon.c, a from-scratch C inference engine that runs Meituan LongCat-Flash-Thinking-FP8 (560B parameters, Mixture-of-Experts) on a consumer laptop with 31 GB of RAM. The engine reads the model's native FP8 safetensors directly via mmap — no format conversion, no GGUF, no llama.cpp, no Python in the hot path — and streams 531 GB of weights through 31 GB of memory using the OS page cache as the eviction layer. Every mathematical operation was validated brick-by-brick against a PyTorch reference to floating-point precision, culminating in a full 28-layer forward pass whose output logits are IDENTICAL to the reference (same top-5 token IDs, logits within ~0.1). The engine generates coherent English. Through four measured optimization rounds (f32 LRU cache → native-precision compute → AVX2+LUT kernels → a GPU-resident attention tier on the RTX 4070) decode latency improved 50× to ~6 s/token, with output invariant across every optimization. We document every dead-end as a control. The build demonstrates the thesis this project exists to prove: the deficit in consumer-hardware AI is drivetrain, not engine — organization of computation and data movement, not parameter count.

---

1. Hardware and model

2. Method: validate every op, advance only on proof

The build discipline, applied without exception:

1. Dump ground-truth tensors from a working Python/PyTorch reference (ref/*.f32).
2. Write the C implementation of ONE operation.
3. Compare element-wise. Advance only when the diff is at floating-point-noise level.
4. Write the result (and every dead-end) into the permanent record before moving on.

This discipline caught real bugs at every stage and — critically — caught a corrupt reference (§6) that would otherwise have burned days.

3. Phase results (every brick, with measured error)

4. The three MLA constants nobody tells you about

Hand-deriving MLA from the paper failed (diff 1.7). Reading Meituan's modeling_longcat_flash.py line-by-line revealed three undocumented-in-practice details, without which the attention is wrong:

1. mla_scale_q_lora = 2.0 — q_pass AND q_rot are multiplied by 2.0.
2. mla_scale_kv_lora = 3.4641016151377544 (= 2√3) — applied to the kv-lora vector AFTER its layernorm.
3. Interleave-before-rope (use_mla=True path): before rotary embedding, the rope dims are reshaped [d/2,2]→transpose→[d] (even indices first, then odd). Softmax scale = 192^(-0.5) = 0.07216878. k_rot is computed ONCE per token and SHARED across all 64 heads.

With these three, hand-MLA matches the module to 6.9e-3 (bf16 noise). These constants are now permanent record; they never have to be rediscovered.

5. The engine (Phase 3): manifest + mmap + page-cache eviction

Design (the "game-engine pattern" — assets in a PAK file, OS does the paging):

• Manifest: Python pre-parses all 75 shard headers ONCE into manifest.bin — 43,756 fixed-width records (name, shard id, dtype, absolute byte offset, shape, scale-tensor location). C mmaps it and bsearches by name. No JSON parsing in C, ever.
• mmap everything: all 75 shards are mmap'd read-only at startup. get_tensor returns pointers into the maps. Weights are NEVER malloc'd/copied (in the final engine — see §8).
• Eviction = the kernel's page cache. 531 GB streams through 31 GB because Linux pages weights in on touch and evicts them under pressure. Zero eviction code. The OS is the streaming layer.
• OpenMP across output rows of every matvec (14 threads).

Result: full 560B forward, 74 s cold. Layer-0 through the native read path matches Python to 2.2e-3.

6. The control that saved the build: the reference was poisoned

While validating the full layer, our composition differed from the saved reference by 3.48 — for an hour this looked like a C bug. It was not. Root cause, found by checking weight.is_meta:

Meituan's router does self.classifier.weight (direct attribute access) instead of calling self.classifier(x). Our Python-side lazy-loading hook fires on __call__ — so it NEVER fired for the router. The router weight stayed on the meta device, the module's MoE routing was garbage, and the garbage was silently baked into the saved reference tensor.

Force-loading the router and regenerating gave a clean reference — which our C matched to 1.9e-6. The C was right all along; the reference was corrupt. Lessons, now permanent controls:

• Never trust a module-derived reference until 0 meta params is verified.
• Any op that touches .weight directly bypasses load hooks.
• dragon.c loads every weight explicitly and is structurally immune to this bug class — the strongest single argument for owning the engine instead of trusting a framework.

Other controls earned earlier and re-confirmed tonight: the BOS token (id 1) NaN-poisons layer 0 and must never be added; the model is a THINKING model — raw completion is out-of-distribution (it predicted "Berlin" in the top-5 for "The capital of France is" — capitals-space, correct neighborhood; clean Python predicts the identical top-5, proving this is model behavior, not an engine bug); the chat template + <longcat_think> is the in-distribution format.

7. Generation: the dragon speaks

Tokenizer stays out of C: Python dumps vocab.bin (131,072 length-prefixed decoded strings) once; C prints text directly. Prompt: the official chat template for "What is 2+2?" → <longcat_user>What is 2+2? /think_on <longcat_assistant><longcat_think>.

First generated words from the from-scratch engine:

"The user is asking a simple arithmetic question"

Fluent, on-topic chain-of-thought. No NaN, no gibberish. This sentence has remained byte-identical across every subsequent engine optimization — it is the correctness canary.

8. The speed war: three rounds, each measured, each a lesson

Round 1 — f32 LRU weight cache + MLA KV cache (dragon_fast.c): 88 s/token. FAILED to speed up. The KV cache itself is correct (reproduces recompute output exactly — "The user…" token-for-token) and kills the O(n²) attention recompute. But the LRU cached weights dequanted to f32, which QUADRUPLES fp8 and DOUBLES bf16: the dense MLPs alone become 51 GB of f32 — cannot fit 31 GB — 41% hit rate — thrash. Measured diagnosis: the bottleneck was never attention. It is WEIGHT BANDWIDTH — each token moves ~50 GB of weights.

Round 2 — native-precision compute (dragon_native.c, scalar): 27 s/token (3.3×). Delete every f32 weight buffer. The matvec reads the mmap'd bf16/fp8 bytes DIRECTLY, dequanting inline per element. Bytes moved drop 2–4×; the dense MLPs in native bf16 are 25 GB, which FITS — so the OS page cache holds the hot working set and "load once" happens for free, in hardware.

Round 3 — the matmul, written properly (AVX2 + LUT): ~9 s/token (10× total), prefill 2.8 s/token.

• fp8 via 256-entry lookup table. e4m3 has exactly 256 possible values. The scalar path was paying a ldexpf LIBRARY CALL PER WEIGHT BYTE of a 560B model. Precompute all 256 floats once; dequant becomes a table read (AVX2 vgatherdps, 8 lanes).
• bf16 via AVX2+FMA. bf16→f32 is literally a 16-bit left shift — free inside SIMD: load 8×u16 → widen → <<16 → that IS the f32 → fmadd. Two accumulators to hide FMA latency. 8 weights per instruction. The 131,072×6144 lm_head uses the same kernel.
• 128-column block structure of the fp8 scales folds the scale multiply out of the inner loop.

Round 4 — the GPU lights up (dragon_hybrid = dragon_native.c + dragon_gpu.cu): ~6 s/token (50× total). Profiling showed the 9 s spread across attention 2.6 + dense 2.8 + MoE 3.7 + head 0.3 — and that the box is fundamentally DISK-bound: the fixed per-token working set (~48 GB) exceeds 31 GB RAM, so every token re-reads tens of GB from NVMe. The correct GPU move on an 8 GB card is therefore NOT "put the model on the GPU" (impossible) and NOT "stream weights over PCIe" (16 GB/s — the trap): it is VRAM as a permanent third cache tier. We park the attention weights — 42 of 56 sub-blocks, 7.5 GB bf16 — in VRAM at startup and never move them again. A warp-per-row bf16 matvec kernel (one warp reduces one output row; __shfl_down_sync for the horizontal sum) computes attention on-card; sub-blocks that don't fit fall back to the CPU path transparently. The win is DOUBLE: attention drops 2.6 s → 0.5 s (5×), AND 7.5 GB leaves the RAM working set, so the page cache holds more of the dense/expert tiers and disk reads shrink across the board.

Output identical at every step — the generated sentence has never changed through a 50× speedup. Correctness never traded for speed.

9. What this proves (the thesis, in working code)

1. Real answers = the war won. The existential question — can a 560B run correctly on owned consumer hardware with owned code — is closed. Everything remaining is tuning, not truth.
2. The ScMoE shortcut is an owned moat. The architecture computes the MoE on block-0's output and applies it at block-1's end — a designed overlap window. Generic runtimes don't exploit it; an owned engine can compute MoE ∥ attention and hide expert-streaming latency inside attention math. This optimization is only accessible to someone who OWNS the engine.
3. C, not Python — the game-engine discipline. No game runs Python in the frame loop; inference IS a frame loop (tight, memory-bound, latency-critical). Python found the bugs (prototype); C runs the frame (engine). The Python stack's own flakiness (silent meta-weights, wedged processes, OOM kills) was demonstrated during this build and is documented above.
4. Bandwidth is the drivetrain. Every real speedup came from moving fewer bytes or moving them smarter (native precision, page cache, SIMD) — never from more FLOPs. The datacenter's answer to this problem is to buy more bandwidth. The drivetrain answer is to stop wasting it.

10. Remaining work (all de-risked, all speed/usability)

• fp8-quantize the dense tier: 25 GB bf16 → 12.7 GB with our own (bit-exact-validated) fp8 path — brings the fixed working set under RAM so disk only ever reads experts. Largest remaining lever.
• ScMoE overlap: thread the MoE against sub-block-1's attention (the moat, §9.2).
• Compressed-latent KV cache: store the 576-dim latent instead of expanded K/V for long context (matters beyond our current 64-token horizon).
• Sampling (currently greedy argmax), longer rope table, incremental vocab.bin mmap.

Artifacts here — full source included

• src/dragon_hybrid.c + src/dragon_gpu.cu — the engine (CPU AVX2/LUT + GPU-resident attention tier)
• src/dragon_native.c — CPU-only engine (no CUDA required)
• src/dragon_forward.c — the proven full-forward validator
• src/dragon_read.c, src/phase2*.c — the six validation bricks (reader, RMSNorm, fp8, MLA, MoE, full layer)
• tools/build_manifest.py + Makefile — make dragon / make dragon-gpu
• launch_card.png (the launch card) · dragon_speaks_card.png (the historic first-words shot) · manifest.bin (the 43,756-tensor index)

python3 tools/build_manifest.py /path/to/LongCat-Flash-Thinking-FP8 .
make dragon        # CPU-only
make dragon-gpu    # with the RTX 4070 attention tier
./dragon 16

— kapitoshkina.ai · bespoke ontology, 2026-07-05. This is the beginning.