dsv4-flash-homunculus

DSV4-Flash Homunculus

Scale models of DeepSeek-V4-Flash, for measuring inference rigs.

⚠️ The weights are random. This is not a language model.

These files have DeepSeek-V4-Flash's architecture and op graph, but every tensor is random and nothing was ever trained or distilled. They will happily load and generate at 300 tokens/s, and every one of those tokens will be garbage. Do not download this expecting a small DeepSeek-V4. They are test articles — measuring equipment shaped like a model.

The one thing they do carry from the real model is the tokenizer, verbatim (129,280 tokens) — see License and provenance.

---

What it's for

Tuning a large model across multiple machines is slow because every data point costs a model load. Changing one flag on a 145 GiB two-node setup costs five minutes before you learn anything, so a sweep of ten configurations is an hour of mostly waiting.

A homunculus is the same graph with most of the weight taken out. A configuration that takes 316 seconds to measure on the real model takes 2.9 seconds on the small one — 110×.

The catch is that a token costs bytes/bandwidth + n_layer × f, and a toy can only preserve one side of that sum. So this repo ships two instruments, one per term, plus a generator that makes more:

Neither instrument covers the bandwidth term itself, and neither needs to: bytes/token is computed from the real model's tensor table (no GPU, no load) and divided by measured bandwidth. That spreadsheet predicted a real requantization win to 0.1% (predicted +8.2% decode, measured +8.3/+8.2/+7.9% at three depths). Compute the bytes term; measure only the latency term.

The h1 result

Across 10 configurations measured on both models, predicted decode throughput for DeepSeek-V4-Flash landed within a median 0.80% of measured, 9 of 10 inside 5%, with no fitted parameters:

predicted_tg = 1000 / (read_budget_gb / gbs * 1000 + n_layer_real * f_toy)

• read_budget_gb — bytes read per token, computed from the real model's tensor table.
• gbs — the rig's memory bandwidth, measured once.
• n_layer_real — the real model's layer count.
• f_toy — per-layer overhead in ms, measured on the homunculus in about three seconds.

Decode t/s, two nodes over 100GbE, llama-bench pp512/tg128, 3 reps.

What f actually is

f is per-layer overhead: scheduler dispatch, the cross-device hop, ggml_backend_synchronize. It is not constant with depth. Measured on homunculi generated at four depths:

Run-to-run noise is CV 0.52%, so 3σ is 3.8 µs/layer. The two-node spread is 51 µs — 13.6× that band, and monotone. The single-node arm is flat, so the depth dependence belongs to the two-node path.

The shape is a fixed per-token cost divided by n_layer. Fitting f(n) = a/n + b gives a = 367 µs/token, b = 222 µs/layer across two nodes, and a ≈ 0 on one, with residuals inside 3σ. a is the per-token cross-device constant — final-logit sync, sampling, embedding gather. Dividing it by n_layer is what makes f look depth-dependent.

This sets the instrument's calibration. The real model's own 43-layer value is 253.25 µs/layer:

• a 12-layer toy reads 250.5 µs/layer — 1.1% below it
• a 43-layer toy reads 232.2 µs/layer — 8.3% below it

The depth-matched toy is the less accurate of the two, because two effects offset at 12 layers: the toy under-charges the true per-layer cost by ~9% (the real model has per-layer compute the toy lacks) while the 1/n term adds ~31 µs at that depth.

So the predictor is an empirical calibration at the (toy 12 → real 43) pairing, not a derivation. Re-anchor before applying it to a real model of different depth, and note h1-n43's raw f needs a ~+8.3% correction before comparing to the real anchor.

Pin your driver. The same config measures 251 µs/layer on stock RADV and 243 µs on a newer Mesa — an 8 µs shift, more than 2× the 3σ band. Every number here is stock-driver; record the driver in every result.

The h3 instrument

Every h1 failure below traces to one cause: h1 moves ~1/256 of the real bytes per layer. The lever that fixes it without the 144 GiB footprint: per-token expert read is experts_used × n_ff_exp × n_embd, which is independent of n_expert, and per-layer bytes are independent of n_layer. So h3 preserves width exactly and shrinks only what per-token bytes never touch:

Verified before generation: routed-expert parameters read per layer per token = 150,994,944 on both h3 and the real model, ratio 1.000; n_head, head_dim, rope all match real exactly.

What h3 is, measured: its serial two-node arm — the topology production actually runs — is precise at sd 0.9 µs/layer, making it the screen for serial-topology strategy questions. It gets byte-scaling signs right where h1 inverts them (the transport-fix replay reads +3.9 µs where h1 reads −29 and real reads +475 — right sign, but 1% of the real magnitude and inside its EP-arm noise of sd 33–64 µs/layer, so h3 screens strategies; it does not measure fence magnitudes).

What h3 is not: a serving tuner. See failure 6.

Where it fails

Known limits of the instruments as they stand. Each was measured, not assumed.

1. Depth. f is not layer-invariant; h1's predictor is calibrated at (toy 12 → real 43) and carries no guarantee at any other pairing.

2. Ranking a code change that alters bytes moved per layer — the sign can invert (h1). Reverting the two largest transport fixes of a fence-reduction campaign costs +475 µs/layer on the real model; on h1 the same revert is 29 µs/layer faster — wrong sign, ~5σ. The fix removes a per-layer transfer of 48 B and 32 KB; the 32 KB scales with n_embd, and h1's is 2 KB. Do not use h1 to decide whether a transport or graph change is worth keeping. A null control (GGML_VK_FAST_SYNC) reproduces correctly at +0.8 µs/layer — the instrument does not invent effects; it misranks the ones that move bytes. h3 gets this sign right but inside noise.

3. Overheads that are themselves proportional to bytes. Expert parallelism is the worked example and h1's only prediction outside 5% (+25.0%). Its overhead is a cross-device exchange of expert tensors; the toy's experts are 606× smaller, so it under-charges the fence. It still gets the sign right — real EP/serial 0.721, toy 0.438, both say EP loses — so it correctly rejected expert parallelism without a single 145 GiB load. Trust the ranking, not the magnitude. This gap is a pure footprint effect, established by elimination: it survives 4× sweeps of n_embd and n_ff_exp (excess flat within noise) and depth-matching at 43 layers (still 1.62× apart) — no structural correction the toy can express repairs it.

4. Pricing a cross-node boundary. Forcing attention weight classes onto the peer with -ot costs 618 µs/layer on h1 and 8176 µs/layer on the real model — 13.2× apart, against a pre-registered 2× kill threshold. Boundary cost is dominated by bytes crossed, not round trips. (Free-standing platform result: that real-model relocation costs 85% of decode — a per-layer serial round trip on this interconnect is ruinous, an upper bound relevant to any per-layer collective scheme.)

5. Anything measured in bytes per token. Load mode moves real prefill by 8.0% and the toy by 2.4% with no consistent ordering; at 244 MiB the weights are resident under every mode. Do not size a quantization win or a load-mode change on a toy — compute it from the tensor table instead (see above: that arithmetic is good to ~0.1%).

6. Serving runtime knobs — neither toy nor llama-bench can green-light one. The measured case: h3 screened a smaller -ub as +4% prefill; llama-bench on the real weights agreed (+8–9% prefill, two depths); the actual server measured the same change at −10% prefill, +11–12% TTFT, +4.1% worse end-to-end latency over an 18-cell serving deck. Reverted. llama-bench shares the toy's blind spots — no draft model, no chat template, no guided decode, different memory pressure — it is a different workload wearing the same weights. The pipeline is therefore three stages, and stage 2 is a second screen, not a confirmation:

1. screen on the homunculus — seconds, wide, sometimes wrong
2. check on llama-bench with real weights — minutes, still not the serve
3. gate on the end-to-end serving benchmark — the only stage that counts

7. Per-op profiler shares on a small homunculus are measurement artifact. When ops are tiny, per-op timing is dominated by query-pool serialization, not GPU work: a "28% hotspot" on a 380M homunculus, gutted to a trivial copy, moved wall clock <1%; an "RMS_NORM 36%" share was ~1% real. Never optimize from a small-model per-op profile — validate any hotspot by op-ablation first (gut the op, measure the wall-clock delta). The profile becomes trustworthy around 5.2B: generate h1-e1024 (--n-embd 1024 --head-dim 64) as the smallest honest kernel-profiling target.

Which instrument for which question

Open work

The known-incorrect bits that are work items, not conclusions:

• Whether any homunculus predicts end-to-end serving outcomes has never been tested — not for h1, not for h3. Both were qualified against llama-bench, which failure 6 shows is itself only a screen for a serving stack. This is the qualification that matters if the toys are to tune a serve rather than a bench.
• Large -b with a moderate -ub is unexplored and promising: the serving deck that killed the smaller--ub change also measured decode +6.9% at depth 24576 with unchanged quality, unexplained. If TTFT could be held while keeping that, there is a real win. Needs its own serving-benchmark run; do not screen it on llama-bench (wrong on this axis in both directions).
• h1-n43 needs its correction factor formalized (~+8.3% on raw f) before it can be used as a drop-in anchor-matched instrument.

The models

Bold values are preserved exactly — they determine which operators run and how the graph branches. h1 scales width away and keeps topology; h3 keeps width and scales depth/footprint away. Every h1 number above was produced with h1-topology-mxfp4.gguf.

Reproducing

Requires a llama.cpp build with deepseek4 support.

# one configuration, ~3 seconds
python qualify.py run \
  --model h1-topology-mxfp4.gguf \
  --profile m6 --bench /path/to/llama-bench \
  --rpc <peer>:50052 --straddle \
  --roofline-gbs <your measured GB/s> \
  --only rpc_split_even --out toy.json

qualify.py derives f per configuration (with the EP critical-path budget correction built in), computes read_budget_gb from any GGUF's tensor table, refuses configs whose per-device share exceeds --max-device-gib (toy-safe is not real-safe), records bench path / git commit / driver env in every result, and kills the whole process group on timeout.

gen_homunculus.py builds every model here. Presets h0, h1, h2 plus override flags:

PYTHONPATH=/path/to/llama.cpp/gguf-py python gen_homunculus.py --preset h1 --out h1.gguf
# the variants used above:
python gen_homunculus.py --preset h1 --n-layer 43 --head-dim 64 --rope-dim 8 --out h1-n43-f16.gguf
python gen_homunculus.py --preset h1 --n-embd 1024 --head-dim 64 --out h1-e1024-f16.gguf
python gen_homunculus.py --preset h1 --n-embd 4096 --n-expert 32 --n-layer 4 --out h3-L4-f16.gguf
# then: llama-quantize <f16> <out> MXFP4_MOE

Caveats for anyone reproducing

• f is normalised per layer, and h1 is 12 layers, not 43. The predictor multiplies f_toy by the real model's layer count. See the depth section — this pairing is a calibration.
• Pin --head-dim 64 alongside any --n-embd sweep, or the generator moves the derived head width and confounds the axis.
• Expert-parallel configurations must run against a build that actually has the TP/EP code. Otherwise the environment variables are silently inert and you measure a plain layer split while believing you measured EP. EP arms also scatter ~5–8 µs/layer (vs 1.3 µs serial) — they need their own repeats, and comparisons belong against a serial run on the same build.

Files

The h3 screening raws and the serving-deck results live on the measurement rig and are not yet mirrored here.

Measurement conditions

Two Strix Halo boxes (128 GB unified each), 100GbE, llama.cpp with two-node RPC. Roofline 209.25 GB/s measured per node. Both nodes pinned to dpm=high. llama-bench pp512/tg128, 3 repetitions. The real model is DeepSeek-V4-Flash-0731, MXFP4 routed experts with Q8_0 dense tensors, 144.3 GiB.

Numbers here are specific to that rig and driver build. The method is not — the claim is about which terms survive scaling, and that argument is hardware-independent even though the constants are not.

License and provenance

gen_homunculus.py, qualify.py, and the results are MIT, matching llama.cpp's gguf-py which the generator builds on. All tensor data in the GGUFs is randomly generated and carries nothing from DeepSeek-V4-Flash.

Two things are taken from the real model, and neither is a weight:

• The tokenizer, verbatim — 129,280 tokens, embedded in every GGUF here and shipped as assets/dsv4_tokenizer.json.gz because the generator cannot run without it. It is DeepSeek's, redistributed from DeepSeek-V4-Flash, and it is why h0-skeleton is 177 MiB despite having only 5 layers: token_embd dominates a model this small.
• The config values — layer counts, expert counts, and the hyper-connection / indexer parameters, i.e. the numbers in the comparison table above.

The tokenizer is present so that a homunculus tokenizes identically to the real model, which keeps prompt lengths and therefore batch shapes honest. Nothing about the measurement needs the tokenizer to be this tokenizer — any 129k-vocab tokenizer would give the same timings — so if its redistribution is inconvenient for you, substitute your own and regenerate.