nanowhale-100m-csa-fp8

nanowhale-100m-csa-fp8 (DeepSeek-V4 fat-attention clone, vLLM-validated geometry)

A ~161M miniaturized DeepSeek-V4 clone that keeps the exact architecture and weight naming of the full model so it is a drop-in for vLLM (model_type: deepseek_v4, FP8). Purpose is inference-stack validation — exercising the real fat attention path (Multi-head Latent Attention, Compressed Sparse Attention + Lightning Indexer, Heavily-Compressed Attention, sliding-window, MoE, manifold Hyper-Connections) at tiny scale — not generation quality.

⚠️ WORK IN PROGRESS — partially trained. This snapshot is an early pretraining checkpoint (~step 2000, train loss ≈ 4.2). Output is not coherent yet; it is published so the format/geometry can be loaded and tested in vLLM. A fully-trained (curriculum pretrain → SFT) version will replace it.

Why the geometry is what it is

vLLM's deepseek_v4 backend uses hand-written CUDA kernels (FlashMLA, fused qnorm-rope-kv-insert, DeepGEMM lightning indexer) that hardcode the production MLA geometry. To be a genuine drop-in, this clone must match them exactly, shrinking only elsewhere:

FP8: block-quantized e4m3 (128×128 blocks, ue8m0 scales), quantization_config.quant_method = "fp8".

Load in vLLM

from vllm import LLM, SamplingParams
llm = LLM(model="Shrinkinator/nanowhale-100m-csa-fp8",
          kv_cache_dtype="fp8", enforce_eager=True, max_model_len=1024)
print(llm.generate(["The capital of France is"], SamplingParams(max_tokens=16))[0].outputs[0].text)

Needs a CUDA-13.0-matched toolchain for the DeepGEMM/TileLang JIT, and (for this miniaturized hidden size) the mHC ops routed to their torch fallback. See the shrink project notes.