Moonlight-V4-1B-h16d256-r8

Moonlight-V4-1B-h16d256-r8

An untrained, ~1B-parameter DeepSeek-V4-architecture configuration (no weights): DeepSeek-V4-architecture 1B config with 16 x 256 attention heads and ratio-8 compressed attention (no indexer). It is the small end of a Moonlight-style down-scaling of DeepSeek-V4, sized to pre-train on two 48 GB GPUs, with attention dimensions chosen so that the TileLang sparse-attention kernel fits GPUs with 99 KB of shared memory (Ada / consumer class). This variant replaces DeepSeek-V4's ratio-4 Compressed Sparse Attention (CSA) layers by ratio-8 compressed attention without a lightning indexer: 7 compressed layers at ratio 8, 6 Heavily Compressed Attention (HCA) layers at ratio 128 and 2 pure sliding-window layers. Every query attends to all causal compressed entries, so no top-k selection is needed.

The sibling repo keeps DeepSeek-V4's ratio-4 CSA layers with the lightning indexer (the faithful architecture, loadable by transformers). Sibling: akoumpa/Moonlight-V4-1B-h16d256.

Lineage

DeepSeek-V4 (Flash: 284B/13B, Pro: 1.6T/49B) keeps head_dim = 512 and 64 or 128 query heads; Moonlight's 16 heads were kept for the 16B analogue, and here 16 heads x 256 dims give the same 4096-wide query space as the 8 x 512 1B config with half the shared-KV width.

Architecture

Per-layer parameters: attention 7.9M (sliding) / 8.4M (ratio 8) / 8.4M (HCA); MoE 39.0M total, 8.3M activated (each expert 1.2M); mHC mixers 196,662. Embedding and head are 132.4M each. KV cache per sequence at 4K / 32K tokens (FP8 non-RoPE dims, bf16 RoPE dims): 1.7 MiB / 9.8 MiB. Core-attention FLOPs per generated token at 4K context: 0.09 GF, against 0.77 GF of linear layers. python count_params.py config.json reproduces these numbers (the script also reproduces the published sizes of Moonlight, DeepSeek-V3, V4-Flash and V4-Pro).

Design rules: head_dim and index_topk are powers of two and index_n_heads divides 128 (kernel requirements); o_lora_rank 1024 keeps DeepSeek-V4's per-group output projection shape; routed_scaling_factor follows Moonlight's recipe (expected 1/||p||_2 of renormalised top-k scores) applied to sqrt-softplus with 32 experts / top-6.

Files

Loading

transformers

transformers' native DeepseekV4Config only maps compression ratios 0 / 4 / 128 to layer types and raises KeyError: 8 on this config. Load it with NeMo Automodel's DeepseekV4Config (below) or with DeepSeek's reference inference/model.py via inference_config.json. The sibling repo akoumpa/Moonlight-V4-1B-h16d256 is transformers-loadable.

The same applies to AutoTokenizer, which parses config.json first. Load the tokenizer directly:

from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("akoumpa/Moonlight-V4-1B-h16d256-r8")     # 129,280 tokens, BOS 0, EOS 1

NeMo Automodel (native training implementation)

from nemo_automodel.components.models.common import BackendConfig
from nemo_automodel.components.models.deepseek_v4.config import DeepseekV4Config
from nemo_automodel.components.models.deepseek_v4.model import DeepseekV4ForCausalLM
cfg = DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-1B-h16d256-r8")
backend = BackendConfig(attn="eager", linear="torch", rms_norm="torch_fp32", rope_fusion=False,
                        dispatcher="torch", experts="torch_mm", enable_hf_state_dict_adapter=False)
model = DeepseekV4ForCausalLM(cfg, backend=backend)
model.initialize_weights(dtype=torch.bfloat16)

Or in a recipe: NeMoAutoModelForCausalLM.from_config with config: DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-1B-h16d256-r8") (see training/pretrain.yaml).

DeepSeek reference code

inference_config.json drops into the inference/ folder of the DeepSeek-V4 release (ModelArgs keys, n_mtp_layers 0).

Training from scratch

The released implementations only ever load checkpoints, so two things must be initialised by hand; training/init_utils.py does both and training/train.py calls it after the recipe's own weight init on a fresh start:

1. Hash-routing table. tid2eid is created as zeros (every token to expert 0). fill_hash_tables writes a balanced token-id hash: 32 experts, 6 distinct experts per token, equal load over the vocabulary.
2. mHC mixers. NeMo Automodel leaves the fn / base / scale tensors uninitialised; init_hyper_connections applies transformers' rule (normal(0, 0.02) projection, zero bias, unit gates).

Recipe notes that cost time to find (all encoded in training/):

• Automodel wraps DeepSeek-V4's fp32 tensors (attention sinks, compressor position biases, mHC mixers, lm_head) as their own FSDP2 units whose forward returns the parameter; if they reshard after forward, attention reads a freed tensor. train.py calls set_reshard_after_forward(False) on those units.
• Use the logits-based MaskedCrossEntropy: the fused linear cross-entropy rejects the fp32 lm_head x bf16 hidden states.
• For iterable datasets the recipe passes no batch size to the DataLoader; set dataloader.batch_size explicitly.
• NanogptDataset is an infinite stream; validation uses FiniteNanogptDataset.
• The indexer's top-k has no gradient path (Automodel freezes its parameters). This variant has no indexer, so nothing is left untrained.
• To train beyond 4K, add DeepSeek-V4's YaRN block (rope_scaling: factor 16, original_max_position_embeddings 65536) and raise max_position_embeddings.

Measured on two RTX 5880 Ada (48 GB, sm_89) GPUs, bf16, torch_mm experts, chunked cross-entropy, single-GPU forward+backward (TileLang sparse attention with Sinkhorn on torch; the eager path for the same model reaches 4.2k tok/s at B=2 x 2048 and runs out of memory at B=4):

Forward-time breakdown at B=2 x 2048: attention 49 ms, MoE 36 ms, mHC mixers 34 ms, compressor 10 ms (147 ms forward). With the recipe's FSDP2 data parallelism over 2 GPUs and AdamW, a 32-sequence x 2048-token global batch is a reasonable starting point (training/pretrain.yaml).

TileLang kernels

Automodel's vendored Miles/TileLang kernels (sparse attention, indexer) and DeepSeek's TileKernels Sinkhorn were written for Hopper's 227 KB of shared memory. On a 99 KB-per-block GPU:

Automodel currently switches all three together (backend.attn: tilelang); the measurements above used per-kernel selection (only sparse attention on TileLang). On Hopper GPUs the default head_dim 512 kernels fit and the 16 x 256 choice is optional.

Limitations and notes

• No trained weights are provided; all numbers are architecture-derived or short from-scratch measurements.
• The name follows the Moonlight / DeepSeek-V4 lineage for clarity; this repository is not affiliated with Moonshot AI or DeepSeek.
• The tokenizer files are DeepSeek's (MIT licence, deepseek-ai/DeepSeek-V4-Flash).

References

• Liu et al., Muon is Scalable for LLM Training (Moonlight), arXiv:2502.16982
• DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv:2606.19348
• Xie et al., Manifold-Constrained Hyper-Connections (mHC), 2026
• Roller et al., Hash Layers for Large Sparse Models, NeurIPS 2021
• DeepSeek-AI, DeepSeek-V3.2 (DeepSeek Sparse Attention, lightning indexer), 2025
• Kernels: Miles (sparse attention / indexer, vendored in NeMo Automodel), TileKernels (Sinkhorn), TileLang