Moonlight-V4-16B-A3B

Moonlight-V4-16B-A3B

A proposed, untrained DeepSeek-V4-architecture configuration at Moonlight-16B-A3B scale (no weights). Moonlight-16B-A3B is the DeepSeek-V3 architecture at "DeepSeek-V3-Small" scale (2048 hidden, 27 layers, 64 experts), trained by Moonshot with Muon. This config is the same idea for DeepSeek-V4: it keeps Moonlight's width, depth, expert count and expert size, and swaps in every V4 architectural change: hybrid Compressed Sparse / Heavily Compressed Attention, manifold-constrained hyper-connections (mHC), a hash-routed first MoE layer, shared-KV MQA with a grouped low-rank output projection, sqrt-softplus routing, attention sinks and clamped SwiGLU. COMPARISON.md in this repo is the full side-by-side analysis against Moonlight; this card is the summary.

Counts are computed from the tensor shapes of the released implementations (count_params.py, which also reproduces the published sizes of Moonlight, DeepSeek-V3, V4-Flash and V4-Pro) and were confirmed by instantiating DeepSeek's reference inference/model.py and transformers' DeepseekV4ForCausalLM on the meta device.

Lineage

Architecture

Per-layer parameters: attention 18.9M (sliding) / 28.4M (CSA: core 18.9M + compressor 4.2M + indexer 5.4M) / 21.0M (HCA); MoE 562.4M total, 60.7M activated (each expert 8.7M); mHC 393,270. Layer totals: sliding 581.7M / CSA 591.3M / HCA 583.9M, activated 80.0M / 89.5M / 82.1M. Embedding and head are 335.5M each.

Where the budget moved relative to Moonlight: all 27 layers are MoE (Moonlight's layer 0 is a 69M dense MLP), the per-token FFN shrinks from 8 to 7 expert-widths (one shared expert instead of two), and attention roughly doubles (0.66B vs 0.37B) because a 512-wide shared-KV head with the low-rank q/o path, compressors, indexers and sinks is heavier than 16 MLA heads. Net: activated parameters within 3% of Moonlight's, hence the same 16B-A3B name.

Long-context efficiency

KV cache per sequence (Moonlight: bf16 MLA latent; this config: FP8 non-RoPE dims + bf16 RoPE dims + FP4 indexer keys):

Core-attention FLOPs per generated token: 2.26 GF vs 0.79 GF at 8K, 290 GF vs 59.4 GF at 1M (linear layers: 5.16 vs 5.29 GF). A CSA query attends to 128 window entries + 512 selected compressed entries, an HCA query to 128 + L/128, so attention cost is nearly flat in context length.

Files

Loading

import torch
from transformers import DeepseekV4Config, DeepseekV4ForCausalLM
cfg = DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-16B-A3B")        # transformers >= 5.8; legacy keys are folded into layer_types / rope_parameters
with torch.device("meta"):
    model = DeepseekV4ForCausalLM(cfg)                    # 16.528B parameters

NeMo Automodel: nemo_automodel.components.models.deepseek_v4.config.DeepseekV4Config.from_pretrained("akoumpa/Moonlight-V4-16B-A3B") with NeMoAutoModelForCausalLM.from_config (the native implementation supports FSDP2 / EP / PP / CP and TileLang kernels). DeepSeek reference code: inference_config.json is a drop-in ModelArgs file.

Tokenizer

The config uses Moonlight's vocabulary: 163,840 embedding rows, of which 163,593 ids are defined by the tokenizer (163,584 BPE tokens + 9 special tokens; BOS 163584, EOS 163585; the remaining rows are padding, as in Moonlight). Moonshot's tokenizer files (tiktoken.model, tokenization_moonshot.py, tokenizer_config.json, MIT licence) are bundled. transformers >= 5.8 builds a fast tokenizer directly from tiktoken.model and reproduces moonshotai/Moonlight-16B-A3B's encodings exactly (verified on English, Chinese, code and special-token probes); older transformers versions use the bundled remote-code class and need trust_remote_code=True and tiktoken:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("akoumpa/Moonlight-V4-16B-A3B")

Training notes

• No weights. Everything here is architecture-derived. Training this size needs a cluster; the 1B repos above ship a recipe that runs on two 48 GB GPUs and exercises the same code paths.
• From-scratch initialisation. Released implementations only ever load checkpoints: the hash-routing table tid2eid is created as zeros (every token to expert 0) and NeMo Automodel leaves the mHC mixer tensors uninitialised. training/init_utils.py fills a balanced token-id hash and applies transformers' mixer init.
• Indexer. The lightning indexer's top-k has no gradient path; DeepSeek trains dense for the first 1T tokens (4K context) and distils the indexer when sparse attention is switched on at 64K. With index_topk 512 the selection becomes active beyond 2K tokens of context.
• Positions. The config carries V4's 1M-token YaRN setup. For a Moonlight-style 8K run, set max_position_embeddings 8192 and drop rope_scaling; V4 itself extends 4K -> 16K -> 64K -> 1M.
• Optimizer. Both Moonlight and DeepSeek-V4 train with Muon (AdamW for embeddings, head, norms and mHC static parameters); V4 uses hybrid Newton-Schulz and no QK-clip.
• transformers' V4 implementation is inference-oriented (no causal mask on appended compressed entries without a cache, per-query gathered keys); use NeMo Automodel or the reference code for training.

Design decisions (and the alternatives)

1. 16 heads, not 32: 32 heads would push activated parameters 23% above Moonlight's.
2. o_lora_rank 1024 keeps V4's exact per-group projection shape; 512 would land exactly on Moonlight's 2.24B activated non-embedding parameters at the cost of a different projection.
3. Indexer 64 x 128 and index_topk 512 are V4-Flash constants (Pro uses top-k 1024).
4. One hash layer mirrors Moonlight's single dense layer; V4-Flash/Pro use three.
5. MTP off (Moonlight precedent); enabling it requires appending a 0 to compress_ratios (V4 hub configs carry num_hidden_layers + num_nextn_predict_layers entries, the MTP block being a sliding-window layer).
6. routed_scaling_factor 2.441 comes from Moonlight's RMS-matching recipe applied to sqrt-softplus with 64 experts / top-6; V4 shipped tuned values (1.5 Flash, 2.5 Pro).

Limitations and notes

• Untrained configuration; numbers are analytic or from meta-device instantiation.
• Not affiliated with Moonshot AI or DeepSeek; the name only marks the lineage.

References

• Liu et al., Muon is Scalable for LLM Training (Moonlight), arXiv:2502.16982
• DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv:2606.19348
• DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437
• Xie et al., Manifold-Constrained Hyper-Connections (mHC), 2026; Roller et al., Hash Layers for Large Sparse Models, NeurIPS 2021