Mini-K3-1H-attn-4mla-rope-v2

Mini-K3-1H-attn-4mla-rope

This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding.

Architecture

• Decoder layers: 13 (0 KDA + 13 Gated MLA)
• KDA layer indices: []
• Gated MLA layer indices: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
• Hidden width / attention heads / KDA head width: 1024 / 12 / 128
• KDA causal depthwise-convolution kernel: not applicable (no KDA layer)
• KDA decay groups per head: not applicable (no KDA layer)
• MLA positional mode: RoPE; output gate: True
• Dense layers before MoE: 1
• Routed/shared experts and top-k: 64 / 2 / 4
• Routed expert hidden width: 512
• Attention Residual block size: 4
• Logical parameters: 981,692,224
• Activated parameters per token: 318,467,904
• Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839

Checkpoint

• Revision/tag: checkpoint-tokens-016000000000-final
• Valid next-token targets consumed: 16,000,000,000
• Optimizer steps completed: 24,415
• Sequence length: 8,192
• Parameter dtype: BF16, with KDA decay, convolution, normalization and router control state retained in FP32 where defined by the implementation
• Optimizer state: deliberately not published

Every numbered checkpoint is an immutable Git tag. main points to the most recent uploaded checkpoint. The final tag is exactly checkpoint-tokens-016000000000-final and is created only after exactly 16,000,000,000 valid loss targets have been processed.

Initialization and comparability

All 20 runs use canonical name-and-shape keyed initialization with base seed 20260914. Parameters shared by two architectures, with the same semantic name and shape, therefore begin byte-identically. Variant-only or shape-changing parameters receive their own deterministic streams. All runs consume the same immutable, append-only mixture schedule in the same order. Packed documents are hard-isolated: MLA uses a document-blocked causal mask, while KDA recurrent state and Q/K/V short-convolution history reset at every segment boundary.

Training recipe

Training uses K3-style per-head Muon for head-expanded Q/K/V matrices, Muon for other matrix parameters, AdamW fallback for vectors and embeddings, weight decay 0.1, per-head QK-Clip, cosine decay, 1% linear warmup, and online 1,000-bin histogram Quantile Balancing. The router selects experts with biased scores and combines them with renormalized unbiased sigmoid scores. No post-training was performed.

The exact frozen source revisions, token quotas, schedule hashes, optimizer configuration, hardware benchmark choice, and validation split hashes are in the JSON manifests included in this repository. Source datasets retain their own licenses and terms; this repository does not redistribute their text.

Files and loading

model.safetensors uses the standalone layout defined by modeling_mini_k3.py and configuration_mini_k3.py. The bundled code and config.json fully define the architecture without relying on the original training checkout. ARCHITECTURE_PACKAGE_README.md, ARCHITECTURE.md, and VARIANT.md (when present) describe the standalone package and the controlled change. See initialize_model.py and smoke_test.py for local use.

Evaluation

This is an intermediate pretraining research checkpoint. It has not yet been evaluated on downstream tasks. Training-time fixed-development NLL and perplexity are logged in W&B and the run's JSONL metrics. Planned downstream and architecture diagnostics are documented in the experiment repository.

Limitations

The model is a small research proxy, is pretrained only, and should not be treated as an instruction-following assistant. Outputs may be inaccurate, biased, unsafe, or repetitive. Architecture rankings at this scale and at an 8K training length need confirmation before extrapolation to full Kimi-K3.