Kimi-K3-0.40B

Kimi-K3-0.40B

This is a tiny version of moonshotai/Kimi-K3 created for testing and development.

Model Details

• Base Model: moonshotai/Kimi-K3
• Architecture: kimi_linear (KimiK3ForConditionalGeneration)
• Total Parameters: 0.40B
• Activated Parameters: ~0.06B (2 of 8 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Architecture Preserved

• Mixed attention: KDA (linear/delta attention) on layers 0–2, 4–6 and MLA (full multi-latent attention) on layers 3, 7 — same 3:1 KDA:MLA ratio as the original
• Attention + MLP residuals: attn_res_block_size=4 enabled on all layers
• MoE: layers 1–7 use sparse MoE with latent expert projection; layer 0 is dense MLP
• Vision tower: included but reduced

Checkpoint Structure

Single-shard checkpoint (model.safetensors). Key prefix: language_model.model.layers.{i}.*, matching the original sharded checkpoint structure.

Usage

import sys
sys.path.insert(0, "/path/to/llm-compressor/src")

from transformers import AutoTokenizer
from llmcompressor.modeling.kimi_k3 import KimiK3ForConditionalGeneration

model = KimiK3ForConditionalGeneration.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", trust_remote_code=True
)

inputs = tokenizer("According to all known laws", return_tensors="pt").to(model.device)
output = model.language_model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Validation Output

Total parameters: 0.396B
Number of layers: 8
Layer 0 attn type: KimiDeltaAttention (KDA/linear)
Layer 3 attn type: KimiMLAAttention (MLA/full)
Layer 7 attn type: KimiMLAAttention (MLA/full)
Layer 0 has MLP: True
Layer 1 has MoE: True
Attention residuals enabled: True
Attn res block size: 4

Forward pass loss: 0.0013
Generated: The FitnessGram Pacer Test is a multistage aerobic capacity test that progressively gets more difficult as it continues. The 20 meter pacer test will begin in 30 seconds

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

1. Inspected the original moonshotai/Kimi-K3 config to identify key architecture parameters
2. Reduced layer count, hidden dimensions, and expert counts to target ~0.4B total parameters
3. Preserved the 3:1 KDA:MLA attention pattern and attention/MLP residual connections
4. Initialized all weights from scratch (normal distribution, std=0.02; norms → 1.0; biases → 0.0)
5. Fine-tuned on a toy copypasta dataset until perplexity < 3.0 (achieved in ~57 epochs)

Notes

• The model uses custom modeling code from llmcompressor.modeling.kimi_k3 — it cannot be loaded with AutoModelForCausalLM without that module on the path
• KimiK3ForConditionalGeneration does not inherit from GenerationMixin; use model.language_model.generate(...) for text generation
• The vision tower is present but untrained for vision tasks