Kimi-K2-Instruct-0905-MLX-6bit

<div align="center">

🌙 Kimi K2 Instruct - MLX 6-bit

State-of-the-Art 671B MoE Model, Optimized for Apple Silicon

Original Model | MLX Framework | More Quantizations

---

</div>

📖 What is This?

This is a premium 6-bit quantized version of Kimi K2 Instruct, optimized to run on Apple Silicon (M1/M2/M3/M4) Macs using the MLX framework. The 6-bit version is the sweet spot - offering near-original quality while being significantly more efficient than 8-bit. Perfect for production deployments!

✨ Why You'll Love It

• 🚀 Massive Context Window - Handle up to 262,144 tokens (~200,000 words!)
• 🧠 671B Parameters - One of the most capable open models available
• ⚡ Apple Silicon Native - Fully optimized for M-series chips with Metal acceleration
• 🎯 6-bit Sweet Spot - Best balance of quality and efficiency
• ⭐ Near-Original Quality - ~95% quality retention from the original model
• 🌏 Bilingual - Fluent in both English and Chinese
• 💬 Instruction-Tuned - Ready for conversations, coding, analysis, and more

🎯 Quick Start

Hardware Requirements

Kimi-K2 is a massive 671B parameter MoE model. Choose your quantization based on available unified memory:

Recommended Configurations

Performance Notes

• Inference Speed: Expect ~5-15 tokens/sec depending on quantization and hardware
• First Token Latency: 10-30 seconds for model loading
• Context Window: Full 128K context supported
• Active Parameters: Only ~37B parameters active per token (MoE architecture)

Installation

pip install mlx-lm

Your First Generation (3 lines of code!)

from mlx_lm import load, generate

model, tokenizer = load("richardyoung/Kimi-K2-Instruct-0905-MLX-6bit")
print(generate(model, tokenizer, prompt="Explain quantum entanglement simply:", max_tokens=200))

That's it! 🎉

💻 System Requirements

🎯 Note: The 6-bit version offers the best quality-to-size ratio for production use!

📚 Usage Examples

Command Line Interface

mlx_lm.generate \
  --model richardyoung/Kimi-K2-Instruct-0905-MLX-6bit \
  --prompt "Write a Python script to analyze CSV files." \
  --max-tokens 500

Chat Conversation

from mlx_lm import load, generate

model, tokenizer = load("richardyoung/Kimi-K2-Instruct-0905-MLX-6bit")

conversation = """<|im_start|>system
You are a helpful AI assistant specialized in coding and problem-solving.<|im_end|>
<|im_start|>user
Can you help me optimize this Python code?<|im_end|>
<|im_start|>assistant
"""

response = generate(model, tokenizer, prompt=conversation, max_tokens=500)
print(response)

Advanced: Streaming Output

from mlx_lm import load, generate

model, tokenizer = load("richardyoung/Kimi-K2-Instruct-0905-MLX-6bit")

for token in generate(
    model,
    tokenizer,
    prompt="Tell me about the future of AI:",
    max_tokens=500,
    stream=True
):
    print(token, end="", flush=True)

🏗️ Architecture Highlights

<details> <summary><b>Click to expand technical details</b></summary>

Model Specifications

Advanced Features

• 🎯 YaRN Rope Scaling - 64x factor for extended context
• 🗜️ KV Compression - LoRA-based (rank 512)
• ⚡ Query Compression - Q-LoRA (rank 1536)
• 🧮 MoE Routing - Top-8 expert selection with sigmoid scoring
• 🔧 FP8 Training - Pre-quantized with e4m3 precision

</details>

🎨 Other Quantization Options

Choose the right balance for your needs:

🔧 How It Was Made

This model was quantized using MLX's built-in quantization:

mlx_lm.convert \
  --hf-path moonshotai/Kimi-K2-Instruct-0905 \
  --mlx-path Kimi-K2-Instruct-0905-MLX-6bit \
  -q --q-bits 6 \
  --trust-remote-code

Result: ~6.5 bits per weight (includes metadata overhead)

⚡ Performance Tips

<details> <summary><b>Getting the best performance</b></summary>

1. Close other applications - Free up as much RAM as possible
2. Use an external SSD - If your internal drive is full
3. Monitor memory - Watch Activity Monitor during inference
4. Adjust batch size - If you get OOM errors, reduce max_tokens
5. Keep your Mac cool - Good airflow helps maintain peak performance
6. Ideal for production - Best balance of quality and performance

</details>

⚠️ Known Limitations

• 🍎 Apple Silicon Only - Won't work on Intel Macs or NVIDIA GPUs
• 💾 Storage Needs - Make sure you have 900+ GB free
• 🐏 RAM Intensive - Needs 64+ GB unified memory minimum
• 🐌 Slower on M1 - Best performance on M2 Ultra or newer
• 🌐 Bilingual Focus - Optimized for English and Chinese

💡 Why 6-bit: The sweet spot for production! Near-original quality (~95%) with significantly smaller size than 8-bit. Perfect when quality matters but you need better efficiency.

📄 License

Apache 2.0 - Same as the original model. Free for commercial use!

🙏 Acknowledgments

• Original Model: Moonshot AI for creating Kimi K2
• Framework: Apple's MLX team for the amazing framework
• Inspiration: DeepSeek V3 architecture

📚 Citation

If you use this model in your research or product, please cite:

@misc{kimi-k2-2025,
  title={Kimi K2: Advancing Long-Context Language Models},
  author={Moonshot AI},
  year={2025},
  url={https://huggingface.co/moonshotai/Kimi-K2-Instruct-0905}
}

🔗 Useful Links

• 📦 Original Model: moonshotai/Kimi-K2-Instruct-0905
• 🛠️ MLX Framework: GitHub
• 📖 MLX LM Docs: GitHub
• 💬 Discussions: Ask questions here!

---

<div align="center">

Built & maintained by Richard Young · DeepNeuro If you find this useful, please ⭐ star the repo and share with others!

Created: October 2025 | Format: MLX 6-bit

</div>