Kimi-Linear-48B-A3B-Instruct-AWQ-4bit

<div align="left"> <a href="https://cyan.kiwi" target="_blank"><img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-cyankiwi.png" width="800" alt="cyankiwi, the LLM inference agent for open models"></a> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Website</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://cyan.kiwi/" target="_blank">cyan.kiwi</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Models</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/cyankiwi" target="_blank">Hugging Face</a> · <a href="https://www.modelscope.ai/organization/cyankiwi" target="_blank">ModelScope</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Connect</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://x.com/cyankiwi_ai" target="_blank">X</a> · <a href="https://www.reddit.com/r/cyankiwi/" target="_blank">Reddit</a> · <a href="mailto:[email protected]">Email</a></td> </tr> </table> </div> <br> <div align="left"> <img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-awq.png" width="800" alt="cyankiwi Activation-aware Weight Quantization"> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Version</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://github.com/vllm-project/llm-compressor" target="_blank">llm-compressor</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Calibration</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset" target="_blank">nvidia/Llama-Nemotron-Post-Training-Dataset</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Model Size</b></td> <td align="left" style="border:none; padding:4px 0;">30.53 GB</td> </tr> </table> </div> <br> <p align="left"> <a href="https://cyan.kiwi/workspace?prompt=Deploy%20cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit" target="_blank"><img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-button-deploy.png" width="333" alt="Deploy on cyan.kiwi private endpoints"></a> </p>

---

<div align="center"> <h3 align="center"> <b> <hr style="width: 700px; border: none; border-top: 2px solid #333; margin: 16px auto;" /> <img src="figures/logo.png" height="16" width="16" style="display: inline-block; vertical-align: middle; margin: 2px;"> Kimi Linear: An Expressive, Efficient Attention Architecture <hr style="width: 700px; border: none; border-top: 2px solid #333; margin: 16px auto;" /> </b> </h3> </div> <div align="center">   <a href="https://huggingface.co/papers/2510.26692" style="margin: 0 8px;"> <img src="figures/arxiv.png" height="16" width="16" style="display: inline-block; vertical-align: middle; margin: 2px;"><b> Paper</b> </a>   <a href="https://github.com/MoonshotAI/Kimi-Linear" style="margin: 0 8px;"> <img src="figures/github.png" height="16" width="16" style="display: inline-block; vertical-align: middle; margin: 2px;"><b> Code</b> </a>   <a href="https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct" style="margin: 0 8px;"> <img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="16" width="16" style="display: inline-block; vertical-align: middle; margin: 2px;"><b> Model</b> </a> </div> <div align="center"> <img width="90%" src="figures/perf_speed.png"> <p><em><b>(a)</b> On MMLU-Pro (4k context length), Kimi Linear achieves 51.0 performance with similar speed as full attention. On RULER (128k context length), it shows Pareto-optimal performance (84.3) and 3.98x speedup. <b>(b)</b> Kimi Linear achieves 6.3x faster TPOT compared to MLA, offering significant speedups at long sequence lengths (1M tokens).</em></p> </div>

Overview

Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.

Kimi Linear achieves superior performance and hardware efficiency, especially for long-context tasks. It reduces the need for large KV caches by up to 75% and boosts decoding throughput by up to $6\times$ for contexts as long as 1M tokens.

We open-source the KDA kernel in FLA, and release two versions model checkpoints trained with 5.7T tokens.

Key Features

• Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
• Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
• Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
• High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).

<div align="center"> <img width="60%" src="figures/arch.png"> </div>

Usage

Inference with Hugging Face Transformers

To use the Kimi Linear model, we recommend the following environment:

• python >= 3.10
• torch >= 2.6
• fla-core >= 0.4.0

pip install -U fla-core

Example Code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "moonshotai/Kimi-Linear-48B-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

messages = [
    {"role": "system", "content": "You are a helpful assistant provided by Moonshot-AI."},
    {"role": "user", "content": "Is 123 a prime?"}
]
input_ids = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)
generated_ids = model.generate(inputs=input_ids, max_new_tokens=500)
response = tokenizer.batch_decode(generated_ids)[0]
print(response)

Deployment

For deployment, you can use the latest vllm to create an OpenAI-compatible API endpoint.

vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 1048576 \
  --trust-remote-code

Citation

If you found our work useful, please cite

@misc{team2025kimi,
    title         = {Kimi Linear: An Expressive, Efficient Attention Architecture},
    author        = {Zhang, Yu  and Lin, Zongyu  and Yao, Xingcheng  and Hu, Jiaxi  and Meng, Fanqing  and Liu, Chengyin  and Men, Xin  and Yang, Songlin  and Li, Zhiyuan  and Li, Wentao  and Lu, Enzhe  and Liu, Weizhou  and Chen, Yanru  and Xu, Weixin  and Yu, Longhui  and Wang, Yejie  and Fan, Yu  and Zhong, Longguang  and Yuan, Enming  and Zhang, Dehao  and Zhang, Yizhi  and T. Liu, Y.  and Wang, Haiming  and Fang, Shengjun  and He, Weiran  and Liu, Shaowei  and Li, Yiwei  and Su, Jianlin  and Qiu, Jiezhong  and Pang, Bo  and Yan, Junjie  and Jiang, Zhejun  and Huang, Weixiao  and Yin, Bohong  and You, Jiacheng  and Wei, Chu  and Wang, Zhengtao  and Hong, Chao  and Chen, Yutian  and Chen, Guanduo  and Wang, Yucheng  and Zheng, Huabin  and Wang, Feng  and Liu, Yibo  and Dong, Mengnan  and Zhang, Zheng  and Pan, Siyuan  and Wu, Wenhao  and Wu, Yuhao  and Guan, Longyu  and Tao, Jiawen  and Fu, Guohong  and Xu, Xinran  and Wang, Yuzhi  and Lai, Guokun  and Wu, Yuxin  and Zhou, Xinyu  and Yang, Zhilin  and Du, Yulun},
    year          = {2025},
    eprint        = {2510.26692},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CL}
}