Qwen3-Coder-30B-A3B-Instruct-RTPurbo

Qwen3-Coder-30B-A3B-Instruct-RTPurbo

Model Overview

• Model Optimizations:
  • Sliding Window Attention: 85%
  • Full Attention: 15%
• Version: 1.0

<img src="./headwise.png" alt="screenshot">

RTPurbo uses hybrid HeadWise Attention to compress the Qwen3Coder model. Specifically, it divides attention into two parts according to attention type:

1. Retrieval Heads: These heads perform Full Attention over the entire sequence (or a large chunk), allowing them to capture rich, long-range dependencies and act as a powerful information retrieval component.
2. non Retrieval Heads: These heads use Sink SWA Attention, processing tokens in a sliding-window or fixed-cache manner. They are highly efficient and ideal for handling very long sequences while maintaining local context.

The following code can be used for inference. HeadWise will be triggered in scenarios where SeqLen > 16,384.

from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

model_name = "RTP-LLM/Qwen3-Coder-30B-A3B-Instruct-RTPurbo"

tokenizer = AutoTokenizer.from_pretrained(model_name)
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config, 
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Write a quick sort algorithm."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=128
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

content = tokenizer.decode(output_ids, skip_special_tokens=True)

print("content:", content)

Evaluation

This model was evaluated in the lm_eval benchmark using Qwen3-Coder-30B-A3B-Instruct as evaluator.

<table style="border-collapse:collapse; border-top:2px solid #000; border-bottom:2px solid #000;"> <thead> <tr style="border-bottom:2px solid #000;"> <th align="center" style="padding:8px 14px;">Longbench</th> <th align="center" style="padding:8px 14px;">lcc</th> <th align="center" style="padding:8px 14px;">repo-p</th> <th align="center" style="padding:8px 14px;">samsum</th> <th align="center" style="padding:8px 14px;">trec</th> <th align="center" style="padding:8px 14px;">lsht</th> <th align="center" style="padding:8px 14px;">2wikim</th> <th align="center" style="padding:8px 14px;">hotpot</th> <th align="center" style="padding:8px 14px;">multi-en</th> <th align="center" style="padding:8px 14px;">multi-zh</th> <th align="center" style="padding:8px 14px;">musique</th> <th align="center" style="padding:8px 14px;">qasper</th> <th align="center" style="padding:8px 14px;">vcsum</th> <th align="center" style="padding:8px 14px;">qmsum</th> <th align="center" style="padding:8px 14px;">PR-en</th> <th align="center" style="padding:8px 14px;">PR-zh</th> <th align="center" style="padding:8px 14px;">Avg. (%)</th> </tr> <tr style="border-bottom:2px solid #000;"> <th align="center" colspan="17" style="padding:10px 14px;">Qwen3-Coder-30B-A3B</th> </tr> </thead> <tbody> <tr style="border-bottom:2px solid #000;"> <td align="center" style="padding:8px 14px;"><b>Full Attn</b></td> <td align="center" style="padding:8px 14px;">34.34</td> <td align="center" style="padding:8px 14px;">27.14</td> <td align="center" style="padding:8px 14px;">45.80</td> <td align="center" style="padding:8px 14px;">81.00</td> <td align="center" style="padding:8px 14px;">47.50</td> <td align="center" style="padding:8px 14px;">42.08</td> <td align="center" style="padding:8px 14px;">57.64</td> <td align="center" style="padding:8px 14px;">52.89</td> <td align="center" style="padding:8px 14px;">65.99</td> <td align="center" style="padding:8px 14px;">38.30</td> <td align="center" style="padding:8px 14px;">39.25</td> <td align="center" style="padding:8px 14px;">13.55</td> <td align="center" style="padding:8px 14px;">23.77</td> <td align="center" style="padding:8px 14px;">99.00</td> <td align="center" style="padding:8px 14px;">99.75</td> <td align="center" style="padding:8px 14px;">51.20</td> </tr> <tr style="border-bottom:2px solid #000;"> <td align="center" style="padding:8px 14px;"><b>RTPurbo</b></td> <td align="center" style="padding:8px 14px;">35.96</td> <td align="center" style="padding:8px 14px;">35.21</td> <td align="center" style="padding:8px 14px;">46.49</td> <td align="center" style="padding:8px 14px;">81.00</td> <td align="center" style="padding:8px 14px;">49.00</td> <td align="center" style="padding:8px 14px;">47.39</td> <td align="center" style="padding:8px 14px;">55.44</td> <td align="center" style="padding:8px 14px;">52.93</td> <td align="center" style="padding:8px 14px;">65.23</td> <td align="center" style="padding:8px 14px;">35.58</td> <td align="center" style="padding:8px 14px;">39.78</td> <td align="center" style="padding:8px 14px;">13.80</td> <td align="center" style="padding:8px 14px;">23.68</td> <td align="center" style="padding:8px 14px;">99.00</td> <td align="center" style="padding:8px 14px;">99.75</td> <td align="center" style="padding:8px 14px;">52.02</td> </tr> </tbody> </table>

Media Coverage

Our work has been featured by Minds in AI (机器之心). Please visit it for more details.