Qwen3.5-35B-A3B-Kazakh

Qwen3.5-35B-A3B-Kazakh

Qwen3.5-35B-A3B-Kazakh is a Kazakh adaptation of Qwen/Qwen3.5-35B-A3B. It extends the tokenizer with an additional Kazakh vocabulary for more efficient tokenization, is continued-pretrained on Kazakh text, and is then re-aligned to the official instruct model. This makes the model more efficient on Kazakh text and adds knowledge, while preserving the capabilities of the base model.

Methodology

This model is produced in two stages on top of our Kazakh continued-pretraining model issai/Qwen3.5-35B-A3B-Base-Kazakh, the official Qwen/Qwen3.5-35B-A3B-Base continued-pretrained on Kazakh text with a +16K Kazakh-token vocabulary extension. No additional post-training (SFT, RLHF, or RLVR) is used.

1. Chat-vector merge, transferring instruct alignment

We transfer the instruction-following / chat alignment of the official post-trained model using the chat vector method (Huang et al., 2024). Let

• θ_inst: weights of the official post-trained Qwen/Qwen3.5-35B-A3B
• θ_base: weights of the official base Qwen/Qwen3.5-35B-A3B-Base
• θ_cpt: weights of our Kazakh base model issai/Qwen3.5-35B-A3B-Base-Kazakh

The chat vector is the task vector of post-training,

$$\tau_\text{chat} = \theta_\text{inst} - \theta_\text{base},$$

and it is added to our Kazakh continued-pretraining model:

$$\theta_\text{merged} = \theta_\text{cpt} + \tau_\text{chat} = \theta_\text{cpt} + (\theta_\text{inst} - \theta_\text{base}),$$

applied per parameter tensor, by name. Because our tokenizer is vocabulary-extended, the input-embedding and output (lm_head) matrices are merged row-wise: the shared original-vocabulary rows receive the chat vector, while the 16K new Kazakh-token rows are kept from θ_cpt (the official models have no counterpart for them). embed_tokens and lm_head are untied and are merged independently. This model is a Mixture-of-Experts; the official checkpoints store the experts fused (experts.gate_up_proj, experts.down_proj), so our per-expert weights are fused to that layout before the merge, and the router / shared-expert weights are merged by name like any other tensor.

Special-token pinning. The rows of the format-control special tokens <think>, </think>, <|im_start|>, <|im_end|> and <|endoftext|> (in both the embedding and lm_head) are set to the official post-trained model's exact values instead of the merged values. These few rows drive the chat template's control flow (emitting </think> to close the reasoning block and <|im_end|> to end the turn). The raw chat vector otherwise detunes them and breaks this template logic, causing the model to loop and never terminate. Pinning them restores reliable termination while preserving the Kazakh knowledge acquired during continued pretraining.

2. Vision re-integration, restoring multimodality

Because our continued pretraining was performed on text only, with the vision branch removed beforehand, the chat-vector merge operates on the language model and produces a text-only model. To restore image and video understanding, the vision encoder and multimodal projector from the official Qwen/Qwen3.5-35B-A3B are re-attached to this Kazakh-adapted, instruct-aligned language backbone, and the architecture is set back to Qwen3_5MoeForConditionalGeneration. The result is a multimodal model that largely preserves the official model's vision capabilities while its language model handles Kazakh (and English) with instruct-style alignment and correct <think>...</think> reasoning.

Evaluation

We evaluate Qwen3.5-35B-A3B-Kazakh against its base model Qwen3.5-35B-A3B on Kazakh and English benchmarks under identical settings (thinking mode enabled):

• temperature = 1.0
• top_p = 0.95
• top_k = 20
• presence_penalty = 1.5
• max_tokens = 81920

Scores are accuracy (%).

<div style="border-left:4px solid #e57373; background:#fdecea; padding:10px 14px; border-radius:6px;"> ⚠️ <b>Note on reasoning-heavy benchmarks.</b> The custom model tends to <i>reason more briefly</i> (shorter chains-of-thought) than the original Qwen model, and this is most visible on the hardest problems. The largest regressions therefore appear on the hardest reasoning benchmarks in our evaluation set, such as PolyMath (5.8 pts lower) and GPQA-Diamond (4.5 pts lower), while on easier benchmarks the effect is minimal and the model preserves the original Qwen model's capabilities. <br><br> ⚠️ <b>Note on vision benchmarks.</b> The merged model was not explicitly trained on vision tasks, which accounts for some degradation on the vision benchmarks. These results can be further improved with dedicated vision-language finetuning. </div>

Kazakh

The Kazakh benchmarks (most of which are available in the ISSAI Qolda suite: language, vision) use Kazakh questions and answer choices. KazMMLU and KazCulture are native Kazakh benchmarks; the rest are Kazakh adaptations of the standard benchmarks.

<table> <tbody> <tr> <td valign="top"> <b>Text</b> <table style="font-size: 1rem"> <thead><tr><th style="background-color:#EDE7F6; color:#111111" align="left">Benchmark</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B-Kazakh</th></tr></thead> <tbody> <tr><td>KazMMLU</td><td align="center">82.6</td><td align="center"><b>84.0</b></td></tr> <tr style="background-color:#f6f8fa"><td>KazCulture</td><td align="center">60.3</td><td align="center"><b>68.7</b></td></tr> <tr><td>MMLU</td><td align="center"><b>87.2</b></td><td align="center">84.4</td></tr> <tr style="background-color:#f6f8fa"><td>MMLU-Pro</td><td align="center"><b>79.3</b></td><td align="center">76.1</td></tr> <tr><td>GPQA</td><td align="center"><b>70.9</b></td><td align="center">69.8</td></tr> <tr style="background-color:#f6f8fa"><td>ARC-Easy</td><td align="center"><b>98.1</b></td><td align="center">97.5</td></tr> <tr><td>ARC-Challenge</td><td align="center"><b>96.3</b></td><td align="center">95.5</td></tr> <tr style="background-color:#f6f8fa"><td>GSM8K</td><td align="center"><b>92.7</b></td><td align="center">89.3</td></tr> <tr><td>Belebele</td><td align="center"><b>89.4</b></td><td align="center">88.9</td></tr> <tr style="background-color:#f6f8fa"><td>MMLU-Redux</td><td align="center"><b>88.4</b></td><td align="center">85.9</td></tr> </tbody> </table> </td> <td valign="top"> <b>Vision</b> <table style="font-size: 1rem"> <thead><tr><th style="background-color:#EDE7F6; color:#111111" align="left">Benchmark</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B-Kazakh</th></tr></thead> <tbody> <tr><td>AI2D</td><td align="center"><b>85.7</b></td><td align="center">85.2</td></tr> <tr style="background-color:#f6f8fa"><td>MMStar</td><td align="center"><b>76.0</b></td><td align="center">75.2</td></tr> <tr><td>RealWorldQA</td><td align="center">77.5</td><td align="center"><b>78.6</b></td></tr> <tr style="background-color:#f6f8fa"><td>MathVista</td><td align="center"><b>83.3</b></td><td align="center">82.8</td></tr> <tr><td>OCRBench</td><td align="center"><b>42.4</b></td><td align="center">34.0</td></tr> <tr style="background-color:#f6f8fa"><td>MathVision</td><td align="center"><b>81.4</b></td><td align="center">78.8</td></tr> <tr><td>MMBench</td><td align="center">91.5</td><td align="center"><b>91.7</b></td></tr> </tbody> </table> </td> </tr> </tbody> </table>

English

<table> <tbody> <tr> <td valign="top"> <b>Text</b> <table style="font-size: 1rem"> <thead><tr><th style="background-color:#EDE7F6; color:#111111" align="left">Benchmark</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B-Kazakh</th></tr></thead> <tbody> <tr><td>MMLU</td><td align="center"><b>91.2</b></td><td align="center">89.3</td></tr> <tr style="background-color:#f6f8fa"><td>MMLU-Pro</td><td align="center"><b>85.5</b></td><td align="center">82.7</td></tr> <tr><td>GPQA-Diamond</td><td align="center"><b>86.9</b></td><td align="center">82.3</td></tr> <tr style="background-color:#f6f8fa"><td>GPQA-Main</td><td align="center"><b>79.7</b></td><td align="center">76.6</td></tr> <tr><td>ARC-Easy</td><td align="center"><b>99.3</b></td><td align="center">97.6</td></tr> <tr style="background-color:#f6f8fa"><td>ARC-Challenge</td><td align="center"><b>98.0</b></td><td align="center">95.7</td></tr> <tr><td>GSM8K</td><td align="center"><b>97.0</b></td><td align="center">94.3</td></tr> <tr style="background-color:#f6f8fa"><td>Belebele</td><td align="center">97.4</td><td align="center"><b>97.7</b></td></tr> <tr><td>MMLU-Redux</td><td align="center"><b>92.1</b></td><td align="center">89.9</td></tr> </tbody> </table> </td> <td valign="top"> <b>Vision</b> <table style="font-size: 1rem"> <thead><tr><th style="background-color:#EDE7F6; color:#111111" align="left">Benchmark</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B</th><th style="background-color:#EDE7F6; color:#111111" align="center">Qwen3.5‑35B‑A3B-Kazakh</th></tr></thead> <tbody> <tr><td>AI2D</td><td align="center"><b>90.8</b></td><td align="center">90.6</td></tr> <tr style="background-color:#f6f8fa"><td>MMStar</td><td align="center">78.8</td><td align="center"><b>79.3</b></td></tr> <tr><td>RealWorldQA</td><td align="center"><b>81.3</b></td><td align="center"><b>81.3</b></td></tr> <tr style="background-color:#f6f8fa"><td>MathVista</td><td align="center">87.0</td><td align="center"><b>87.8</b></td></tr> <tr><td>OCRBench</td><td align="center"><b>87.6</b></td><td align="center">86.5</td></tr> <tr style="background-color:#f6f8fa"><td>MathVision</td><td align="center"><b>83.3</b></td><td align="center">81.3</td></tr> <tr><td>MMBench</td><td align="center"><b>91.2</b></td><td align="center">91.0</td></tr> </tbody> </table> </td> </tr> </tbody> </table>

Model usage

The following package versions are used:

• transformers>=5.12.1
• a recent torch build (matching your CUDA)
• vllm>=0.24.0 (for serving)
• optional: flash-linear-attention and causal-conv1d for faster linear-attention inference

The model reasons by default, emitting a chain-of-thought inside <think> ... </think>.

Transformers

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "issai/Qwen3.5-35B-A3B-Kazakh"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [
    {"role": "user", "content": [{"type": "text", "text": "Балқаш көлінің қандай ерекшелігі бар?"}]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Image input:

messages = [
    {"role": "user", "content": [
        {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
        {"type": "text", "text": "Суретте не бейнеленген?"},
    ]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

To disable thinking, pass enable_thinking=False to apply_chat_template.

vLLM

Serve an OpenAI-compatible endpoint:

vllm serve issai/Qwen3.5-35B-A3B-Kazakh \
    --served-model-name qwen \
    --reasoning-parser qwen3 \
    --max-model-len 90000 \
    --enable-expert-parallel \
    --max-num-seqs 64

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": """Төмендегі экзопланеталардың қайсысында тығыздығы ең жоғары?
a) Жермен бірдей құрамы бар, бірақ Жерден 5 есе ауыр планета.
b) Жермен бірдей құрамды, бірақ Жерден екі есе жеңіл планета.
c) Массасы Жерден 2 есе үлкен және тығыздығы шамамен 5.5 г/см^3 болатын планета.
d) Жердің массасы мен радиусымен тең планета."""}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.reasoning)          # chain-of-thought
print(resp.choices[0].message.content)            # final answer

Image input:

import base64

image = base64.b64encode(open("image.png", "rb").read()).decode()

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
        {"type": "text", "text": "Суретте қандай фигуралар бар?"},
    ]}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.content)

To disable thinking, add "chat_template_kwargs": {"enable_thinking": false} to extra_body.

License

This model is released under the Apache 2.0 license, inherited from the original Qwen3.5 model.