Qwen3.5-4B-Claude-Distill-v2

Qwen3.5-4B-Claude-Distill-v2

Full supervised fine-tuning of Qwen3.5-4B on Claude-Distills data. This model preserves all capabilities of the base Qwen3.5-4B while acquiring enhanced instruction-following and reasoning skills from Claude-generated training data.

This is the v2 of Qwen3.5-4B-Claude-distill, trained with improved data and full-parameter SFT.

Model Description

• Base model: Qwen/Qwen3.5-4B
• Training data: Kassadin88/Claude-Distills
• Training method: Full SFT (all parameters)
• Training framework: ms-swift
• Training hardware: 1 node × 8× H100 80GB

Training Data

The model was trained on 125,175 Claude-distilled conversation pairs (with 6,588 held out for validation).

Data Distribution

Categories were classified using an automated LLM-based classifier on a representative sample:

The dataset is dominated by math and code (~80%), with significant coverage of science, knowledge, and general conversation.

Data Format

Each sample is a multi-turn conversation in standard chat format:

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Benchmark Results

Base model (Qwen3.5-4B) performance on language benchmarks:

For full benchmark details including vision-language results, see the base model card.

Quickstart

Note: This model operates in thinking mode by default, generating thinking content signified by -thinking\n... before producing the final response. To disable thinking, refer to Non-Thinking Mode.

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Kassadin88/Qwen3.5-4B-Claude-Distill-v2"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Write a Python function to find the longest common subsequence of two strings."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=32768)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

OpenAI-Compatible API (via vLLM / SGLang)

# vLLM
vllm serve Kassadin88/Qwen3.5-4B-Claude-Distill-v2 --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3

# SGLang
python -m sglang.launch_server --model-path Kassadin88/Qwen3.5-4B-Claude-Distill-v2 --port 8000 --tp-size 1 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

messages = [
    {"role": "user", "content": "Solve: find all integer solutions to x^2 + y^2 = 25"}
]

chat_response = client.chat.completions.create(
    model="Kassadin88/Qwen3.5-4B-Claude-Distill-v2",
    messages=messages,
    max_tokens=32768,
    temperature=1.0,
    top_p=0.95,
    presence_penalty=1.5,
    extra_body={"top_k": 20},
)
print(chat_response.choices[0].message.content)

Non-Thinking Mode

To get direct responses without chain-of-thought reasoning:

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

Or via API:

chat_response = client.chat.completions.create(
    model="Kassadin88/Qwen3.5-4B-Claude-Distill-v2",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

Recommended Sampling Parameters

Training Details

Best Practices

1. Adequate Output Length: Use at least 32,768 tokens for output. For complex math/coding problems, set max output length to 81,920 tokens.
2. No Thinking Content in History: In multi-turn conversations, historical model output should only include the final response, not thinking content. The provided chat template handles this automatically.
3. Standardize Output Format: For math problems, include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.

Limitations

• The model is primarily trained on English and Chinese data; performance on other languages may be limited.
• Math and code dominate the training distribution (~80%), which may affect performance on other domains.
• As a distilled model, it may inherit biases present in the Claude-generated training data.
• The model has not been aligned with RLHF or similar safety techniques.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{qwen3.5,
    title  = {{Qwen3.5}: Towards Native Multimodal Agents},
    author = {{Qwen Team}},
    month  = {February},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.5}
}

@misc{qwen35-4b-claude-distill-v2,
    title={Qwen3.5-4B-Claude-Distill-v2: Full SFT on Claude Distill Data},
    author={Kassadin88},
    year={2026},
    url={https://huggingface.co/Kassadin88/Qwen3.5-4B-Claude-Distill-v2}
}