Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled

Qwen3-Coder-Next — Opus 4.6 Reasoning Distilled (Full Fine-Tune)

Full parameter fine-tune of Qwen/Qwen3-Coder-Next (~80B total / ~3B active, MoE) with Claude Opus 4.6 reasoning distillation. Trained on 8x NVIDIA H100 80GB SXM with DeepSpeed ZeRO-3.

Model Details

Benchmark Results

Evaluated by Claude Opus 4.6 across 26 tests in 6 categories, scored 1-10 on correctness, completeness, clarity, and adherence to instructions.

Detailed Test Results

Coding (6 tests)

Bug Detection (5 tests)

Probability (5 tests)

Tool Calling (5 tests) — Largest improvement

Logic (2 tests)

Instruction Following (3 tests)

Key Findings

• Tool Calling: Largest improvement (+3.8). Base model outputs only the first tool call and stops. Opus Distilled plans full multi-step tool chains with reasoning between steps.
• Bug Detection: Opus Distilled provides more structured analysis with severity tables, timeline diagrams, and catches more edge cases (+0.6).
• Coding: Opus Distilled favors class-based architectures with better design patterns. Caught a SQL bug (window function in WHERE clause) that Base missed.
• Probability: Base is more concise and made fewer computation errors. Opus Distilled made an error on a Markov Chain steady-state calculation.
• Logic: Base makes better progress within token budgets — Opus Distilled spends more tokens on preamble.
• Instruction Following: Base adheres more strictly to output format constraints (e.g., "output ONLY valid JSON").

Verdict

Opus-Distilled wins overall driven by massively better tool calling and slightly better bug detection and coding. Base wins on math/probability (fewer errors), logic (better token efficiency), and instruction following (better constraint adherence). For coding assistant use cases where tool calling matters, Opus-Distilled is clearly superior.

Performance

Both models run at comparable speeds on RTX PRO 6000 Blackwell (96GB):

Training Details

Hardware & Infrastructure

• GPUs: 8x NVIDIA H100 80GB SXM with NVLink
• System RAM: 2 TB DDR5
• Distribution: DeepSpeed ZeRO-3 (parameters sharded across all 8 GPUs)
• Optimizer Offload: AdamW optimizer states offloaded to CPU RAM (~700GB)
• Platform: RunPod

Hyperparameters

Training Progression

Datasets

3,204 examples after quality filtering (required <think> tags and >200 characters of assistant content):

Data Format

Each training example follows this structure:

<|im_start|>user
{problem}<|im_end|>
<|im_start|>assistant
<think>
{chain-of-thought reasoning}
</think>

{final answer}<|im_end|>

Quality Filter

Examples were filtered to require:

1. At least one assistant message containing <think> tags
2. Assistant content longer than 200 characters

This removed low-quality or non-reasoning examples from the combined dataset.

Reasoning Format

The model produces reasoning inside <think>...</think> tags:

<think>
Let me analyze this step by step...
1. First consideration
2. Second consideration
3. Conclusion
</think>

Here is the final answer based on my analysis.

When serving with llama.cpp, use --reasoning-format deepseek with a thinking-aware chat template to separate reasoning from visible output.

GGUF Quantizations

See samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled-GGUF for quantized GGUF versions (Q4_K_M, Q6_K, Q8_0, BF16).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "samuelcardillo/Qwen3-Coder-Next-Opus-4.6-Reasoning-Distilled",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Implement a thread-safe LRU cache in Python"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))