Qwen3-0.6B-SciGen-SLERP

Qwen3-0.6B-SciGen-SLERP

A SLERP merge of two domain-specialized full-SFT Qwen3-0.6B experts.

596M parameters (same as base) · bfloat16 · Architecture: Qwen3ForCausalLM

TL;DR

Two Qwen3-0.6B models were full-fine-tuned on different domains (physics + general chat), then merged via SLERP at t=0.3. The result averages +5.9% over base on the HF Open LLM Leaderboard v1 task suite (ARC, HellaSwag, BoolQ, PIQA, Winogrande). On the harder Leaderboard v2 task suite (BBH, MMLU-Pro, MuSR, IFEval, MATH Hard) results are mixed: SLERP wins IFEval and MuSR, but loses BBH, MMLU-Pro, and MATH Hard. The merge also suppresses Qwen3's thinking behavior, making it faster and more direct but less suitable for tasks that benefit from explicit reasoning. See the v2 interpretation section for important caveats on the IFEval gain.

Intended Use: Research and experimentation only.

Training

Expert 1 — Science

Expert 2 — General

Merge

Tool: mergekit v0.1.4

base_model: expert_science/checkpoint-600
dtype: bfloat16
merge_method: slerp
modules:
  default:
    slices:
    - sources:
      - layer_range: [0, 28]
        model: expert_science/checkpoint-600
      - layer_range: [0, 28]
        model: expert_general/checkpoint-750
parameters:
  t:
  - value: 0.3

t=0.3 means 70% science expert, 30% general expert, applied uniformly across all 28 layers.

Note on t selection: Two SLERP t values were tested manually (0.3 and 0.5). t=0.3 scored higher on the v1 task suite (0.5819 vs 0.5738 average). A per-layer CMA-ES search over t values was also attempted but did not beat the global t=0.3 baseline. No systematic sweep over additional t values was performed.

Evaluation settings (all tasks)

Evaluation — HF Open LLM Leaderboard v1 task suite

(The "v1" / "v2" naming refers to two different HuggingFace benchmark suites with different task sets — not different versions of our evaluation.)

Evaluated with EleutherAI lm-evaluation-harness v0.4.11. Full datasets, no sample limit, dtype=bfloat16. Each task uses HF Leaderboard v1 standard few-shot settings.

All numbers were verified with an independent rerun matching the originals to within standard error. The BoolQ +10% gain is the largest single contributor and traces back to the science expert's factual Q&A training format.

Evaluation — HF Open LLM Leaderboard v2 task suite

Evaluated with lm-evaluation-harness v0.4.11. Note: --apply_chat_template was NOT used because Qwen3-0.6B exhibits degenerate behavior (collapsing to a single answer choice on >94% of MMLU-Pro questions) when chat-templated for loglikelihood multiple-choice tasks. This was verified by inspecting per-sample answer distributions. Generation was run with max_gen_toks=2048, thinking enabled (default), greedy decoding.

Loglikelihood tasks (multiple choice)

Generation tasks

v2 interpretation

SLERP wins IFEval, MuSR, and loses BBH, MMLU-Pro, MATH Hard at the greedy max_gen_toks=2048 setting these benchmarks were run at.

Important caveat on the IFEval gain: these v2 numbers were all collected with greedy decoding and a 2048-token generation budget — the same settings that the qualitative comparison shows cause base Qwen3-0.6B to frequently run out of tokens mid-<think> and never produce a final answer. Because SLERP suppresses thinking, it almost always completes its response within the budget. The IFEval gain is therefore at least partly an artifact of "completes within token budget" rather than "follows instructions more precisely." When given proper sampling settings (see qualitative section, prompt 1), base actually follows strict format rules better than SLERP on the example tested.

The MATH Hard regression is more straightforward: math benefits from explicit reasoning, and neither expert was trained on math. SLERP's thinking suppression makes the regression worse. Don't use this merge for math.

Net characterization: SLERP is a faster, more direct variant of Qwen3-0.6B. Its benchmark advantages partly come from being more reliable about producing some answer within a tight token budget, not from being a stronger reasoner.

Qualitative comparison

5 prompts run on both models using Qwen3's recommended inference settings: temperature=0.6, top_p=0.95, top_k=20, do_sample=True, max_new_tokens=8192. Chat template applied with thinking enabled. Designed to probe expected strengths (instruction following, science, helpfulness) and known weaknesses (math). Full outputs at results/qualitative_comparison_proper.json.

(Note: this differs from the v1/v2 benchmark settings, which use greedy decoding for reproducibility. These qualitative settings reflect what a real user calling the model in chat mode would see.)

Prompt 1 — Strict instruction following

Write exactly 3 bullet points explaining why the sky is blue. Use no more than 15 words per bullet. End your response with "---END---"

Base (305 tokens): Used thinking to plan, counted words, produced 3 properly-formatted bullets each under 15 words, ended with ---END---. Followed all instructions.

SLERP (135 tokens): Skipped thinking, used numbered points instead of bullets, exceeded the 15-word limit, omitted the ---END--- marker.

Prompt 2 — Science explanation

Explain the difference between conduction, convection, and radiation in heat transfer.

Base (839 tokens): Used thinking, then produced a complete structured explanation with mechanism, medium, direction, and example for each — plus a summary.

SLERP (466 tokens): Skipped thinking, gave decent definitions but truncated mid-radiation explanation.

Prompt 3 — Helpful explanation

I'm learning Python and struggling with list comprehensions. Can you explain them simply with one example?

Base (591 tokens): Complete answer with code example, bullet-pointed explanation, and listed use cases.

SLERP (170 tokens): Working code example with brief explanation. Concise but less detailed.

Prompt 4 — Arithmetic

A store sells notebooks for $2.75 each. If I buy 7 notebooks and pay with a $20 bill, how much change do I get?

Base (843 tokens): Worked through the problem in thinking, double-checked using cents, produced final boxed answer $0.75. Correct.

SLERP (111 tokens): Direct calculation, $0.75. Correct, ~1/7 the tokens.

Prompt 5 — Multi-step reasoning

A train leaves Station A at 9:00 AM traveling at 60 mph. Another train leaves Station B (180 miles away) at 10:00 AM traveling toward Station A at 90 mph. At what time do they meet?

Base (7966 tokens): Set up the correct equation 60t + 90(t-1) = 180, solved t = 1.8 hours. Then bungled the final clock conversion: wrote "9:00 AM + 1 hour + 48 minutes = 9:48 AM" (correct addition is 10:48 AM). Reasoning correct, final arithmetic wrong.

SLERP (449 tokens): Used the wrong equation entirely (forgot Train B departs an hour later), then produced a confused conclusion claiming the trains meet at "10:12 AM and 12:00 PM" simultaneously. Wrong.

Both wrong, but base was structurally correct and SLERP wasn't. The correct answer is ~10:48 AM.

Pattern observed

• The merge suppresses thinking behavior. Across all 5 prompts, base Qwen3-0.6B opens with <think>...</think> and reasons through the problem. SLERP outputs an empty <think>\n\n</think> and goes straight to the answer on every prompt. This was not an intentional design choice — the ultrachat (general) expert was trained on conversational data without thinking traces, and merging it appears to have weighted the model away from using the thinking block.
• SLERP is consistently faster. It produces 3-7× fewer tokens per response across all prompts. For latency-sensitive use cases that's a real practical benefit.
• Base is more thorough and more correct on complex prompts. When given proper sampling settings (temp=0.6, top_p=0.95), base completes its responses, follows complex instructions accurately, and sets up multi-step reasoning correctly. SLERP's brevity costs accuracy on harder prompts.
• Both can solve simple arithmetic. Both got prompt 4 right.
• Neither reliably solves multi-step word problems at this scale, but base's structured reasoning at least produces a checkable workflow.
• Practical implication: SLERP is positioned as a faster, more direct variant of Qwen3-0.6B. Use it when you want short answers without verbose chain-of-thought. Use the base model when you want full reasoning capability or strict instruction compliance.

Reproducing the evaluation

v1 (per task)

Base:

lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16 \
    --tasks arc_challenge --num_fewshot 25 --batch_size 4
# Repeat for hellaswag (10), boolq (0), piqa (0), winogrande (5)

This merge:

lm_eval --model hf \
    --model_args pretrained=OzdowntheR/Qwen3-0.6B-SciGen-SLERP,dtype=bfloat16 \
    --tasks arc_challenge --num_fewshot 25 --batch_size 4
# Repeat for hellaswag (10), boolq (0), piqa (0), winogrande (5)

v2 (per task, no chat template)

Base:

lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16 \
    --tasks leaderboard_bbh \
    --batch_size auto \
    --gen_kwargs max_gen_toks=2048 \
    --log_samples \
    --output_path results/base_v2_clean_leaderboard_bbh
# Repeat for leaderboard_mmlu_pro, leaderboard_musr,
#            leaderboard_ifeval, leaderboard_math_hard

This merge:

lm_eval --model hf \
    --model_args pretrained=OzdowntheR/Qwen3-0.6B-SciGen-SLERP,dtype=bfloat16 \
    --tasks leaderboard_bbh \
    --batch_size auto \
    --gen_kwargs max_gen_toks=2048 \
    --log_samples \
    --output_path results/slerp_v2_clean_leaderboard_bbh
# Repeat for leaderboard_mmlu_pro, leaderboard_musr,
#            leaderboard_ifeval, leaderboard_math_hard

Limitations

• Thinking suppression. The merge produces empty <think> blocks and skips chain-of-thought reasoning entirely. This is an unintended side effect of the general expert's training data (ultrachat) not containing thinking traces. For tasks requiring explicit reasoning, use the base model.
• 0.6B is small. Absolute scores on hard reasoning benchmarks (MATH Hard, MMLU-Pro, BBH) are low for both base and merge.
• MATH Hard regression (-6%) is the largest individual loss. If your use case involves math, prefer the base model.
• Trained only on English synthetic data (GPT-4 generated). Quality ceiling is bounded by the teacher.
• Tested only with thinking enabled at max_gen_toks=2048. Different generation budgets may produce different results.

Provenance

• v1 results verified with independent rerun (April 9, 2026). All numbers reproduced within standard error.
• v2 results were initially run with --apply_chat_template, then re-run without after we discovered the chat template causes Qwen3-0.6B to collapse to degenerate answer distributions on loglikelihood multiple-choice. Only the no-chat-template results are reported here.
• Per-sample logs from all v2 runs are saved as JSONL files alongside the result JSONs.

Citations

• Qwen3 base model: Qwen/Qwen3-0.6B
• mergekit (Goddard et al.): https://github.com/arcee-ai/mergekit
• lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
• camel-ai/physics dataset: camel-ai/physics
• ultrachat_200k dataset: HuggingFaceH4/ultrachat_200k