Qwen3-8B-no-layer21

Qwen3-8B-no-layer21

Ablated version of Qwen/Qwen3-8B with layer 21 removed.

Key Finding

Removing layer 21 improves performance on MATH500 by 75%!

Model Details

• Base Model: Qwen3-8B
• Modification: Layer 21 removed
• Original Layers: 36
• Current Layers: 35
• Parameters: ~8B (slightly fewer than original)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "a-scarlett/Qwen3-8B-no-layer21",
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("a-scarlett/Qwen3-8B-no-layer21", trust_remote_code=True)

prompt = "Solve: What is 15 * 23?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Research Context

This model demonstrates that not all layers in a transformer are equally important. Layer 21 in Qwen3-8B appears to introduce noise or interfere with mathematical reasoning, and removing it actually improves performance.

Experimental Results

Tested on MATH500 dataset (500 competition-level math problems):

• Baseline (36 layers): 6.67% accuracy
• Without layer 21: 11.67% accuracy ✅
• Without layer 23: 5.00% accuracy ❌

License

Apache 2.0 (same as base model)

Citation

@misc{qwen3-8b-ablated,
  title={Qwen3-8B with Layer 21 Removed},
  author={a-scarlett},
  year={2025},
  howpublished={\url{https://huggingface.co/a-scarlett/Qwen3-8B-no-layer21}}
}