Qwen2.5-0.5B-Instruct-Expanded-28L

Qwen2.5-0.5B-Instruct Expanded (28 Layers)

This model is a depth-expanded version of Qwen/Qwen2.5-0.5B-Instruct, extended from 24 to 28 layers using a gated layer expansion technique.

Model Architecture

Expansion Details

Four new decoder layers were inserted after positions 6, 12, 18, and 24 with gated initialization:

Each inserted layer is a copy of its source layer with a learnable gate: output = residual + gate * (layer_output - residual). The small initial gate value ensures the expanded model initially behaves closely to the original, allowing stable fine-tuning.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "jack-song-123/Qwen2.5-0.5B-Instruct-Expanded-28L",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "jack-song-123/Qwen2.5-0.5B-Instruct-Expanded-28L",
    trust_remote_code=True
)

messages = [{"role": "user", "content": "你好，请介绍一下你自己。"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Note: trust_remote_code=True is required because the model uses custom modeling_qwen2.py and configuration_qwen2.py for compatibility with transformers < 4.37.0.

Requirements

• transformers >= 4.36.0
• torch >= 2.0.0
• tokenizers >= 0.19.0

License

This model inherits the Apache 2.0 License from the original Qwen2.5 model.