Hush-Qwen2.5-7B-Preview

Performance Highlights

Hush-Qwen2.5-7B-Preview was created using the YoYo v3 merge technique, achieving a new high on the IFEVAL test for 7B models with a score of 79.62%. This makes it the second-best model in that category, though the leading model is currently unavailable, meaning we might be in first place by default!

Strengths

• High IFEVAL Score: 79.62%, among the best for 7B models.
• Well-rounded performance: Decent scores across various benchmarks.

Weaknesses

• Low MATH Score: 35%, which is significantly lower than our past models (which scored at least 45%). Improving this would make the model substantially better overall.

Benchmark Results

Next Steps

• Finetune on Math: Bringing up the math score is a priority to create a well-balanced model.
• Explore YoYo v4: The next step could be merging this model with another one that is strong in math using the YoYo v4 technique. However, YoYo v4 lacks proper documentation, making it a challenge to implement.
• Develop a Math-Strong Model: An alternative approach is to build a new model that performs decently in all benchmarks but excels in math, then merge it with this one.

Conclusion

Hush-Qwen2.5-7B-Preview is a strong contender in the IFEVAL category, achieving one of the highest scores among 7B models. However, improving the math benchmark is a key priority for future iterations. By either finetuning or leveraging new merge techniques like YoYo v4, we can push the model to new heights.