Qwen2.5-14B-Fusion

Arcee Fusion: The Art of Selective Merging

Arcee Fusion works in three key stages:

1.Importance Scoring:

Instead of blindly merging all parameters, Arcee Fusion calculates an importance score for each parameter, combining the absolute difference between model parameters with a divergence measure based on softmax distributions and KL divergence. This ensures that only meaningful changes are considered.

2.Dynamic Thresholding:

The algorithm analyzes the distribution of importance scores, calculating key quantiles (median, Q1, and Q3) and setting a dynamic threshold using median + 1.5 × IQR (a standard technique for outlier detection). This intelligently filters out less significant changes.

3.Selective Integration:

A fusion mask is created based on the importance scores and the threshold. Only the most significant elements are incorporated into the base model, ensuring that the merge process is adaptive and selective. This preserves the base model's stability while integrating the most valuable updates from the other model.

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Qwen/Qwen2.5-14B-Instruct
merge_method: arcee_fusion
base_model: Qwen/Qwen2.5-14B
parameters:
  normalize: true
  int8_mask: true
  rescale: false
dtype: float16
out_dtype: float16
tokenizer_source: base