Gemma-Ko-Merge

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the breadcrumbs_ties merge method using lemon07r/Gemma-2-Ataraxy-9B as a base.

Models Merged

The following models were included in the merge:

• wzhouad/gemma-2-9b-it-WPO-HB
• rtzr/ko-gemma-2-9b-it + ghost613/gemma9_on_korean_summary_events
• rtzr/ko-gemma-2-9b-it

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: lemon07r/Gemma-2-Ataraxy-9B
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.7
            gamma: 0.03
      - model: wzhouad/gemma-2-9b-it-WPO-HB
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
      - model: rtzr/ko-gemma-2-9b-it
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
      - model: rtzr/ko-gemma-2-9b-it+ghost613/gemma9_on_korean_summary_events # lora model loading
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
merge_method: breadcrumbs_ties
base_model: lemon07r/Gemma-2-Ataraxy-9B
dtype: bfloat16

Open LLM Leaderboard Evaluation Results

Detailed results can be found here