Magot-v2-Gemma2-8k-9B

Magot-v2-Gemma2-8k-9B

This repo contains a merge of pre-trained language models created using mergekit.

There is still some damage to the model from merging, but that has been partially mitigated by retaining embed_tokens and lm_head weights from the base model. This approach has proven to be more stable than task arithmetic merger.

Tested at temp=1, minP=0.01. Coherence is moderately high, though not perfect. The low weight (0.01) infusion of the Magnum model provided variety to text generation. This model is being released because it is "good enough" and interesting. Inherent model safety is still strong due to Instruct base, but narratives are less bounded by positivity.

There appear to be some melodramatic motifs that the model inherently steers toward when allowed to go its own way. The notion of favored tropes baked into a model is likely understudied, and is another lens through which model explainability can be viewed.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• anthracite-org/magnum-v3-9b-customgemma2
• grimjim/Kitsunebi-v1-Gemma2-8k-9B

Configuration

The following YAML configuration was used to produce this model:

slices:
- sources:
  - model: grimjim/Kitsunebi-v1-Gemma2-8k-9B
    layer_range: [0, 42]
  - model: anthracite-org/magnum-v3-9b-customgemma2
    layer_range: [0, 42]
merge_method: slerp
base_model: grimjim/Kitsunebi-v1-Gemma2-8k-9B
parameters:
  t:
    - filter: embed_tokens
      value: 0.0
    - filter: lm_head
      value: 0.0
    - value: 0.01
dtype: bfloat16