BigMaid-20B-v2.0

BigMaid-20B-v2.0

This is a merge of pre-trained language models created using mergekit. FP32 version

Tests

• model retains qualities of BigMaid-20B-v1.0 and it's also more creative and coherent.

Merge Details

Merge Method

• This model was merged using the passthrough merge method.
• Merge parameters were inspired by work of DavidAU and jukofyork
• thread
• thread

Models Merged

The following models were included in the merge:

• KatyTheCutie_EstopianMaid-13B

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [0, 16]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [8, 24]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [17, 32]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [25, 40]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
merge_method: passthrough
dtype: float32

All comments are greatly appreciated, download, test and if you appreciate my work, consider buying me my fuel: <a href="https://www.buymeacoffee.com/TeeZee" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>