Magnolia-v3-12B-8bpw_h8_exl3

Magnolia-v3-12B-8bpw_h8_exl3

This repo contains a 4 bpw with 8 head bits exl3 quant of a merge of pre-trained language models created using mergekit.

This merge takes advantage of task arithmetic to incorporate the influence of two additional models at low weight in order to improve default creative outputs.

Tested at temperature 1.0 with minP 0.01; that was it. Mistral prompt formats for Nemo should work, although I successfully tested with a variant:

User message prefix: [INST]user
User message suffix: [/INST]
Assistant message prefix: [INST]assistant
Assistant message suffix: [/INST]

Testing with a blank sysprompt was not awful, though a good sysprompt will elicit more desired behavior.

Merge Details

Merge Method

This model was merged using the task arithmetic merge method using grimjim/mistralai-Mistral-Nemo-Base-2407 as a base.

Models Merged

The following models were included in the merge:

• grimjim/magnum-consolidatum-v1-12b
• nbeerbower/Mistral-Nemo-Prism-12B
• TheDrummer/Rocinante-12B-v1.1
• grimjim/magnum-twilight-12b
• grimjim/mistralai-Mistral-Nemo-Instruct-2407

Configuration

The following YAML configuration was used to produce this model:

base_model: grimjim/mistralai-Mistral-Nemo-Base-2407
dtype: bfloat16
merge_method: task_arithmetic
parameters:
  normalize: true
slices:
- sources:
  - layer_range: [0, 40]
    model: grimjim/mistralai-Mistral-Nemo-Base-2407
  - layer_range: [0, 40]
    model: grimjim/mistralai-Mistral-Nemo-Instruct-2407
    parameters:
      weight: 0.9
  - layer_range: [0, 40]
    model: grimjim/magnum-consolidatum-v1-12b
    parameters:
      weight: 0.1
  - layer_range: [0, 40]
    model: grimjim/magnum-twilight-12b
    parameters:
      weight: 0.001
  - layer_range: [0, 40]
    model: TheDrummer/Rocinante-12B-v1.1
    parameters:
      weight: 0.001
  - layer_range: [0, 40]
    model: nbeerbower/Mistral-Nemo-Prism-12B
    parameters:
      weight: 0.05