kunoichi-lemon-royale-v2ext-32K-7B

kunoichi-lemon-royale-v2ext-32K-7B

This merge amounts to the grafting of a model derived from Mistral v0.1 7B (4K sliding window context, to a maximum of 8K practical context length) onto a model derived from Mistral v0.2 7B (32K context length, to a maximum of 16K practical context length).

Code from the Mistral Python script extend_model_vocab.py was used to extend the vocabulary from size 32000 to 32768, adapted to work on safetensor shards. The resulting model appears to be broken, unfortunately. However, this opens up the model to unconventional merger with Mistral v0.3 7B models.

This was a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the task arithmetic merge method using grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B as a base.

Models Merged

The following models were included in the merge:

• grimjim/kunoichi-lemon-royale-7B

Configuration

The following YAML configuration was used to produce this model:

base_model: grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B
dtype: bfloat16
merge_method: task_arithmetic
slices:
- sources:
  - layer_range: [0, 32]
    model: grimjim/Mistral-7B-Instruct-demi-merge-v0.2-7B
  - layer_range: [0, 32]
    model: grimjim/kunoichi-lemon-royale-7B
    parameters:
      weight: 1.00