Lambent-Mira-Mashup-Aphrodite-27B

<div align="center"> # ϻira Λphrodite 27B [Edit: v2 with an actual Mira model as base for DeLerp. Haven't played around with it much.] <img src="https://cdn.britannica.com/23/176123-050-EC7A3FD6/Aphrodite.jpg" width="200" alt="Aphrodite">

All credits go to Lambent for the wonderful model series.

</div>

---

📖 Overview

This is a three-stage merge of pre-trained language models created using mergekit. It utilizes Karcher for the base consolidation, CABS for pruning/uniqueness, and DeLerp for intelligence and writing ability.

🧬 Merge Pipeline - DeLerp Experimenting, likely lacking in the essence of Mira

    %% Stage 1
    S1[Mira Mixture] -->|Karcher Mean| A(Λnanke)
    
    %% Stage 2
    A -->|Base| B(ϛoul)
    M1[Mira v1.17 Heretic] -->|CABS| B
    M2[Mira Mix 3] -->|CABS| B
    
    %% Stage 3
    B -->|DeLerp| C(ϻira Λphrodite)
    O[Lambent_Mira-v1.14 27B] -->|DeLerp| C

---

The Recipe

Stage 1 | Λnanke | Method: Karcher Mean

<details> <summary><b>View Configuration</b></summary>

name: Λnanke
models:
    - model: ./Ingredients/Lambent_Mira-v1.17-27B-Custom-Heretic
    - model: ./Ingredients/Lambent_Mira-v1-dpo-27B
    - model: ./Ingredients/Lambent_Mira-1.10-dpo-27B
    - model: ./Ingredients/Lambent_Mira-v1.8.1a-27B
    - model: ./Ingredients/Lambent_Mira-v1.12-Ties-27B
    - model: ./Ingredients/Lambent_Mira-v1.3-27B
    - model: ./Ingredients/Lambent_Mira-v1.5-27B
merge_method: karcher
parameters:
    max_iter: 1000
    tol: 1e-9
    normalize: false
    int8_mask: true
tokenizer:
    source: union
dtype: bfloat16
out_dtype: bfloat16

</details>

Stage 2 | ϛoul | Method: CABS

<details> <summary><b>View Configuration</b></summary>

name: Mira soul
models:
    - model: ./Ingredients/Lambent_Mira-v1.8.1a-27B
    - model: ./Ingredients/Lambent_Mira-v1.17-27B-Custom-Heretic
      parameters:
          weight: 0.4
          n_val: 16
          m_val: 32
    - model: ./Ingredients/mira-Ananke
      parameters:
          weight: 0.2
          n_val: 12
          m_val: 32
merge_method: cabs
default_n_val: 8
default_m_val: 32
pruning_order:
    - ./Ingredients/Lambent_Mira-v1.17-27B-Custom-Heretic
    - ./Ingredients/mira-Ananke
base_model: ./Ingredients/Lambent_Mira-v1.8.1a-27B
dtype: bfloat16
out_dtype: bfloat16
tokenizer:
    source: union

</details>

Stage 3 | ϻira Λphrodite | Method: DeLerp

<details> <summary><b>View Configuration</b></summary>

name: ϻira Λphrodite
models:
    - model: ./Ingredients/mira-soul
    - model: ./Ingredients/Lambent_Mira-v1.14-27B
merge_method: delerp
base_model: ./Ingredients/Lambent_Mira-v1.14-27B
parameters:
    t:
        - value: 0.999
dtype: float32
out_dtype: bfloat16
tokenizer:
    source: union

</details>

---

Quant Recipe (Q5_K_X)

<details> The quantization scheme is inspired by [ddh0](https://huggingface.co/ddh0/Q4_K_X.gguf) <summary><b>Quant Recipe</b></summary>

#!/usr/bin/env bash
set -euo pipefail
# Quant scheme inspired by: https://huggingface.co/ddh0/Q4_K_X.gguf
llama_quant="$LLAMAQUANT"
imatrix="$KITCHEN/Bartowski-Gemma-3-27B-imatrix.gguf" # Thanks =)
model=$KITCHEN/Models_cooking/Mira-Aphrodite-00001-of-00003.gguf
outpath="=$KITCHEN/Models/Mira-Aphrodite-27B-Q5_K_X.gguf"
quant_type="Q8_0"

blk_all="blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61)"
blk_step="blk\.(0|1|2|3|4|5|6|9|12|15|18|21|24|27|30|33|36|39|42|45|48|51|54|55|56|57|58|59|60|61)"
blk_alt="blk\.(7|8|10|11|13|14|16|17|19|20|22|23|25|26|28|29|31|32|34|35|37|38|40|41|43|44|46|47|49|50|52|53)"

custom=(
    --tensor-type token_embd.weight=Q5_K
    --tensor-type "${blk_all}\.attn_k.weight=Q8_0"
    --tensor-type "${blk_all}\.attn_output.weight=Q6_K"
    --tensor-type "${blk_all}\.attn_q.weight=Q5_K"
    --tensor-type "${blk_step}\.attn_v.weight=Q8_0" # eh, let her fly
    --tensor-type "${blk_step}\.ffn_down.weight=Q6_K"
    --tensor-type "${blk_all}\.ffn_gate.weight=Q5_K"
    --tensor-type "${blk_all}\.ffn_up.weight=Q5_K"
    --tensor-type "${blk_alt}\.attn_v.weight=Q8_0"
    --tensor-type "${blk_alt}\.ffn_down.weight=Q5_K"
)

"$llama_quant" --imatrix "$imatrix" "${custom[@]}" "$model" "$outpath" "$quant_type"

</details>

---

Nice People

• Lambent - For the wonderful Mira model series.
• Bartowski - For the Imatrix, work with Arcee, and being a pillar of the community.
• ddh0 - For the inspiration on the quant scheme.
• ubergarm - For all your public guides, from quanting to perplexity, overall positivity, and making things that appear intimidating much more approachable. I would not have been doing any of this if it weren't for you. Thank you.
• win10 - For the Karcher merge method implementation.
• grimjim - For the DeLerp merge method implementation.