Llama3-8B-SuperNova-Spectrum-dare_ties

Llama3-8B-SuperNova-Spectrum-dare_ties

Llama3-8B-SuperNova-Spectrum-dare_ties is a dare_ties merge of the following models using LazyMergekit:

• yuvraj17/Llama-3-8B-spectrum-25
• ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
• arcee-ai/Llama-3.1-SuperNova-Lite

DARE_TIES Merging

TIES Merging

TIES Merging, introduced by Yadav et al. (2023), is a method for merging multiple specialized models into one general-purpose model. It solves two key challenges:

• Redundancy Removal: Identifies and eliminates overlapping or unnecessary information between models, making the final model more efficient.
• Conflict Resolution: Reconciles differences between models by creating a unified sign vector that represents the most dominant direction of change across all models.

TIES stands for TRIM, ELECT SIGN & MERGE (TIES-MERGING).

<figure> <img src="https://cdn-uploads.huggingface.co/production/uploads/66137d95e8d2cda230ddcea6/2vBgcGko-tcsaAkLUzHnU.png" width="1000" height="768"> <figcaption> How TIES-Merging Works <a href="//arxiv.org/pdf/2306.01708">Reference</a> </figcaption> </figure>

DARE Merging

Introduced by Yu et al. (2023), DARE uses an approach similar to TIES with two main differences:

• Weight Pruning: Randomly resets some fine-tuned weights to their original values, reducing model complexity.
• Weight Scaling: Adjusts the remaining weights by scaling and combining them with the base model's weights to maintain consistent performance.

DARE stands for DROP AND RESCALE

Mergekit’s implementation of DARE-Merging has two flavours: with the sign election step of TIES (dare_ties) or without (dare_linear). I have chosen dare_ties for this merge.

For more information refer this Merge Large Language Models with MergeKit by Maxime Labonne

Also, if you want to get in-depth knowledge about Model-Merging and its different types, I highly recommend this YouTube Video by Julien Simon

🧩 Configuration

models:
  - model: NousResearch/Meta-Llama-3-8B
    # No parameters necessary for base model
  - model: yuvraj17/Llama-3-8B-spectrum-25
    parameters:
      density: 0.56
      weight: 0.12
  - model: ruggsea/Llama3-stanford-encyclopedia-philosophy-QA
    parameters:
      density: 0.56
      weight: 0.12
  - model: arcee-ai/Llama-3.1-SuperNova-Lite
    parameters:
      density: 0.58
      weight: 0.55
merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
dtype: bfloat16

💻 Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "yuvraj17/Llama3-8B-SuperNova-Spectrum-dare_ties"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

A large language model is a type of artificial intelligence (AI) model designed to understand and generate human language. It is trained on a massive corpus of text data, which it uses to learn patterns and relationships between words and concepts. Large language models are typically based on a deep learning approach called transformer architecture, which was introduced by the Google research paper "Attention Is All You Need" (2017). These models are designed to handle the complexity of natural language by capturing long-range dependencies and contextual relationships between words. Large language models can perform a variety of tasks, including:

• Natural language processing (NLP): large language models can understand and generate text, and can be used for tasks such as text classification, sentiment analysis, and named entity recognition.
• Text generation: large language models can generate human-like text, such as chatbots, language translation, and text summarization.
• Question answering: large language models can answer questions based on the text they have been trained on.
• Conversational AI: large language models can be used to create conversational agents that can understand and respond to user input.

🏆 Evaluation Scores

Nous

AGIEval

Average: 38.32%

TruthfulQA

Average: 57.15%

Bigbench

Average: 43.91%

Special thanks & Reference

• Maxime Labonne for their easy-to-use colab-notebook Merging LLMs with MergeKit, Blog and LLM-AutoEva Notebookl
• Authors of Mergekit

Open LLM Leaderboard Evaluation Results

Detailed results can be found here