TinyLlama-1.1B-Slerp-v1

TinyLlama-1.1B-Slerp-v1

This model is a merge of pre-trained language models created using mergekit. It combines the robustness of an intermediate checkpoint with the instructional capabilities of the Chat version, resulting in a model that balances foundational knowledge with instruction following.

⚡ Merge Details

This merge uses the SLERP (Spherical Linear Interpolation) method, which interpolates between the weights of the two models on a hypersphere. This method is generally preferred over simple linear averaging (soups) as it better preserves the geometric properties of the original models' feature spaces.

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
• TinyLlama/TinyLlama-1.1B-Chat-v1.0

Configuration

The following YAML configuration was used to produce this model:

merge_method: slerp
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
parameters:
  t:
    - value: 0.5
dtype: float16
slices:
  - sources:
      - model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
        layer_range: [0, 22]
      - model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
        layer_range: [0, 22]
        parameters:
          t:
            - value: 0.5

## 💻 Usage

You can use this model easily with the `pipeline` API, which handles the chat templates automatically.

```python
import torch
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="arirajuns/TinyLlama-1.1B-Slerp-v1",
    torch_dtype=torch.float16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain the concept of 'Model Merging' in one sentence."},
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

outputs = pipe(
    prompt,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95
)

print(outputs[0]["generated_text"])