slicexai_-_Llama3.1-elm-turbo-4B-instruct-4bits

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Llama3.1-elm-turbo-4B-instruct - bnb 4bits

• Model creator: https://huggingface.co/slicexai/
• Original model: https://huggingface.co/slicexai/Llama3.1-elm-turbo-4B-instruct/

Original model description:

license: llama3.1 language:

• en

---

SliceX AI™ ELM Turbo

ELM (which stands for Efficient Language Models) Turbo is the next generation model in the series of cutting-edge language models from SliceX AI that is designed to achieve the best in class performance in terms of quality, throughput & memory.

<div align="center"> <img src="https://raw.githubusercontent.com/slicex-ai/elm-turbo/main/elm-turbo-training.png" width="768"/> </div>

ELM is designed to be a modular and customizable family of neural networks that are highly efficient and performant. Today we are sharing the second version in this series: ELM Turbo models (named Starfruit).

Model: ELM Turbo introduces a more adaptable, decomposable LLM architecture thereby yielding flexibility in (de)-composing LLM models into smaller stand-alone slices. In comparison to our previous version, the new architecture allows for more powerful model slices to be learnt during the training process (yielding better quality & higher generative capacity) and a higher level of control wrt LLM efficiency - fine-grained slices to produce varying LLM model sizes (depending on the user/task needs and deployment criteria, i.e., Cloud or Edge device constraints).

Training: ELM Turbo introduces algorithmic optimizations that allows us to train a single model but once trained the ELM Turbo model can be sliced in many ways to fit different user/task needs. We formulate the entire training procedure for ELM Turbo as a continual learning process during which we apply "slicing" operations & corresponding optimizations during the pre-training and/or fine-tuning stage. In a nutshell, this procedure teaches the model to learn & compress its knowledge into smaller slices.

Fast Inference with Customization: As with our previous version, once trained, ELM Turbo model architecture permits flexible inference strategies at runtime depending on deployment & device constraints to allow users to make optimal compute/memory tradeoff choices for their application needs. In addition to the blazing fast speeds achieved by native ELM Turbo slice optimization, we also layered in NVIDIA's TensorRT-LLM integration to get further speedups. The end result 👉 optimized ELM Turbo models that achieve one of the world's best LLM performance.

• Blog: Medium

• Github: https://github.com/slicex-ai/elm-turbo

• HuggingFace (access ELM Turbo Models in HF): 👉 here

ELM Turbo Model Release (version for sliced Llama 3.1)

In this version, we employed our new, improved decomposable ELM techniques on a widely used open-source LLM, meta-llama/Meta-Llama-3.1-8B-Instruct (8B params) (check Llama-license for usage). After training, we generated three smaller slices with parameter counts ranging from 3B billion to 6B billion.

• Section 1. 👉 instructions to run ELM-Turbo with the Huggingface Transformers library.

NOTE: The open-source datasets from the HuggingFace hub used for instruction fine-tuning ELM Turbo include, but are not limited to: allenai/tulu-v2-sft-mixture, microsoft/orca-math-word-problems-200k, mlabonne/WizardLM_evol_instruct_70k-ShareGPT, and mlabonne/WizardLM_evol_instruct_v2_196K-ShareGPT. We advise users to exercise caution when utilizing ELM Turbo, as these datasets may contain factually incorrect information, unintended biases, inappropriate content, and other potential issues. It is recommended to thoroughly evaluate the model's outputs and implement appropriate safeguards for your specific use case.

1. Run ELM Turbo models with Huggingface Transformers library.

There are three ELM Turbo slices derived from the Meta-Llama-3.1-8B-Instruct model:

1. slicexai/Llama3.1-elm-turbo-3B-instruct (3B params)
2. slicexai/Llama3.1-elm-turbo-4B-instruct(4B params)
3. slicexai/Llama3.1-elm-turbo-6B-instruct (6B params)

Make sure to update your transformers installation via pip install --upgrade transformers.

Example - To run the slicexai/Llama3.1-elm-turbo-4B-instruct

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

elm_turbo_model = "slicexai/Llama3.1-elm-turbo-4B-instruct"
model = AutoModelForCausalLM.from_pretrained( 
    elm_turbo_model,  
    device_map="cuda",  
    torch_dtype=torch.bfloat16,  
    trust_remote_code=True,
)
messages = [ 
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"}, 
]

tokenizer = AutoTokenizer.from_pretrained(elm_turbo_model, legacy=False) 
pipe = pipeline( 
    "text-generation", 
    model=model, 
    tokenizer=tokenizer, 
) 

generation_args = { 
    "max_new_tokens": 500, 
    "return_full_text": False,
    "repetition_penalty": 1.2,
    "temperature": 0.0, 
    "do_sample": False, 
} 

output = pipe(messages, **generation_args) 
print(output[0]['generated_text'])