Downloads · 30 days
250
4% of all-time downloads
RichardErkhov/neuralmagic_-_Llama-2-7b-dolphin-open_platypus-pruned_70-gguf
neuralmagic_-_Llama-2-7b-dolphin-open_platypus-pruned_70-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
250
4% of all-time downloads
All-time downloads
6.4K
Public
Repo size
76.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf88.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Llama-2-7b-dolphin-open_platypus-pruned_70 - GGUF
base_model: neuralmagic/Llama-2-7b-pruned70-retrained inference: true model_type: llama pipeline_tag: text-generation datasets:
This repo contains a 70% sparse Llama 2 7B finetuned for instruction-following tasks using a blend of the Platypus + Open Orca + Dolphin datasets.
Official model weights from Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment.
Authors: Neural Magic, Cerebras
Below we share some code snippets on how to get quickly started with running the model.
By leveraging a pre-sparsified model's structure, you can efficiently fine-tune on new data, leading to reduced hyperparameter tuning, training times, and computational costs. Learn about this process here.
This model may be run with the transformers library. For accelerated inference with sparsity, deploy with nm-vllm or deepsparse.
# pip install transformers accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Llama-2-7b-pruned70-retrained-instruct")
model = AutoModelForCausalLM.from_pretrained("Llama-2-7b-pruned70-retrained-instruct", device_map="auto")
input_text = "Write a recipe for banana bread:\n"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))
Model evaluation metrics and results.
| Benchmark | Metric | Llama-2-7b-instruct | Llama-2-7b-pruned70-retrained-instruct |
|---|---|---|---|
| MMLU | 5-shot | 48.60% | 42.33% |
| HellaSwag | 10-shot | 79.45% | 77.21% |
| WinoGrande | 5-shot | 75.69% | 71.90% |
| ARC-c | 25-shot | 53.92% | 47.35% |
| TruthfulQA | 0-shot | 43.63% | 42.25% |
| GSM8K | 5-shot | 15.92% | 14.25% |
This model was obtained by sparse-tranfer of the sparse foundational model Llama-2-7b-pruned50-retrained on a blend of Open Platypus, 10% Open Orca and 10% Dolphin datasets. Training was perfomerd for 6 epochs.
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community