Downloads · 30 days
837
6% of all-time downloads
RichardErkhov/neuralmagic_-_Llama-2-7b-ultrachat200k-gguf
neuralmagic_-_Llama-2-7b-ultrachat200k-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
837
6% of all-time downloads
All-time downloads
13.5K
Public
Repo size
76.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf88.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Llama-2-7b-ultrachat200k - GGUF
base_model: meta-llama/Llama-2-7b-hf inference: true model_type: llama pipeline_tag: text-generation datasets:
This repo contains a Llama 2 7B finetuned for chat tasks using the UltraChat 200k dataset.
Official model weights from Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment.
Authors: Neural Magic, Cerebras
Below we share some code snippets on how to get quickly started with running the model.
By leveraging a pre-sparsified model's structure, you can efficiently fine-tune on new data, leading to reduced hyperparameter tuning, training times, and computational costs. Learn about this process here.
This model may be run with the transformers library. For accelerated inference with sparsity, deploy with nm-vllm or deepsparse.
# pip install transformers accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("neuralmagic/Llama-2-7b-ultrachat")
model = AutoModelForCausalLM.from_pretrained("neuralmagic/Llama-2-7b-ultrachat", device_map="auto")
input_text = "Write me a poem about Machine Learning."
input_ids = tokenizer.apply_chat_template(input_text, add_generation_prompt=True, return_tensors="pt").to("cuda")
outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))
Model evaluation metrics and results.
| Benchmark | Metric | Llama-2-7b-ultrachat | Llama-2-7b-pruned50-retrained-ultrachat |
|---|---|---|---|
| MMLU | 5-shot, top-1 | xxxx | xxxx |
| HellaSwag | 0-shot | xxxx | xxxx |
| WinoGrande | partial score | xxxx | xxxx |
| ARC-c | xxxx | xxxx | |
| TruthfulQA | 5-shot | xxxx | xxxx |
| HumanEval | pass@1 | xxxx | xxxx |
| GSM8K | maj@1 | xxxx | xxxx |
Coming soon.
For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community