Downloads · 30 days
12
4% of all-time downloads
alokabhishek/Llama-2-7b-chat-hf-5.0-bpw-exl2
Llama-2-7b-chat-hf-5.0-bpw-exl2 is a text generation model from alokabhishek. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
This repo contains 5-bit quantized (using ExLlamaV2) model of Meta's meta-llama/Llama-2-7b-chat-hf
Downloads · 30 days
12
4% of all-time downloads
All-time downloads
292
Public
Repo size
6.1 GB
Likes
1
Public
Click a slice to open those files.
.safetensors6.1 GB · 100%
From the Hugging Face model README
This repo contains 5-bit quantized (using ExLlamaV2) model of Meta's meta-llama/Llama-2-7b-chat-hf
Use the code below to get started with the model.
# Install ExLLamaV2
!git clone https://github.com/turboderp/exllamav2
!pip install -e exllamav2
from huggingface_hub import login, HfApi, create_repo
from torch import bfloat16
import locale
import torch
import os
# Define the model ID for the desired model
model_id = "alokabhishek/Llama-2-7b-chat-hf-5.0-bpw-exl2"
BPW = 5.0
# define variables
model_name = model_id.split("/")[-1]
!git-lfs install
# download the model to loacl directory
!git clone https://{username}:{HF_TOKEN}@huggingface.co/{model_id} {model_name}
# Run model
!python exllamav2/test_inference.py -m {model_name}/ -p "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
import sys, os
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from exllamav2 import (
ExLlamaV2,
ExLlamaV2Config,
ExLlamaV2Cache,
ExLlamaV2Tokenizer,
)
from exllamav2.generator import ExLlamaV2BaseGenerator, ExLlamaV2Sampler
import time
# Initialize model and cache
model_directory = "/model_path/Llama-2-7b-chat-hf-5.0-bpw-exl2/"
print("Loading model: " + model_directory)
config = ExLlamaV2Config(model_directory)
model = ExLlamaV2(config)
cache = ExLlamaV2Cache(model, lazy=True)
model.load_autosplit(cache)
tokenizer = ExLlamaV2Tokenizer(config)
# Initialize generator
generator = ExLlamaV2BaseGenerator(model, cache, tokenizer)
# Generate some text
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.85
settings.top_k = 50
settings.top_p = 0.8
settings.token_repetition_penalty = 1.01
settings.disallow_tokens(tokenizer, [tokenizer.eos_token_id])
prompt = "Tell me a funny joke about Large Language Models meeting a Blackhole in an intergalactic Bar."
max_new_tokens = 512
generator.warmup()
time_begin = time.time()
output = generator.generate_simple(prompt, settings, max_new_tokens, seed=1234)
time_end = time.time()
time_total = time_end - time_begin
print(output)
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]
[More Information Needed]