Downloads · 30 days
151
5% of all-time downloads
RichardErkhov/kz919_-_sliding_llama3_8b_no_finetune-gguf
kz919_-_sliding_llama3_8b_no_finetune-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
151
5% of all-time downloads
All-time downloads
3.1K
Public
Repo size
92.9 GB
Likes
0
Public
Click a slice to open those files.
.gguf108 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
sliding_llama3_8b_no_finetune - GGUF
Model Name: Sliding Llama
Base Model: Llama 3
Description: Sliding Llama is a variant of the Llama 3 model that introduces the ability to configure different layers with a sliding window approach. This configuration allows users to customize the attention and memory mechanisms across different layers.
sliding_windows argument.To use Sliding Llama for inference, you need to have a customized Hugging Face Transformers library installed. If you don't have it installed yet, you can do so with the following command:
pip install git+https://github.com/kyleliang919/transformers
This is important because we need a custom hybrid cache implementation for cached inference, since some of the model layers have different length of context (window).
For training, you can use the default transformers as it's.
The sliding_windows argument is a list where each element specifies the window size for the corresponding layer.
You can load the Sliding Llama model using the following code snippet:
For instance, in the example below there is one full attention in every four layers and have a total interpolated context of 32K (originally llama3 8b has 8K context length)
from transformers import AutoConfig, AutoTokenizer
from modeling_sliding_llama import LlamaForCausalLM
# Load the tokenizer and model
config = AutoConfig.from_pretrained("kz919/sliding_llama3_8b_no_finetune", trust_remote_code=True)
config.sliding_windows = [512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0, 512, 512, 512, 0]
config.rope_scaling = {
"factor": 4.0,
"high_freq_factor": 4.0,
"low_freq_factor": 1.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3"
}
tokenizer = AutoTokenizer.from_pretrained("kz919/sliding_llama3_8b_no_finetune")
model = LlamaForCausalLM.from_pretrained("kz919/sliding_llama3_8b_no_finetune",
config = config,
device_map="auto",
trust_remote_code=True)
prompt = "Your prompt here"
inputs = tokenizer(prompt, return_tensors = "pt")
outputs = model.generate(**inputs, use_cache = True)
print(tokenizer.decode(outputs[0]))
Notice in this repo, the weights are not finetuned (as indicated in the name), the weights are exactly identical as Llama3, you should be able to swap the weights or add a lora on top to accustom it to longer context. To use Lora adapters, you can use the following command after you load the model as above
from peft import PeftModel
model = PeftModel.from_pretrained(model, "path_to_your_adepter")
model = model.merge_and_unload()
Then you can do inference, generation calls as usual.
We thank the developers and researchers behind Llama 3 and the Hugging Face community for their contributions and support.
If you use this model in your research, please cite:
@inproceedings{slidingllama2024,
title={Sliding Llama},
author={Kaizhao Liang},
year={2024}
}
The Sliding Llama model is released under the Apache License 2.0.
For more details and updates, visit the Sliding Llama GitHub repository.