Llama-3-8B-Instruct-80K-QLoRA-Merged-GGUF

<div align="center"> <h1>Llama-3-8B-Instruct-80K-QLoRA-Merged</h1>

<a href="https://github.com/FlagOpen/FlagEmbedding/tree/master/Long_LLM/longllm_qlora">[Data&Code]</a>

</div>

We extend the context length of Llama-3-8B-Instruct to 80K using QLoRA and 3.5K long-context training data synthesized from GPT-4. The entire training cycle is super efficient, which takes 8 hours on a 8xA800 (80G) machine. Yet, the resulted model achieves remarkable performance on a series of downstream long-context evaluation benchmarks.

NOTE: This repo contains the quantized model of namespace-Pt/Llama-3-8B-Instruct-80K-QLoRA-Merged. The quantization is conducted with llama.cpp (Q4_K_M and Q8_0).

All the following evaluation results are based on the UNQUANTIZED MODEL. They can be reproduced following instructions here. However, after quantization, you may observe quality degradation.

Needle in a Haystack

We evaluate the model on the Needle-In-A-HayStack task using the official setting. The blue vertical line indicates the training context length, i.e. 80K.

<img src="data/needle.png"></img>

LongBench

We evaluate the model on LongBench using 32K context length and the official prompt template. For meta-llama/Meta-Llama-3-8B-Instruct, we use 8K context length.

InfiniteBench

We evaluate the model on InfiniteBench using 80K context length and the official prompt template. The results of GPT-4 is copied from the paper. For meta-llama/Meta-Llama-3-8B-Instruct, we use 8K context length.

Topic Retrieval

We evaluate the model on Topic Retrieval task with [5,10,15,20,25,30,40,50,60,70] topics.

<img src="data/topic.png"></img>

MMLU

We evaluate the model's zero-shot performance on MMLU benchmark as a reflection of its short-context capability.

Environment

llama_cpp
torch==2.1.2
transformers==4.39.3

Usage

huggingface-cli download namespace-Pt/Llama-3-8B-Instruct-80K-QLoRA-Merged-GGUF --local-dir . --local-dir-use-symlinks False

In python,

from llama_cpp import Llama

llm = Llama(
  model_path="./Llama-3-8B-Instruct-80K-QLoRA-Merged-Q4_K_M.gguf",  # path to GGUF file
  n_ctx=81920,
  n_threads=96,
  n_gpu_layers=32,
)

with open("./data/needle.txt") as f:
    text = f.read()
    inputs = f"{text}\n\nWhat is the best thing to do in San Francisco?"

print(
  llm.create_chat_completion(
    messages = [
        {
            "role": "user",
            "content": inputs
        }
    ],
    temperature=0,
    max_tokens=50
  )
)

# The best thing to do in San Francisco is sitting in Helmer Dolores Park on a sunny day, eating a double cheeseburger with ketchup, and watching kids playing around.