Llama-3.1-70B-Instruct-GGUF

GGUF Model

---

Here are Quantized versions of Llama-3.1-70B-Instruct using GGUF

🤔 What Is GGUF

GGUF is designed for use with GGML and other executors. GGUF was developed by @ggerganov who is also the developer of llama.cpp, a popular C/C++ LLM inference framework. Models initially developed in frameworks like PyTorch can be converted to GGUF format for use with those engines.

☑️Uploaded Quantization Types

Here are the quantized versions that I have available:

• Q2_K
• Q3_K_S
• Q3_K_M
• Q3_K_L
• Q4_K_S
• Q4_K_M ~ Recommended
• Q5_K_S ~ Recommended
• Q5_K_M ~ Recommended
• Q6_K
• Q8_0 ~ NOT Recommended
• F16 ~ NOT Recommended
• F32 ~ NOT Recommended

Feel Free to reach out to me if you need a specific Quantization Type that I do not currently offer.

📈All Quantization Types Possible

Below is a table of all the Quantization Types that are possible as well as short descriptions.

💪 Benefits of using GGUF

By using a GGUF version of Llama-3.1-70B-Instruct, you will be able to run this LLM while having to use significantly less resources than you would using the non quantized version. This also allows you to run this 70B Model on a machine with less memory than a non quantized version.

⚙️️Installation

---

Here are 2 different methods you can use to run the quantized versions of Llama-3.1-70B-Instruct

1️⃣ Text-generation-webui

Text-generation-webui is a web UI for Large Language Models that you can run locally.

☑️ How to install Text-generation-webui

If you already have Text-generation-webui then skip this section

git clone https://github.com/oobabooga/text-generation-webui.git

✅Using Llama-3.1-70B-Instruct-GGUF with Text-generation-webui

2️⃣ Ollama

Ollama runs as a local service. Although it technically works using a command-line interface, Ollama's best attribute is their REST API. Being able to utilize your locally ran LLMs through the use of this API can give you almost endless possibilities!

Feel free to reach out to me if you would like to know some examples that I use this API for

☑️ How to install Ollama

Go To the URL below, and then select which OS you are using

https://ollama.com/download

Using Windows, or Mac you will then download a file and run it. If you are using linux it will just provide a single command that you need to run in your terminal window. That's about it for installing Ollama

✅Using Llama-3.1-70B-Instruct-GGUF with Ollama

Ollama does have a Model Library where you can download models:

https://ollama.com/library

This Model Library offers many different LLM versions that you can use. However at the time of writing this, there is no version of Llama-3.1-Instruct offered in the Ollama library.

If you would like to use Llama-3.1-Instruct (70B), do the following:

ollama run hf.co/hierholzer/Llama-3.1-70B-Instruct-GGUF:Q4_K_M

Replace Q4_K_M with whatever version you would like to use from this repository.

---