Meta-Llama-3-8B-imatrix-GGUF

Quantizations of https://huggingface.co/meta-llama/Meta-Llama-3-8B

Update (May 1, 2024): re-uploaded models after this merge: https://github.com/ggerganov/llama.cpp/pull/6920 Models now work correctly (tried with 7777+3333 and 3333+777 using Q8_0, both gave correct results)

From original readme

How to use

This repository contains two versions of Meta-Llama-3-8B, for use with transformers and with the original llama3 codebase.

Use with transformers

See the snippet below for usage with Transformers:

>>> import transformers
>>> import torch

>>> model_id = "meta-llama/Meta-Llama-3-8B"

>>> pipeline = transformers.pipeline(
    "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
>>> pipeline("Hey how are you doing today?")

Use with llama3

Please, follow the instructions in the repository.

To download Original checkpoints, see the example command below leveraging huggingface-cli:

huggingface-cli download meta-llama/Meta-Llama-3-8B --include "original/*" --local-dir Meta-Llama-3-8B

For Hugging Face support, we recommend using transformers or TGI, but a similar command works.