llama-v1-2bit-gguf

ikawrakow/llama-v1-2bit-gguf

llama-v1-2bit-gguf is a machine learning model from ikawrakow. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

This repository contains 2-bit quantized LLaMA-v1 models in GGUF format for use with llama.cpp. All tensors are quantized with Q2K, except for output.weight, which is Q6K, and, in the case of LLaMA-v2-70B, attnv, whic…

Downloads · 30 days

113

7% of all-time downloads

All-time downloads

1.6K

Public

Repo size

39 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.gguf39 GB · 100%

At a glance

License: apache-2.0
Access: Public
Created: Dec 6, 2023
Updated: Dec 7, 2023
SHA: 40601485

Model card

From the Hugging Face model README

This repository contains 2-bit quantized LLaMA-v1 models in GGUF format for use with llama.cpp. All tensors are quantized with Q2_K, except for output.weight, which is Q6_K, and, in the case of LLaMA-v2-70B, attn_v, which is Q4_K. The quantized models differ from the standard llama.cpp 2-bit quantization in two ways:

These are actual 2-bit quantized models instead of the mostly 3-bit quantization provided by the standard llama.cpp Q2_K quantization method
The models were prepared with a refined (but not yet published) k-quants quantization approach

The table shows Wikitext perplexities for a context length of 2048 tokens computed with these models using llama.cpp

Model	Perplexity
7B	6.4023
13B	5.3967
30B	4.5065
65B	3.9136

License: apache-2.0
Created: Dec 6, 2023
Updated: Dec 7, 2023