Meta-Llama-3.1-70B-Instruct-IQ_1S

Model Information

The Llama 3.1 instruction tuned text only 70B model is optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.

This repository stores a experimental IQ_1S quantized GGUF Llama 3.1 instruction tuned 70B model.

Model developer: Meta

Model Architecture: Llama 3.1 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.

Supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

Quantization Information

Dataset used for re-calibration: Mix of standard_cal_data

The generated imatrix can be downloaded from imatrix.dat

Usage: with llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="npc0/Meta-Llama-3.1-70B-Instruct-IQ_1S",
	filename="GGUF_FILE",
)

llm.create_chat_completion(
		messages = [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
)