Meta-Llama-3-70B-Instruct-4bit-gguf

<div align="center">

📦 Meta-Llama-3-70B-Instruct-4bit-gguf

meta-llama/Meta-Llama-3-70B-Instruct converted to GUFF format

<a href="https://github.com/codewithdark-git/QuantLLM">⭐ Star QuantLLM on GitHub</a>

</div>

---

📖 About This Model

This model is meta-llama/Meta-Llama-3-70B-Instruct converted to GUFF format.

🚀 Quick Start

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("QuantLLM/Meta-Llama-3-70B-Instruct-4bit-gguf")
tokenizer = AutoTokenizer.from_pretrained("QuantLLM/Meta-Llama-3-70B-Instruct-4bit-gguf")

# Generate text
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With QuantLLM

from quantllm import TurboModel

# Load with automatic optimization
model = TurboModel.from_pretrained("QuantLLM/Meta-Llama-3-70B-Instruct-4bit-gguf")

# Generate
response = model.generate("Write a poem about coding")
print(response)

Requirements

pip install transformers torch

📊 Model Details

---

🚀 Created with QuantLLM

<div align="center">

Convert any model to GGUF, ONNX, or MLX in one line!

from quantllm import turbo

# Load any HuggingFace model
model = turbo("meta-llama/Meta-Llama-3-70B-Instruct")

# Export to any format
model.export("guff", quantization="Q4_K_M")

# Push to HuggingFace
model.push("your-repo", format="guff")

<a href="https://github.com/codewithdark-git/QuantLLM"> <img src="https://img.shields.io/github/stars/codewithdark-git/QuantLLM?style=social" alt="GitHub Stars"> </a>

📚 Documentation · 🐛 Report Issue · 💡 Request Feature

</div>