Downloads · 30 days
6
6% of all-time downloads
ronantakizawa/higgs-llama-3-70b-awq
higgs-llama-3-70b-awq is a machine learning model from ronantakizawa. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.
This is a 4-bit AWQ quantized version of bosonai/Higgs-Llama-3-70B, optimized for efficient deployment with minimal quality degradation.
Downloads · 30 days
6
6% of all-time downloads
All-time downloads
108
Public
Parameters
70.6B
39.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors39.8 GB · 100%
How the weights are stored.
I3268.5B · 97%
From the Hugging Face model README
This is a 4-bit AWQ quantized version of bosonai/Higgs-Llama-3-70B, optimized for efficient deployment with minimal quality degradation.
Comprehensive evaluation across multiple task categories:
| Category | Accuracy | Avg Latency |
|---|---|---|
| General Knowledge | 100% | 51.74s |
| Reasoning | 100% | 55.86s |
| Code Generation | 100% | 51.52s |
| Creative Writing | 50% | 51.17s |
| Mathematics | 50% | 51.85s |
| Overall | 83% | 52.66s |
✅ Strengths:
⚠️ Limitations:
pip install autoawq transformers accelerate
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_id = "ronantakizawa/higgs-llama-3-70b-awq"
# Load model
model = AutoAWQForCausalLM.from_quantized(
model_id,
fuse_layers=True,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Generate
prompt = "Explain quantum computing in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.7,
top_p=0.95
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# For better quality (slower)
outputs = model.generate(
**inputs,
max_new_tokens=500,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1
)
# For faster inference (greedy decoding)
outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=False
)
from vllm import LLM, SamplingParams
llm = LLM(
model="ronantakizawa/higgs-llama-3-70b-awq",
quantization="awq",
dtype="float16",
gpu_memory_utilization=0.9
)
sampling_params = SamplingParams(
temperature=0.7,
top_p=0.95,
max_tokens=200
)
outputs = llm.generate(
"Explain the theory of relativity.",
sampling_params
)
✅ Working:
## Base Model Citation
Please refer to the [Higgs-Llama-3-70B model card](https://huggingface.co/bosonai/Higgs-Llama-3-70B) for the base model citation and additional details.
## Acknowledgments
- **Bosonai** for the Higgs-Llama-3-70B base model
## License
This model inherits the [Llama 3 Community License](https://llama.meta.com/llama3/license/) from the base model.
## Model Card Contact
For questions or issues with this quantized model, please open an issue on the [model repository](https://huggingface.co/ronantakizawa/higgs-llama-3-70b-awq).
---
**Model Version:** 1.0
**Quantization Method:** AWQ 4-bit (GEMM)