Downloads · 30 days
32
22% of all-time downloads
dranreb1660/medimaven-llama3-8b-awq
medimaven-llama3-8b-awq is a text generation model from dranreb1660. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Drop-in 4-bit AWQ quantisation of the MediMaven fp16 weights – fits on a 16 GB GPU (e.g. T4).
Downloads · 30 days
32
22% of all-time downloads
All-time downloads
143
Public
Parameters
8B
5.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.7 GB · 100%
How the weights are stored.
I327B · 87%
From the Hugging Face model README
Drop-in 4-bit AWQ quantisation of the MediMaven fp16 weights – fits on a 16 GB GPU (e.g. T4).
| Footprint | ≈ 5.9 GB on disk / VRAM |
| Throughput | ~29 tok/s on a single T4 (batch = 1) |
| Accuracy loss | < 0.3 ROUGE vs fp16 |
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("dranreb1660/medimaven-llama3-8b-awq")
model = AutoModelForCausalLM.from_pretrained(
"dranreb1660/medimaven-llama3-8b-awq",
device_map="auto",
torch_dtype="auto" # bitsandbytes picks int4 automatically
)
AWQ group_size=128, zero_point=True, zero_sym=True.
Calibrated on 128 in-domain prompts (medical Q&A).
Exported with AutoAWQ v0.2.3.
The model inherits all limitations and licensing terms of the fp16 weights.
For maximum accuracy in secondary fine-tuning, use the fp16 repo instead.
@misc{medimaven2025llama3,
title = {MediMaven Llama-3.1-8B},
author = {Kyei-Mensah, Bernard},
year = {2025},
howpublished = {\url{https://huggingface.co/dranreb1660/medimaven-llama3-8b-fp16}}
}