Downloads · 30 days
24
13% of all-time downloads
Llama-3.1-8B-Instruct-FP16-GGUF
annus-lums/Llama-3.1-8B-Instruct-FP16-GGUF
Llama-3.1-8B-Instruct-FP16-GGUF is a machine learning model from annus-lums. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.1.
Llama 3.1 8B Instruct in FP16 (baseline)
All-time downloads
185
Public
Repo size
16.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf16.1 GB · 100%
At a glance
- License
- llama3.1
- Access
- Public
- Created
- Dec 21, 2025
- Updated
- Dec 21, 2025
- SHA
- 44106d32
Base models
Model card
From the Hugging Face model README
Llama-3.1-8B-Instruct-FP16-GGUF
Description
Llama 3.1 8B Instruct in FP16 (baseline)
File: llama-3.1-8b-instruct-f16.gguf
Size: 16G
Format: GGUF
Category: baseline
Quick Start
Download
huggingface-cli download annus-lums/Llama-3.1-8B-Instruct-FP16-GGUF llama-3.1-8b-instruct-f16.gguf --local-dir .
Usage with llama.cpp
./llama-cli -m llama-3.1-8b-instruct-f16.gguf -p "Explain quantum computing" -n 128
Benchmark
./llama-bench -m llama-3.1-8b-instruct-f16.gguf -r 3
Project: AI on Edge Devices
This model is part of an LLM compression research project.
Pipeline
- Pruning: 20% structured Taylor pruning (MLP layers only)
- SmoothQuant: Activation smoothing for stable quantization
- Mixed Precision: Sensitivity-based bit-width allocation (Q4/Q5/Q6)
Results
- Size: 73.6% reduction (15 GB → 4 GB)
- Speed: 273% faster inference (1.16 → 4.33 tok/s)
- Deployment: Successfully runs on Raspberry Pi 4
Model Card
Created by: Group 2 (Annus, Arslan, Naveed, Danyal)
Institution: LUMS
Date: December 2024
Citation
@misc{llama31-compressed-Llama-3.1-8B-Instruct-FP16-GGUF,
author = {Group 2},
title = {Llama-3.1-8B-Instruct-FP16-GGUF},
year = {2024},
publisher = {HuggingFace},
url = {https://huggingface.co/annus-lums/Llama-3.1-8B-Instruct-FP16-GGUF}
}
- License
- llama3.1
- Created
- Dec 21, 2025
- Updated
- Dec 21, 2025