Downloads · 30 days
24
8% of all-time downloads
ggml-model-Q4_K_M
pavel-tolstyko/ggml-model-Q4_K_M
ggml-model-Q4_K_M is a machine learning model from pavel-tolstyko. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a quantized version of TinyLlama-1.1B-Chat-v1.0.
All-time downloads
290
Public
Repo size
668 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf668 MB · 100%
At a glance
- Access
- Public
- Created
- Nov 3, 2024
- Updated
- Nov 3, 2024
- SHA
- 1fb02450
Base models
Model card
From the Hugging Face model README
Model Card for TinyLlama-1.1B-Chat-v1.0 (Quantized)
This is a quantized version of TinyLlama-1.1B-Chat-v1.0.
Performance Evaluation
The quantized model was tested on the hellaswag dataset with the following results:
| Metric | Base Model | Quantized Model | Change |
|---|---|---|---|
| hellaswag accuracy | 0.456 | 0.462 | unchanged |
| hellaswag normalized accuracy | 0.64 | 0.64 | unchanged |
| eval time (GPU) - seconds | 219.67 | 209.34 | 4.70% decrease |
The quantized version of TinyLlama-1.1B-Chat-v1.0 maintains similar accuracy while achieving a 4.7% reduction in evaluation time. This evaluation was conducted using GPU resources on a subset of 100 hellaswag samples for expediency. For production purposes, it is recommended to perform a full evaluation.
Quantization Approach
The model was quantized to 4-bits using the Q4_K_M method with llama.cpp, specifically designed for optimized GPU performance. The following steps were used:
-
Convert the original model to GGUF format:
python ./llama.cpp/convert_hf_to_gguf.py ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/ -
Quantize the GGUF model to 4-bit Q4_K_M:
./llama.cpp/build/bin/llama-quantize ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/ggml-model-Q4_K_M.gguf q4_k_m
- Created
- Nov 3, 2024
- Updated
- Nov 3, 2024