Downloads · 30 days
31
15% of all-time downloads
erichennings/TinyLlama-1.1B-Chat-v1.0-GGUF-Q4_K_M
TinyLlama-1.1B-Chat-v1.0-GGUF-Q4_K_M is a text generation model from erichennings. Use it when you need the model to write or continue text. It is set up for transformers.
This is just a test project - don't really use it!
Downloads · 30 days
31
15% of all-time downloads
All-time downloads
204
Public
Parameters
1.1B
4.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors4.4 GB · 100%
From the Hugging Face model README
This is just a test project - don't really use it!
This is a quantized version of TinyLlama-1.1B-Chat-v1.0
It was tested for performance with these results:
| Metric | Base Model | Quantized Model | Change |
|---|---|---|---|
| hellaswag acc | 0.485 | 0.495 | unchanged |
| hellaswag acc_norm | 0.64 | 0.64 | unchanged |
| eval time (CPU) - seconds | 970 | 751 | -23% |
Hellaswag accuracy was essentially maintained in the quantized version of the model, while time to complete the hellaswag eval decreased by ~23% (in a run on a low end CPU, with no access to a GPU).
For expediency a limited version of hellaswag was run with just 1% of samples evaluated - if this was going to be used for real we'd want to run the full eval set.
Note that all of the work to quantize and test the model were done using only CPU.
A quantized version of TinyLlama-1.1B-Chat-v1.0
The model was quantized to 4-bits (using llama.cpp and the Q4_K_M method), i.e. using commands like the following.
Convert original model to gguf:
python ./llama.cpp/convert_hf_to_gguf.py ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/
Quantize gguf to 4-bit Q4_K_M:
./llama.cpp/build/bin/llama-quantize ./llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/TinyLlama-1.1B-Chat-v1.0-F16.gguf q4_k_m