smollm-q

SmolLM Model Quantization with GGUF and llama.cpp

This project demonstrates the quantization of the SmolLM model using the GGUF format and llama.cpp to optimize performance for low-resource environments. Quantization reduces model size and inference latency, making it easier to deploy models on devices with limited hardware capabilities.

---

🚀 Usage

Model ID:

The MODEL_ID refers to the identifier of the model you want to quantize.

Quantization Method:

The QUANTIZATION_METHOD defines the approach used for reducing the precision of the model weights.

Example command:

python quantize_model.py --model_id <MODEL_ID> --quant_method <QUANTIZATION_METHOD>
⚙️ Quantization Methods
The quantization methods follow the naming convention q<bit_depth>_<variant>. Below is a comprehensive list of the available methods and their use cases:

Quant Method	Description
q2_k	Q4_K for attention.vw and feed_forward.w2 tensors, Q2_K for the rest.
q3_k_l	Q5_K for attention.wv, attention.wo, feed_forward.w2 tensors; Q3_K for the rest.
q3_k_m	Q4_K for attention.wv, attention.wo, feed_forward.w2 tensors; Q3_K for the rest.
q3_k_s	Q3_K for all tensors.
q4_0	The original 4-bit quantization method.
q4_1	Higher accuracy than q4_0 but with faster inference than q5.
q4_k_m	Q6_K for half of attention.wv and feed_forward.w2 tensors, Q4_K for the rest.
q4_k_s	Q4_K for all tensors.
q5_0	Higher accuracy, higher resource usage, and slower inference than q4.
q5_1	Even higher accuracy with increased resource usage and slower inference.
q5_k_m	Q6_K for half of attention.wv and feed_forward.w2 tensors, Q5_K for the rest.
q5_k_s	Q5_K for all tensors.
q6_k	Q8_K for all tensors.
q8_0	Near float16 precision, but with high resource use. Not recommended for most use cases.
Recommendation: For most cases, use Q5_K_M as it preserves the model's performance while optimizing memory usage. Use Q4_K_M if memory efficiency is more critical.

📊 Performance Guide
The following table summarizes the trade-offs between quantization methods:

Method	Precision	Speed	  Memory Usage	Recommended Use Case
q4_0	Medium	    Fast	  Low	            General-purpose, small models
q4_1	Higher	    Fast	  Low	            Faster inference, good accuracy
q5_0	High	    Moderate  Medium	        When higher accuracy is needed
q5_k_m	High	    Slow	  Moderate	        Best balance of accuracy and speed
q6_k	Very High	Slow	  High	            For accuracy-sensitive tasks
q8_0	Very High	Very Slow Very              High	Rarely recommended, only for high-precision tasks

📜 License
This project is licensed under the MIT License.

🤝 Contributing
We welcome contributions! If you'd like to help improve this project, please open an issue or submit a pull request.

🌟 Acknowledgements
This project is based on the model cards and quantization techniques shared by TheBloke.

For more details, visit TheBloke's GitHub.

📧 Contact
For any queries or support, please reach out at [[email protected]].