Downloads · 30 days
54
5% of all-time downloads
tifin-india/sarvam-m-24b-q5-k-m-gguf
sarvam-m-24b-q5-k-m-gguf is a text generation model from tifin-india. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This repository contains the Q5KM quantized version of sarvam-m-24b in GGUF format.
Downloads · 30 days
54
5% of all-time downloads
All-time downloads
1K
Public
Repo size
16.8 GB
Likes
2
Public
Click a slice to open those files.
.gguf16.8 GB · 100%
From the Hugging Face model README
This repository contains the Q5_K_M quantized version of sarvam-m-24b in GGUF format.
# Download the model
huggingface-cli download tifin-india/sarvam-m-24b-q5_k_m-gguf
# Run inference
./main -m sarvam-m-24b-Q5_K_M.gguf -p "Your prompt here"
from llama_cpp import Llama
# Load the model
llm = Llama(
model_path="./sarvam-m-24b-Q5_K_M.gguf",
n_ctx=2048, # Context length
n_gpu_layers=35, # Adjust based on your GPU
verbose=False
)
# Generate text
response = llm("Your prompt here", max_tokens=100)
print(response['choices'][0]['text'])
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
model_name = "tifin-india/sarvam-m-24b-q5_k_m-gguf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_quantized(model_name)
| Aspect | Rating |
|---|---|
| Speed | ⭐⭐ |
| Quality | ⭐⭐⭐⭐ |
| Memory | ⭐⭐ |
This is a quantized version of the original model. For the full-precision version and more details, please refer to the original model repository.
This model was quantized using llama.cpp's quantization tools. The Q5_K_M format provides a good balance of model size, inference speed, and output quality for most use cases.
This model follows the same license as the original model (Apache 2.0).
If you use this model, please cite the original model authors and acknowledge the quantization.