Downloads · 30 days
67
4% of all-time downloads
RichardErkhov/Haary_-_TinyLlama-1.1B-indo-v1-gguf
Haary_-_TinyLlama-1.1B-indo-v1-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
67
4% of all-time downloads
All-time downloads
1.8K
Public
Repo size
12.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf14.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
TinyLlama-1.1B-indo-v1 - GGUF
| Name | Quant method | Size |
|---|---|---|
| TinyLlama-1.1B-indo-v1.Q2_K.gguf | Q2_K | 0.4GB |
| TinyLlama-1.1B-indo-v1.IQ3_XS.gguf | IQ3_XS | 0.44GB |
| TinyLlama-1.1B-indo-v1.IQ3_S.gguf | IQ3_S | 0.47GB |
| TinyLlama-1.1B-indo-v1.Q3_K_S.gguf | Q3_K_S | 0.47GB |
| TinyLlama-1.1B-indo-v1.IQ3_M.gguf | IQ3_M | 0.48GB |
| TinyLlama-1.1B-indo-v1.Q3_K.gguf | Q3_K | 0.51GB |
| TinyLlama-1.1B-indo-v1.Q3_K_M.gguf | Q3_K_M | 0.51GB |
| TinyLlama-1.1B-indo-v1.Q3_K_L.gguf | Q3_K_L | 0.55GB |
| TinyLlama-1.1B-indo-v1.IQ4_XS.gguf | IQ4_XS | 0.57GB |
| TinyLlama-1.1B-indo-v1.Q4_0.gguf | Q4_0 | 0.59GB |
| TinyLlama-1.1B-indo-v1.IQ4_NL.gguf | IQ4_NL | 0.6GB |
| TinyLlama-1.1B-indo-v1.Q4_K_S.gguf | Q4_K_S | 0.6GB |
| TinyLlama-1.1B-indo-v1.Q4_K.gguf | Q4_K | 0.62GB |
| TinyLlama-1.1B-indo-v1.Q4_K_M.gguf | Q4_K_M | 0.62GB |
| TinyLlama-1.1B-indo-v1.Q4_1.gguf | Q4_1 | 0.65GB |
| TinyLlama-1.1B-indo-v1.Q5_0.gguf | Q5_0 | 0.71GB |
| TinyLlama-1.1B-indo-v1.Q5_K_S.gguf | Q5_K_S | 0.71GB |
| TinyLlama-1.1B-indo-v1.Q5_K.gguf | Q5_K | 0.73GB |
| TinyLlama-1.1B-indo-v1.Q5_K_M.gguf | Q5_K_M | 0.73GB |
| TinyLlama-1.1B-indo-v1.Q5_1.gguf | Q5_1 | 0.77GB |
| TinyLlama-1.1B-indo-v1.Q6_K.gguf | Q6_K | 0.84GB |
| TinyLlama-1.1B-indo-v1.Q8_0.gguf | Q8_0 | 1.09GB |
Base Model TinyLlama/TinyLlama-1.1B-Chat-v1.0
Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.
pip3 install --upgrade transformers optimum
# If using PyTorch 2.1 + CUDA 12.x:
pip3 install --upgrade auto-gptq
# or, if using PyTorch 2.1 + CUDA 11.x:
pip3 install --upgrade auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
If you are using PyTorch 2.0, you will need to install AutoGPTQ from source. Likewise if you have problems with the pre-built wheels, you should try building from source:
pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
git checkout v0.5.1
pip3 install .
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model_name_or_path = "daptheHuman/Merak-7B-v4-GPTQ"
# To use a different branch, change revision
# For example: revision="gptq-4bit-32g-actorder_True"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
device_map="auto",
trust_remote_code=False,
revision="main")
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
prompt = "Tell me about AI"
prompt_template=f'''### Instruction:
{prompt}
### Response:
'''
print("\n\n*** Generate:")
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
print(tokenizer.decode(output[0]))
# Inference can also be done using transformers' pipeline
print("*** Pipeline:")
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
max_new_tokens=512,
do_sample=True,
temperature=0.7,
top_p=0.95,
top_k=40,
repetition_penalty=1.1
)
print(pipe(prompt_template)[0]['generated_text'])
TinyLlama/TinyLlama-1.1B-Chat-v1.0 for README template.
pixabay.com