Downloads · 30 days
116
5% of all-time downloads
RichardErkhov/mesolitica_-_malaysian-tinyllama-1.1b-16k-instructions-gguf
mesolitica_-_malaysian-tinyllama-1.1b-16k-instructions-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
116
5% of all-time downloads
All-time downloads
2.4K
Public
Repo size
12.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf14.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
malaysian-tinyllama-1.1b-16k-instructions - GGUF
language:
We use exact Llama2 Instruct chat template.
WandB report, https://wandb.ai/mesolitica/fpf-mallam-5b-instructions-16k/reports/Instruction-finetuning--Vmlldzo2MjE5Njg2
Dataset gathered at https://huggingface.co/collections/mesolitica/malaysian-synthetic-dataset-656c2673fe7fe0b1e9e25fe2
Notebook to prepare dataset at https://github.com/mesolitica/malaysian-dataset/blob/master/llm-instruction/combine-malay-no-alignment-multitasks-partial-ultrachat-v2.ipynb
This model is a quick demonstration that the base model can be easily fine-tuned to achieve some performance. It does have minimal moderation mechanisms.
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
def parse_llama_chat(
messages,
function_call = None,
default_system = 'Anda adalah pembantu AI yang berguna dan mampu jawab segala soalan yang diberikan.'
):
if messages[0]['role'] != 'system':
system = default_system
start_index = 0
else:
system = messages[0]['content']
start_index = 1
user_query = messages[-1]['content']
users, assistants = [], []
for q in messages[start_index:-1]:
if q['role'] == 'user':
users.append(q['content'])
elif q['role'] == 'assistant':
assistants.append(q['content'])
texts = [f'<s>[INST] <<SYS>>\n{system}\n<</SYS>>\n\n']
if function_call:
fs = []
for f in function_call:
f = json.dumps(f, indent=4)
fs.append(f)
fs = '\n\n'.join(fs)
texts.append(f'\n[FUNCTIONCALL]\n{fs}\n')
for u, a in zip(users, assistants):
texts.append(f'{u.strip()} [/INST] {a.strip()} </s><s>[INST] ')
texts.append(f'{user_query.strip()} [/INST]')
prompt = ''.join(texts).strip()
return prompt
TORCH_DTYPE = 'bfloat16'
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=getattr(torch, TORCH_DTYPE)
)
tokenizer = AutoTokenizer.from_pretrained('mesolitica/malaysian-tinyllama-1.1b-16k-instructions')
model = AutoModelForCausalLM.from_pretrained(
'mesolitica/malaysian-tinyllama-1.1b-16k-instructions',
use_flash_attention_2 = True,
quantization_config = nf4_config
)
messages = [
{'role': 'system', 'content': 'awak adalah AI yang mampu jawab segala soalan'},
{'role': 'user', 'content': 'kwsp tu apa'}
]
prompt = parse_llama_chat(messages)
inputs = tokenizer([prompt], return_tensors='pt', add_special_tokens=False).to('cuda')
generate_kwargs = dict(
inputs,
max_new_tokens=1024,
top_p=0.95,
top_k=50,
temperature=0.9,
do_sample=True,
num_beams=1,
)
r = model.generate(**generate_kwargs)
print(tokenizer.decode(r[0]))
<s> [INST] <<SYS>>
awak adalah AI yang mampu jawab segala soalan
<</SYS>>
kwsp tu apa [/INST] KWSP (Kumpulan Wang Simpanan Pekerja) merupakan sistem persaraan yang disediakan oleh kerajaan Malaysia untuk memberikan simpanan dan kebajikan kepada pekerja dan pekerja yang berumur 55 tahun ke atas. KWSP adalah singkatan bagi "Kumpulan Wang Simpanan Pekerja" dan ia merupakan salah satu dana persaraan yang popular di Malaysia. </s>