Downloads · 30 days
9
26% of all-time downloads
RichardErkhov/mesolitica_-_malaysian-tinyllama-1.1b-16k-instructions-4bits
mesolitica_-_malaysian-tinyllama-1.1b-16k-instructions-4bits is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
9
26% of all-time downloads
All-time downloads
34
Public
Parameters
1.1B
808 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors807 MB · 100%
How the weights are stored.
U8969M · 87%
From the Hugging Face model README
Quantization made by Richard Erkhov.
malaysian-tinyllama-1.1b-16k-instructions - bnb 4bits
language:
We use exact Llama2 Instruct chat template.
WandB report, https://wandb.ai/mesolitica/fpf-mallam-5b-instructions-16k/reports/Instruction-finetuning--Vmlldzo2MjE5Njg2
Dataset gathered at https://huggingface.co/collections/mesolitica/malaysian-synthetic-dataset-656c2673fe7fe0b1e9e25fe2
Notebook to prepare dataset at https://github.com/mesolitica/malaysian-dataset/blob/master/llm-instruction/combine-malay-no-alignment-multitasks-partial-ultrachat-v2.ipynb
This model is a quick demonstration that the base model can be easily fine-tuned to achieve some performance. It does have minimal moderation mechanisms.
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
def parse_llama_chat(
messages,
function_call = None,
default_system = 'Anda adalah pembantu AI yang berguna dan mampu jawab segala soalan yang diberikan.'
):
if messages[0]['role'] != 'system':
system = default_system
start_index = 0
else:
system = messages[0]['content']
start_index = 1
user_query = messages[-1]['content']
users, assistants = [], []
for q in messages[start_index:-1]:
if q['role'] == 'user':
users.append(q['content'])
elif q['role'] == 'assistant':
assistants.append(q['content'])
texts = [f'<s>[INST] <<SYS>>\n{system}\n<</SYS>>\n\n']
if function_call:
fs = []
for f in function_call:
f = json.dumps(f, indent=4)
fs.append(f)
fs = '\n\n'.join(fs)
texts.append(f'\n[FUNCTIONCALL]\n{fs}\n')
for u, a in zip(users, assistants):
texts.append(f'{u.strip()} [/INST] {a.strip()} </s><s>[INST] ')
texts.append(f'{user_query.strip()} [/INST]')
prompt = ''.join(texts).strip()
return prompt
TORCH_DTYPE = 'bfloat16'
nf4_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=getattr(torch, TORCH_DTYPE)
)
tokenizer = AutoTokenizer.from_pretrained('mesolitica/malaysian-tinyllama-1.1b-16k-instructions')
model = AutoModelForCausalLM.from_pretrained(
'mesolitica/malaysian-tinyllama-1.1b-16k-instructions',
use_flash_attention_2 = True,
quantization_config = nf4_config
)
messages = [
{'role': 'system', 'content': 'awak adalah AI yang mampu jawab segala soalan'},
{'role': 'user', 'content': 'kwsp tu apa'}
]
prompt = parse_llama_chat(messages)
inputs = tokenizer([prompt], return_tensors='pt', add_special_tokens=False).to('cuda')
generate_kwargs = dict(
inputs,
max_new_tokens=1024,
top_p=0.95,
top_k=50,
temperature=0.9,
do_sample=True,
num_beams=1,
)
r = model.generate(**generate_kwargs)
print(tokenizer.decode(r[0]))
<s> [INST] <<SYS>>
awak adalah AI yang mampu jawab segala soalan
<</SYS>>
kwsp tu apa [/INST] KWSP (Kumpulan Wang Simpanan Pekerja) merupakan sistem persaraan yang disediakan oleh kerajaan Malaysia untuk memberikan simpanan dan kebajikan kepada pekerja dan pekerja yang berumur 55 tahun ke atas. KWSP adalah singkatan bagi "Kumpulan Wang Simpanan Pekerja" dan ia merupakan salah satu dana persaraan yang popular di Malaysia. </s>