llama-2-7b-chat-tagalog-v0.3

Taga-llama-v0.3:

• Test model fine-tuned on an experimental Tagalog-focused dataset of ~1k items (based off Tagalog sentences augmented by LLaMA-2-13b base to create a mostly 3-turn dialogue dataset between Human and Assistant)
• Base: LLaMA-2 7b chat
• GGMLs, GGUFs
• QLoras (hf and GGML)

USAGE

This is meant to be mainly a chat model.

Use "Human" and "Assistant" and prompt with Tagalog. Example:

"Ito ay isang chat log sa pagitan ng AI Assistant na nagta-Tagalog at isang Pilipino. Magsimula ng chat:\nHuman: Hello po?\nAssistant:"

HYPERPARAMS

• Trained for 1 epoch
• rank: 16
• lora alpha: 32
• lora dropout: 0.5
• lr: 2e-4
• batch size: 2
• warmup ratio: 0.075
• grad steps: 4

WARNINGS AND DISCLAIMERS

Note that aside from formatting and other minor edits, dataset used is mostly as is augmented by LM. As such, while this version may be better at coherency or chatting than our previous Tagalog ones, conversations may still switch between languages or easily derail.

There is a chance that the model may switch back to English (albeit still understand Tagalog inputs) as conversations grow longer, resulting in English-Tagalog conversations: this may be because of the limited 3-turn nature of the dataset. Additionally, Taglish occuring in the dataset or any use of English may sometimes make the model more likely to output Taglish or even English responses.

Note that we use a partially synthetic dataset due to the lack of readily available Tagalog dialogue datasets, but take this as an opportunity to observe the Tagalog capability of base LLaMA-2. However, we plan to further curate the dataset (and fine tune later model versions on this) and release a final cleaned version.

Finally, this model is not guaranteed to output aligned or safe outputs nor is it meant for production use - use at your own risk!