SeaLLM-13B-Chat

<p align="center"> <img src="seal_logo.png" width="200" /> </p>

SeaLLMs - Large Language Models for Southeast Asia

UPDATE: SeaLLM-7B-v2 is released with state-of-the-art performance.

<p align="center"> <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b" target="_blank" rel="noopener"> 🤗 Tech Memo</a> &nbsp;&nbsp; <a href="https://huggingface.co/spaces/SeaLLMs/SeaLLM-Chat-13b" target="_blank" rel="noopener"> 🤗 DEMO</a> &nbsp;&nbsp; <a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a> &nbsp;&nbsp; <a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a> </p>

We introduce SeaLLMs - a family of language models optimized for Southeast Asian (SEA) languages. The SeaLLM-base models (to be released) were pre-trained from Llama-2, on a tailored publicly-available dataset, which comprises texts in Vietnamese 🇻🇳, Indonesian 🇮🇩, Thai 🇹🇭, Malay 🇲🇾, Khmer 🇰🇭, Lao 🇱🇦, Tagalog 🇵🇭 and Burmese 🇲🇲. The SeaLLM-chat underwent supervised fine-tuning (SFT) and specialized self-preferencing DPO using a mix of public instruction data and a small number of queries used by SEA language native speakers in natural settings, which adapt to the local cultural norms, customs, styles and laws in these areas.

SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages such as Thai, Khmer, Lao, and Burmese.

• DEMO: SeaLLMs/SeaLLM-7B.
• Technical report: Arxiv: SeaLLMs - Large Language Models for Southeast Asia.
• Model weights:
  • (New) SeaLLM-7B-v2: A new model with state-of-the-art performance in reasoning capabilities across all tasks and languages, while half the size of SeaLLM-13B
  • SeaLLM-7B-v1: Lower capability than SeaLLM-13B-Chat but much faster and memory-efficient.

<blockquote style="color:red"> <p><strong style="color: red">Terms of Use and License</strong>: By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>. </blockquote>

Disclaimer: We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation. Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.

The logo was generated by DALL-E 3.

The following sections summarize the performance evaluations of SeaLLMs and the training process.

Evaluation

Sea-bench Peer Comparison

One of the most reliable ways to compare chatbot models is peer comparison. With the help of native speakers, we built an instruction test set, called Sea-bench that focuses on various aspects expected in a user-facing chatbot, namely: (1) task-solving (e.g., translation & comprehension), (2) math-reasoning (e.g., math and logical reasoning questions), (3) general-instruction (e.g., instructions in general domains), (4) natural-questions (e.g., questions about local context often written informally), and (5) safety-related questions. The test set also covers all languages that we are concerned with. Similar to MT-bench, we use GPT-4 as an evaluator to rate the comparison between our models versus ChatGPT-3.5 and other baselines.

We evaluate Sea-bench in 2 modes: Score-based grading (0 to 10) and Peer comparison.

As shown in the figure above, as aggregated by task category (left radar chart), our SeaLLM-13b model performs on-par or surpasses ChatGPT-3.5 across many linguistic and writing tasks. This is despite reported evidence that GPT-4 evaluator may favor ChatGPT more often than humans do.

When comparing instruction-following capabilities of models from the angle of the different SEA languages, as shown, SeaLLM-13b outperforms ChatGPT-3.5 by large margins in most non-Latin languages, such as Burmese (Mya), Lao, Khmer and Thai. In combination with the fact that SeaLLM can encode these languages with up to 9 times fewer tokens, our models are not only superior but also cheaper to operate in these languages than ChatGPT. This helps democratize the benefits of large language models to under-represented and potentially developing communities.

<div class="row" style="display: flex; clear: both;"> <img src="img/seallm10l_vs_chatgpt_by_lang.png" alt="Forest" style="float: left; width: 49.5%"> <img src="img/seallm10l_vs_chatgpt_by_cat.png" alt="Snow" style="float: left; width: 49.5%"> </div>

We also compare our model head-on with ChatGPT in peer comparison, as seen above. SeaLLM-13b is equal or better than ChatGPT for up to 40% of the time for Latin-based languages (Eng, Vie, Ind, Msa). In contrast, for non-Latin languages, SeaLLM-13b surpasses ChatGPT by up to 90%.

Safety Enchancement in Local Context

There is growing evidence that western-built LLMs often neglect safety protection in many lower-resource languages, or even promote contents that may be locally perceived as harmful, inappropriate or illegal by local norms and laws. We take great effort in adapting and safeguarding our SeaLLM models so as to achieve wider adoption and compliance for the regional audiences of Southeast Asia.

The dropdown table below showcases examples of potentially harmful content that ChatGPT generates, whereas our model behaves safer and complies with the regulations.

<details> <summary><span style="color: red">WARNING:</span> The dropdown will display potentially harmful content.</summary> <br> </details>

M3Exam - World Knowledge in Regional Languages

M3Exam is a collection of real-life and native official human exam question benchmark. This benchmark covers questions from multiple countries in the SEA region, which require strong multilingual proficiency and cultural knowledge across various critical educational periods, from primary- to high-school levels of difficulty.

As shown in the table, our SeaLLM model outperforms most 13B baselines and reaches closer to ChatGPT's performance. Notably, for Thai - a seemingly low-resource language, our model is just 1% behind ChatGPT despite the large size difference.

MMLU - Preserving English-based knowledge

On the 5-shot MMLU, our SeaLLM models not only preserve but also slightly outperforms 13B LLama-2 and Llama-2-chat, despite the fact that optimizing for this English dominant test set is not part of our goal.

Machine Translation

We use the Flores-200 to test our model's ability in machine translation. As shown in the above figure, SeaLLM-13B exhibits clear superiority over ChatGPT-3.5 in low-resource languages, such as Lao and Khmer, while maintaining comparable performances with ChatGPT-3.5 in most high-resource languages (e.g., Vietnamese and Indonesian).

Training process

Vocabulary Expansion

Like many English/Latin-dominant LLMs, Llama-2's BPE tokenizer breaks non-European and non-Latin linguistic texts into unsustainably long byte-level sequences that cover much shorter semantic meanings, leading to degraded performance. For instance, it takes 4.3x more tokens to encode the same sentence in Thai compared to that in English (see below table). This leads to the models failing to perform tasks requiring long context modeling.

Our goal for vocabulary expansion is threefold: (1) the number of newly-added tokens must be minimal and only cover the new languages, (2) the tokens should bring the compression ratios of new languages close to that of English, and (3) minimize the disruption of existing European tokens to preserve Llama-2 knowledge. In the end, we obtain ~16K new tokens from SEA languages to augment the original 32K-token vocabulary. Our expansion technique is detailed in our technical report.

As seen in the table below, our new vocabulary reduces the compression ratio from 4.29 to 1.57 for Thai - meaning it can now encode 2.7x longer Thai text given the same context length.

Pre-training Data

The pre-training dataset of SeaLLMs is formed by the documents from diverse public sources, including web texts (e.g., Common Crawl), news documents (e.g., CC-News), academic articles, and texts with expert knowledge (e.g., Wikipedia). We firstly employ FastText language indentifier to filter out the documents that do not belong to SEA languages. To further remove harmful or undesirable content, we develop a pipeline with various data cleaning and filtering modules to preprocess the collected data. Meanwhile, to maintain the English performance of SeaLLMs, we also introduce a set of high-quality English texts sampled from RedPajama-Data into pre-training.

Pre-training Strategies

We conduct pre-training in multiple stages. Each stage serves a different specific objective and involves dynamic control of (unsupervised and supervised) data mixture, as well as data specification and categorization. We also employ novel sequence construction and masking techniques during these stages. Details are provided in the technical report.

Supervised fine-tuning (SFT) Data

Our supervised fine-tuning (SFT) data consists of many categories. The largest and most dominant of them are public and open-source. As these data are available only in English, we employ several established automatic techniques to gather more instruction data for SEA languages through synthetic means. For a small number of SFT data, we engaged native speakers to vet, verify and modify SFT responses so that they adapt to the local cultural customs, norms, and laws. We also collect country-relevant safety data that cover many culturally and legally sensitive topics in each of these SEA countries - such data tend to be ignored, or may even appear in conflict with Western safety data. Therefore, we believe that our models are more native country-friendly and abide by local rules to a higher degree.

SFT Strategies

We conduct SFT with a relatively balanced mix of SFT data from different categories. We make use of the system prompt during training, as we found it helps induce a prior which conditions the model to a behavioral distribution that focuses on safety and usefulness. Details are provided in the technical report.

Self-preferencing DPO

To save on costs of human preference annotation work, some have sought to use powerful LLMs like GPT-4 to play as a preference data generator. However, that may not even be feasible for low-resource non-Latin languages because of the unfavorable tokenization of ChatGPT as explained above. In other words, even short prompts would exceed their context-length and the API-call costs would explode by up to 17 times.

Therefore, we use our own SeaLLM SFT models to generate preference data using a special prompting strategy, which we later use to employ direct preference optimization (DPO) to significantly improve the models' abilities as an AI agent. As such, our models are free from relying on powerful close-sourced models like GPT-4 to improve the performance on low-resource languages.

Acknowledgement to Our Linguists

We would like to express our special thanks to our professional and native linguists, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.

Citation

If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: l.bing@alibaba-inc.com

@article{damonlpsg2023seallm,
  author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
            Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
            Chaoqun Liu, Hang Zhang, Lidong Bing},
  title = {SeaLLMs - Large Language Models for Southeast Asia},
  year = 2023,
  Eprint = {arXiv:2312.00738},
}