Taiwan-LLaMa-v1.0-GGUF

<!-- header start --> <!-- header end -->

Taiwan-LLaMa-v1.0 - GGUF

• Model creator: Yen-Ting Lin
• Original model: Language Models for Taiwanese Culture v1.0

Description

This repo contains GGUF format model files for Yen-Ting Lin's Language Models for Taiwanese Culture v1.0.

About GGUF

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

The key benefit of GGUF is that it is a extensible, future-proof format which stores more information about the model as metadata. It also includes significantly improved tokenization code, including for the first time full support for special tokens. This should improve performance, especially with models that use new special tokens and implement custom prompt templates.

As of August 25th, here is a list of clients and libraries that are known to support GGUF:

• llama.cpp.
• text-generation-webui, the most widely used web UI. Supports GGUF with GPU acceleration via the ctransformers backend - llama-cpp-python backend should work soon too.
• KoboldCpp, now supports GGUF as of release 1.41! A powerful GGML web UI, with full GPU accel. Especially good for story telling.
• LM Studio, version 0.2.2 and later support GGUF. A fully featured local GUI with GPU acceleration on both Windows (NVidia and AMD), and macOS.
• LoLLMS Web UI, should now work, choose the c_transformers backend. A great web UI with many interesting features. Supports CUDA GPU acceleration.
• ctransformers, now supports GGUF as of version 0.2.24! A Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
• llama-cpp-python, supports GGUF as of version 0.1.79. A Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
• candle, added GGUF support on August 22nd. Candle is a Rust ML framework with a focus on performance, including GPU support, and ease of use.

Repositories available

• 2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference
• 2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference (deprecated)
• Yen-Ting Lin's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

<!-- footer start --> <!-- footer end -->

Original model card: Yen-Ting Lin's Language Models for Taiwanese Culture v1.0

Language Models for Taiwanese Culture

<p align="center"> ✍️ <a href="https://chat.twllm.com/" target="_blank">Online Demo</a> • 🤗 <a href="https://huggingface.co/yentinglin" target="_blank">HF Repo</a> • 🐦 <a href="https://twitter.com/yentinglin56" target="_blank">Twitter</a> • 📃 <a href="https://arxiv.org/pdf/2305.13711.pdf" target="_blank">[Paper Coming Soon]</a> • 👨️ <a href="https://yentingl.com/" target="_blank">Yen-Ting Lin</a> <br/><br/> <img src="https://www.csie.ntu.edu.tw/~miulab/taiwan-llama/logo-v2.png" width="100"> <br/> <a href="https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE"> <img src="https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg"></a> <a href="https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE"> <img src="https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg"></a> <br/> </p>

Overview

Taiwan-LLaMa is a full parameter fine-tuned model based on LLaMa 2 for Traditional Mandarin applications.

Taiwan-LLaMa v1.0 pretrained on over 5 billion tokens and instruction-tuned on over 490k conversations both in traditional mandarin.

Demo

A live demonstration of the model can be accessed at Hugging Face Spaces.

Key Features

1. Traditional Mandarin Support: The model is fine-tuned to understand and generate text in Traditional Mandarin, making it suitable for Taiwanese culture and related applications.

2. Instruction-Tuned: Further fine-tuned on conversational data to offer context-aware and instruction-following responses.

3. Performance on Vicuna Benchmark: Taiwan-LLaMa's relative performance on Vicuna Benchmark is measured against models like GPT-4 and ChatGPT. It's particularly optimized for Taiwanese culture.

4. Flexible Customization: Advanced options for controlling the model's behavior like system prompt, temperature, top-p, and top-k are available in the demo.

Work in progress

• Improved pretraining: A refined pretraining process (e.g. more data from Taiwan, training strategies) is under development, aiming to enhance model performance for better Taiwanese culture.
• Extend max length: Utilizing the Rope mechanism as described in the paper, the model's length will be extended from 4k to 8k.

Taiwanese Culture Examples

Model

We provide a number of model checkpoints that we trained. Please find them on Hugging Face here. Here are some quick links to the checkpoints that are finetuned from LLaMa 2:

Data

Here are some quick links to the datasets that we used to train the models:

Architecture

Taiwan-LLaMa is based on LLaMa 2, leveraging transformer architecture, <a href="https://github.com/Dao-AILab/flash-attention" target="_blank">flash attention 2</a>, and bfloat16.

It includes:

• Pretraining Phase: Pretrained on a vast corpus of over 5 billion tokens, extracted from common crawl in Traditional Mandarin.
• Fine-tuning Phase: Further instruction-tuned on over 490k multi-turn conversational data to enable more instruction-following and context-aware responses.

Generic Capabilities on Vicuna Benchmark

The data is translated into traditional mandarin for evaluating the general capability.

<img src="./images/zhtw_vicuna_bench_chatgptbaseline.png" width="700">

The scores are calculated with ChatGPT as the baseline, represented as 100%. The other values show the relative performance of different models compared to ChatGPT.

How to deploy the model on my own machine?

We recommend hosting models with 🤗 Text Generation Inference. Please see their license for details on usage and limitations.

bash run_text_generation_inference.sh "yentinglin/Taiwan-LLaMa-v1.0" NUM_GPUS DIR_TO_SAVE_MODEL PORT MAX_INPUT_LEN MODEL_MAX_LEN

Prompt format follows vicuna-v1.1 template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {user} ASSISTANT:

Setup development environment

conda create -n taiwan-llama python=3.10 -y 
conda activate taiwan-llama
pip install -r requirements.txt

Citations

If you use our code, data, or models in your research, please cite this repository. You can use the following BibTeX entry:

@inproceedings{lin-chen-2023-llm,
    title = "{LLM}-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models",
    author = "Lin, Yen-Ting  and Chen, Yun-Nung",
    booktitle = "Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.nlp4convai-1.5",
    pages = "47--58"
}

@misc{taiwanllama,
    author={Lin, Yen-Ting and Chen, Yun-Nung},
    title={Language Models for Taiwanese Culture},
    year={2023},
    url={https://github.com/MiuLab/Taiwan-LLaMa},
    note={Code and models available at https://github.com/MiuLab/Taiwan-LLaMa},
}

Collaborate With Us

If you are interested in contributing to the development of Traditional Mandarin language models, exploring new applications, or leveraging Taiwan-LLaMa for your specific needs, please don't hesitate to contact us. We welcome collaborations from academia, industry, and individual contributors.

License

The code in this project is licensed under the Apache 2.0 License - see the LICENSE file for details.

The models included in this project are licensed under the LLAMA 2 Community License. See the LLAMA2 License for full details.

OpenAI Data Acknowledgment

The data included in this project were generated using OpenAI's models and are subject to OpenAI's Terms of Use. Please review OpenAI's Terms of Use for details on usage and limitations.

Acknowledgements

We thank Meta LLaMA team and Vicuna team for their open-source efforts in democratizing large language models.