Taiwan-LLaMa-v1.0-4bits-GPTQ

<!-- header start --> <!-- header end -->

Taiwan-LLaMa-v1.0-GPTQ

• Model creator: Yen-Ting Lin
• Original model: Language Models for Taiwanese Culture v1.0

Description

This repo contains GPTQ format model files for Yen-Ting Lin's Language Models for Taiwanese Culture v1.0.

Intro

• The 4bits-GQTQ model was converted from Taiwan-LLaMa-v1.0 13b by the package auto-gptq

How to use gptq model pyhton code

• Install gptq package: pip install auto-gptq
• Here is the example code

from transformers import AutoTokenizer,TextStreamer,TextIteratorStreamer
from auto_gptq import AutoGPTQForCausalLM


class TaiwanLLaMaGPTQ:
    def __init__(self, model_dir):
        self.tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=True)
        self.model = AutoGPTQForCausalLM.from_quantized(model_dir,
            trust_remote_code=True,
            use_safetensors=True,
            device_map="auto",
            use_triton=False,
            strict=False)
        self.chat_history = []
        self.system_prompt = """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

            If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."""

        self.streamer = TextStreamer(self.tokenizer, skip_prompt=True, skip_special_tokens=True)
        self.thread_streamer = TextIteratorStreamer(self.tokenizer, skip_special_tokens=True)
    def get_prompt(self, message: str, chat_history: list[tuple[str, str]]) -> str:
        texts = [f'[INST] <<SYS>>\n{self.system_prompt}\n<</SYS>>\n\n']
        for user_input, response in chat_history:
            texts.append(f'{user_input.strip()} [/INST] {response.strip()} </s><s> [INST] ')
        texts.append(f'{message.strip()} [/INST]')
        return ''.join(texts)

    def generate(self, message: str):
        prompt = self.get_prompt(message, self.chat_history)
        tokens = self.tokenizer(prompt, return_tensors='pt').input_ids
        generate_ids = self.model.generate(input_ids=tokens.cuda(), max_new_tokens=4096, streamer=self.streamer)
        output = self.tokenizer.decode(generate_ids[0, len(tokens[0]):-1]).strip()
        self.chat_history.append([message, output])
        return output
    
    def thread_generate(self, message:str):
        from threading import Thread
        prompt = self.get_prompt(message, self.chat_history)
        inputs = self.tokenizer(prompt, return_tensors="pt")

        generation_kwargs = dict(
            inputs=inputs.input_ids.cuda(),
            attention_mask=inputs.attention_mask,
            temperature=0.1,
            max_new_tokens=1024,
            streamer=self.thread_streamer,
        )

        # Run generation on separate thread to enable response streaming.
        thread = Thread(target=self.model.generate, kwargs=generation_kwargs)
        thread.start()
        for new_text in self.thread_streamer:
            yield new_text

        thread.join()

inferencer = TaiwanLLaMaGPTQ("weiren119/Taiwan-LLaMa-v1.0-4bits-GPTQ")


s = ''
while True:
    s = input("User: ")
    if s != '':
        print ('Answer:')
        print (inferencer.generate(s))
        print ('-'*80)

<!-- footer start --> <!-- footer end -->

Original model card: Yen-Ting Lin's Language Models for Taiwanese Culture v1.0

Language Models for Taiwanese Culture

<p align="center"> ✍️ <a href="https://huggingface.co/spaces/yentinglin/Taiwan-LLaMa2" target="_blank">Online Demo</a> • 🤗 <a href="https://huggingface.co/yentinglin" target="_blank">HF Repo</a> • 🐦 <a href="https://twitter.com/yentinglin56" target="_blank">Twitter</a> • 📃 <a href="https://arxiv.org/pdf/2305.13711.pdf" target="_blank">[Paper Coming Soon]</a> • 👨️ <a href="https://yentingl.com/" target="_blank">Yen-Ting Lin</a> <br/><br/> <img src="https://www.csie.ntu.edu.tw/~miulab/taiwan-llama/logo-v2.png" width="100"> <br/> <a href="https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE"> <img src="https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg"></a> <a href="https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE"> <img src="https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg"></a> <br/> </p>

Overview

Taiwan-LLaMa is a full parameter fine-tuned model based on LLaMa 2 for Traditional Chinese applications.

Taiwan-LLaMa v1.0 pretrained on over 5 billion tokens and instruction-tuned on over 490k conversations both in traditional chinese.

Demo

A live demonstration of the model can be accessed at Hugging Face Spaces.

Key Features

1. Traditional Chinese Support: The model is fine-tuned to understand and generate text in Traditional Chinese, making it suitable for Taiwanese culture and related applications.

2. Instruction-Tuned: Further fine-tuned on conversational data to offer context-aware and instruction-following responses.

3. Performance on Vicuna Benchmark: Taiwan-LLaMa's relative performance on Vicuna Benchmark is measured against models like GPT-4 and ChatGPT. It's particularly optimized for Taiwanese culture.

4. Flexible Customization: Advanced options for controlling the model's behavior like system prompt, temperature, top-p, and top-k are available in the demo.

Work in progress

• Improved pretraining: A refined pretraining process (e.g. more data from Taiwan, training strategies) is under development, aiming to enhance model performance for better Taiwanese culture.
• Extend max length: Utilizing the Rope mechanism as described in the paper, the model's length will be extended from 4k to 8k.

Taiwanese Culture Examples

Model

We provide a number of model checkpoints that we trained. Please find them on Hugging Face here. Here are some quick links to the checkpoints that are finetuned from LLaMa 2:

Data

Here are some quick links to the datasets that we used to train the models:

Architecture

Taiwan-LLaMa is based on LLaMa 2, leveraging transformer architecture, <a href="https://github.com/Dao-AILab/flash-attention" target="_blank">flash attention 2</a>, and bfloat16.

It includes:

• Pretraining Phase: Pretrained on a vast corpus of over 5 billion tokens, extracted from common crawl in Traditional Chinese.
• Fine-tuning Phase: Further instruction-tuned on over 490k multi-turn conversational data to enable more instruction-following and context-aware responses.

Generic Capabilities on Vicuna Benchmark

The data is translated into traditional Chinese for evaluating the general capability.

<img src="./images/zhtw_vicuna_bench_chatgptbaseline.png" width="700">

The scores are calculated with ChatGPT as the baseline, represented as 100%. The other values show the relative performance of different models compared to ChatGPT.

How to deploy the model on my own machine?

We recommend hosting models with 🤗 Text Generation Inference. Please see their license for details on usage and limitations.

bash run_text_generation_inference.sh "yentinglin/Taiwan-LLaMa" NUM_GPUS DIR_TO_SAVE_MODEL PORT MAX_INPUT_LEN MODEL_MAX_LEN

Prompt format follows vicuna-v1.1 template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {user} ASSISTANT:

Setup development environment

conda create -n taiwan-llama python=3.10 -y 
conda activate taiwan-llama
pip install -r requirements.txt

Citations

If you use our code, data, or models in your research, please cite this repository. You can use the following BibTeX entry:

@inproceedings{lin-chen-2023-llm,
    title = "{LLM}-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models",
    author = "Lin, Yen-Ting  and Chen, Yun-Nung",
    booktitle = "Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.nlp4convai-1.5",
    pages = "47--58"
}

@misc{taiwanllama,
    author={Lin, Yen-Ting and Chen, Yun-Nung},
    title={Taiwanese-Aligned Language Models based on Meta-Llama2},
    year={2023},
    url={https://github.com/adamlin120/Taiwan-LLaMa},
    note={Code and models available at https://github.com/adamlin120/Taiwan-LLaMa},
}

Collaborate With Us

If you are interested in contributing to the development of Traditional Chinese language models, exploring new applications, or leveraging Taiwan-LLaMa for your specific needs, please don't hesitate to contact us. We welcome collaborations from academia, industry, and individual contributors.

License

The code in this project is licensed under the Apache 2.0 License - see the LICENSE file for details.

The models included in this project are licensed under the LLAMA 2 Community License. See the LLAMA2 License for full details.

OpenAI Data Acknowledgment

The data included in this project were generated using OpenAI's models and are subject to OpenAI's Terms of Use. Please review OpenAI's Terms of Use for details on usage and limitations.

Acknowledgements

We thank Meta LLaMA team and Vicuna team for their open-source efforts in democratizing large language models.