Bailong-instruct-7B-GGUF

Bailong-instruct-7B-GGUF

• Model creator: 群創光電 智能推進處 Text&AIGC 研發團隊
• Original model: Bailong-instruct-7B

Description

This repo contains GGUF format model files for 群創光電智能推進處 Text&AIGC Bailong-instruct-7B.

<!-- README_GGUF.md-about-gguf start -->

About GGUF

(Copy from Audrey Tang)

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

Here is an incomplete list of clients and libraries that are known to support GGUF:

• llama.cpp. The source project for GGUF. Offers a CLI and a server option.
• text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
• KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
• GPT4All, a free and open source local running GUI, supporting Windows, Linux and macOS with full GPU accel.
• LM Studio, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration. Linux available, in beta as of 27/11/2023.
• LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
• Faraday.dev, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
• llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
• candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
• ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server. Note, as of time of writing (November 27th 2023), ctransformers has not been updated in a long time and does not support many recent models.

<!-- README_GGUF.md-about-gguf end -->

Original model card

license: llama2

Bailong: Bilingual transfer learning based on QLoRA and zip-tie embedding

<div align="center">

•🐉 <a href="https://huggingface.co/INX-TEXT/Bailong-instruct-7B" target="_blank">[Bailong-instruct 7B]</a>•🏛️ <a href="https://huggingface.co/datasets/INX-TEXT/Bailong-bench" target="_blank">[Bailong-bench]</a> • 📃 [Paper coming soon]</a>

</div> <p align="center"> <img width="800" src="https://github.com/blaze7451/Bailong/blob/main/Bailong_pics/Bailong_image.png?raw=true" alt="Bailong Logo">

Model Card for Bailong-instruct 7B

This model card contains the information and the results of our released Bailong (白龍) project. Bailong, which stands for Bilingual trAnsfer learnIng based on qLOra and zip-tie embeddiNG, is our newest project aimed at enhancing the Traditional Chinese performance in open-source large language model (LLM). All the consequences are listed as follows:

• Bailong 7B(not released): Bailong 7B is an autogressive language model with 7B parameters and decoder-only transformer architecture. It is derived from implementing secondary pretraining on Llama 2 7B with tied embedding and expanded vocabulary. The model is trained with context length of 2048 tokens and the training dataset is primarily composed of Traditional Chinese data with a minor portion of English one. Specially, motivated by the Chinese-LLaMA paper, we implemented QLoRA during the secondary pretraining stage to train the model, as opposed to the standard full-parameter training method. This approach significantly reduces the computational cost while achieving satisfactory model performance simultaneously.
• Bailong-instruct 7B: Bailong-instruct 7B is the fine-tuned version of Bailong 7B optimized for multi-turn dialogue use case. Similar to secondary pretraining stage, we use QLoRA to fine-tune the model. To facilitate the development and communication within the research community in Traditional Chinese NLP, we decide to release this model on Hugging Face.
• Bailong-bench: Most existing language models claiming to support Traditional Chinese are adapted from continuously pre-trained open-source models, primarily trained on English data. In certain cases, models fine-tuned with instructions using this approach may respond to Traditional Chinese instructions in English and vice versa. This could pose a significant problem when deploying the model for real-world applications. Consequently, it is essential to have a benchmark dataset specifically designed to assess a model's proficiency in following both English and Traditional Chinese instructions. To address this issue, we propose Bailong-bench, a benchmark dataset crafted not only to evaluate the model's performance in various real-world application scenarios but also to assess its ability to maintain language consistency.
• Technical report: We intend to release a technical report in the future, providing a detailed overview of the Bailong project. Please stay tuned for further updates.

Bailong features

🚀 Fast and efficient tokenizer
We expand the vocabulary size of Llama 2 from 32000 to 59241 by merging original Llama 2's tokenizer with our self-made tokenizer. As shown in the following figure, with the fixed context length, Bailong's tokenizer generates less tokens and tokenize more efficiently compared to Breeze-7B's and Taiwan-LLM's tokenizers while tokenizing the Traditional Chinese sequences. <img width="800" src="https://github.com/blaze7451/Bailong/blob/main/Tokenizer%20performance.png?raw=true" alt="Tokenizers performance">

💦 Aggresive cleaning
In addition to utilizing the conventional fuzzy deduplication, we also implement semantic deduplication such as SemDeDup during our data cleaning processes to improve the quality of the pretraining data.

🏃 Memory efficient training
Further saving more gpu memory than Chinese-LLaMA by using QLoRA during the secondary pretraining and supervised fine-tuning.

💪 Advanced methods for initiallizing embedding layer
We propose zip-tie embedding to initiallize embedding of appended vocabs. The proposed embedding method combined with appropriate learning rate save steps during the training. The detail of the zip-tie embedding can be found in the upcoming technical report.

🙋‍♂️ Advanced instruction tuning method for multi-turn dialogue
We leverage TargetLMLoss and training method proposed in FireFly project to endow model with multi-turn dialogue capability. For basic introduction of this method, please refer to zhihu article.

Applications (Bailong-instruct 7B)

The following tables present, but are not limited to, several possible scenarios for the applications of Bailong-instruct 7B.

<details><summary><b>Medical consultation(保健諮詢)</b></summary> </details> <details><summary><b>Product copywriting generation(文案生成)</b></summary> </details> <details><summary><b>Creative writing(創意寫作)</b></summary> </details> <details><summary><b>Knowledge base QA(知識庫問答)</b></summary> </details> <details><summary><b>Multi-turn dialogue(多輪對話)</b></summary> <img width="800" src="https://github.com/blaze7451/Bailong/blob/main/Multi_turn_dialogue.PNG?raw=true" alt="multi-turn dialogue"> </details> <details><summary><b>Mail assistant(信件助理)</b></summary> </details> <details><summary><b>Summary generation(摘要生成)</b></summary> </details> <details><summary><b>Open QA(開放式問題)</b></summary> </details> <details><summary><b>English instruct(英語指令)</b></summary> </details> <details><summary><b>Proofreading assistant(校對助理)</b></summary> </details> <details><summary><b>Translation(中英語翻譯)</b></summary> </details> <details><summary><b>Morality and ethics (道德與倫理)</b></summary> </details> <details><summary><b>Others (其他)</b></summary> </details>

Comparison between Bailong 7B and Llama 2 7B

All the following results are generated under the same setting (top-p=0.9, top-k=40, temperature=0.6, repetition-penalty=1.0, max-new-tokens=200)

Model Performance

We show the evaluation results of models on Bailong-bench here. For judging, we implement LLM-as-a-judge and prompt GPT-4 to act as judges and assess the quality of the models' responses. Note that in our implementation, the consistency of the language used between the user and the chat assistant has been included in the evaluation criteria. Unless it is a translation task or specifically specified by the user, the language used by the model should be consistent with the language used by the user.

For MT-bench:

Model Usage

Download model

Here is the example for you to download Bailong-instruct 7B with huggingface transformers:

# install the dependencies
# pip install transformers torch accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("INX-TEXT/Bailong-instruct-7B")
model = AutoModelForCausalLM.from_pretrained("INX-TEXT/Bailong-instruct-7B", device_map="auto", torch_dtype=torch.bfloat16)

Prompt format

Following the training method in Firefly project, for single-turn dialogue, the prompt template of Bailong-instruct 7B can be formatted as:

<s>prompt</s>

In the context of multi-turn dialogues, the multi-turn dialogue will be concatenated in the following format.

<s>prompt1</s>answer1</s>prompt2</s>answer2</s>...

With the tokenizer of Bailong-instruct 7B, you can easily generate the prompt format of Bailong-instruct 7B with apply_chat_template:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("INX-TEXT/Bailong-instruct-7B")

messages = [
   {"role": "user", "content": "請生成一句口號，以強調節能減碳的重要性"},
   {"role": "assistant", "content": "「保護地球，從我們做起！節能減碳，守護未來！」"},
   {"role": "user", "content": "此句口號與節能減碳的關聯是什麼？"},
]

res = tokenizer.apply_chat_template(messages, tokenize=False)

print(res)
# <s>請生成一句口號，以強調節能減碳的重要性</s>「保護地球，從我們做起！節能減碳，守護未來！」</s>此句口號與節能減碳的關聯是什麼？</s>

Inference

Here is the quick example to run inference by using pipeline:

from transformers import pipeline
import torch

pipe = pipeline("text-generation", 
                model="INX-TEXT/Bailong-instruct-7B", 
                torch_dtype=torch.bfloat16, 
                device_map="auto", 
                return_full_text=False)

messages = [
   {"role": "user", "content": "請生成一句口號，以強調節能減碳的重要性"},
   {"role": "assistant", "content": "「保護地球，從我們做起！節能減碳，守護未來！」"},
   {"role": "user", "content": "此句口號與節能減碳的關聯是什麼？"},
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
res = pipe(prompt, max_new_tokens=200, do_sample=True, top_k=50, top_p=0.9)[0]["generated_text"]
print(res)

Ethical Considerations and Limitations

• Incorrect facts and information: As with all LLMs, the model may generate incorrect responses to user's prompts. Users should consider the model's output as suggestions or references rather than definitive and sole answers.
• Language limitation: The model is mainly designed to follow Traditional Chinese and English instructions. Therefore, the model may lack the capability to follow instructions composed in other languages including Simplified Chinese, leading to potential misinterpretations or errors in response.
• Potential toxicity, bias, and uncontrollability: Due to the lack of further training through reinforcement learning from human feedback, there's a possibility that model may generate harmful and societal biased responses or deviate from the expected responses. As a result, before deploying any applications of our model, developers should perform safety testing and tuning tailored to their specific applications of the model.

Citation

@article{chen2024bailong,
  title={Bailong: Bilingual transfer learning based on QLoRA and zip-tie embedding},
  author={Chen Lung-Chuan and Li Zong-Ru},
  journal={arXiv},
  year={2024}
}