Breexe-8x7B-Instruct-v0_1-GGUF

Original Model: MediaTek-Research/Breexe-8x7B-Instruct-v0_1

Breexe-8x7B-Instruct-v0_1

Breexe-8x7B is a language model family that builds on top of Mixtral-8x7B, specifically intended for Traditional Chinese use.

Breexe-8x7B-Base is the base model for the Breexe-8x7B series. Breexe-8x7B-Base expands the original vocabulary with additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, Breexe-8x7B operates at twice the inference speed for Traditional Chinese to Mixtral-8x7B. [See Inference Performance.]

Breexe-8x7B-Instruct derives from the base model Breexe-8x7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks, such as Q&A, RAG, multi-round chat, and summarization. Breexe-8x7B-Instruct demonstrates impressive performance in benchmarks for Traditional Chinese and English, on par with OpenAI's gpt-3.5-turbo-1106. [See Chat Model Performance.]

The current release version of Breexe-8x7B is v0.1.

The models were trained on Nvidia's Taipei-1. Special thanks for Nvidia's technical support.

A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.

BreeXe API

<p style="color:red;">We offer a trial API for business integration and academic benchmarking.</p>

API service open time: 14:00 - 20:00 (from 2024/4/3 to 2024/5/3)

The API is in beta testing. If you are experiencing connectivity issues, please bear with us.

Free trial API key: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoyLCJ1c2VybmFtZSI6ImdlbmVyYWxfcHVibGljIiwibm90ZSI6ImdlbmVyYWwgcHVibGljIn0.kCp68nRw3RSh3jbMm8FvhG0NIkStflgI1wTHLviRPQE'

Get started here:

import time

from openai import OpenAI

API_KEY = <API_KEY_HERE>
BASE_URL = 'https://api-mtkresearch.com/v1'
MODEL_NAME = 'BreeXe-8x7B'

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)


def ask_breexe(messages):    
    completion = client.chat.completions.create(
        model=MODEL_NAME,
        messages=messages,
        temperature=0.01,
        top_p=0.01,
        max_tokens=512
    )
    response = completion.choices[0].message.content
    time.sleep(3)  # due to a rate limit of 200 requests per 10 minutes
    return response

messages = [
    {
        'role': 'system',
        'content': 'You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.'
    },
    {
        'role': 'user',
        'content': '請問在量子物理領域中，疊加原理是什麼？它又是如何與量子糾纏現象相關聯的？'
    }
]

response = ask_breexe(messages)
print(response)

BreeXe Demo

<a href="https://huggingface.co/spaces/MediaTek-Research/Demo-MR-Breexe-8x7B" style="color:red;font-weight:bold;">Try Demo Here 👩‍💻🧑🏻‍💻</a>

Some examples are shown below. (System Prompt: You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan., Temperature: 0.01, Top-P: 0.01, Repetition-Penalty: 1.1)

<details><summary><b>Education</b></summary> </details> <details><summary><b>Open Question</b></summary> </details> <details><summary><b>Email Writing</b></summary> </details> <details><summary><b>Table Understanding</b></summary> </details> <details><summary><b>Language Assistance</b></summary> </details> <details><summary><b>Summary</b></summary> </details> <details><summary><b>Roleplay</b></summary> </details> <details><summary><b>Extraction</b></summary> </details> <details><summary><b>Coding</b></summary> </details> <details><summary><b>Math</b></summary> </details> <details><summary><b>Taiwan Knowledge</b></summary> </details>

Features

• Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
• 8k-token context length
• Multi-turn dialogue (without special handling for harmfulness)
• Sparse mixture of experts (MoE)

Inference Performance

In this test, we use the first 700 characters of the web article as the input and ask the model to write the same article again. All inferences run on 4 RTX A6000 GPUs (using vllm, with a tensor-parallel size of 4).

Chat Model Performance

TMMLU+, Table, and MT-Bench-tw source from MediaTek-Research/TCEval-v2, which derives from TCEval-v1 and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train. MT-Bench source from lmsys/mt_bench_human_judgments. We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, Table, and MMLU. All choice problems adapt the selection by the log-likelihood. We use the code revised from fastchat llm_judge (GPT4 as judge) to evaluate MT-Bench-tw and MT-Bench.

* Taiwan-LLM models responds to multi-turn questions (English) in Traditional Chinese.

Base Model Performance

TMMLU+ and Table source from MediaTek-Research/TCEval-v2, which derives from TCEval-v1 and ikala/tmmluplus. MMLU sources from hails/mmlu_no_train. We use the code revised from EleutherAI/lm-evaluation-harness to evaluate TMMLU+, Table, and MMLU. All choice problems adapt the selection by the log-likelihood.

Use in Transformers

First install direct dependencies:

pip install transformers torch accelerate

If you want faster inference using flash-attention2, you need to install these dependencies:

pip install packaging ninja
pip install flash-attn

Then load the model in transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "MediaTek-Research/Breexe-8x7B-Instruct-v0_1",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2" # optional
)

The structure of the query is

<s> SYS_PROMPT  [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST] 

where SYS_PROMPT, QUERY1, RESPONSE1, and QUERY2 can be provided by the user.

The suggested default SYS_PROMPT is

You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.

We also integrate chat_template into tokenizer_config.json, so you can apply_chat_template to get the prompt.

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("MediaTek-Research/Breexe-8x7B-Instruct-v0_1")
>>> chat = [
...   {"role": "user", "content": "你好，請問你可以完成什麼任務？"},
...   {"role": "assistant", "content": "你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。"},
...   {"role": "user", "content": "太棒了！"},
... ]
>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.   [INST] 你好，請問你可以完成什麼任務？ [/INST] 你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。 [INST] 太棒了！ [/INST] "
# Tokenized results
# ['▁', '你好', '，', '請問', '你', '可以', '完成', '什麼', '任務', '？']
# ['▁', '你好', '，', '我', '可以', '幫助', '您', '解決', '各種', '問題', '、', '提供', '資訊', '和', '協助', '您', '完成', '許多', '不同', '的', '任務', '。', '例如', '：', '回答', '技術', '問題', '、', '提供', '建議', '、', '翻譯', '文字', '、', '尋找', '資料', '或', '協助', '您', '安排', '行程', '等', '。', '請', '告訴', '我', '如何', '能', '幫助', '您', '。']
# ['▁', '太', '棒', '了', '！']

Citation

@article{breexe8x7b2024,
  title={},
  author={},
  journal={arXiv},
  year={2024}
}