yuren-13b-chatml

Yuren-13B （羽人13B）

Github

Yuren 13B is a large-scale language model that has been continuously trained based on Llama 2 13B. Focused on the field of information synthesis and built upon the data-centric work of Pleisto, this model achieves state-of-the-art levels in data synthesis scenarios such as information extraction in multiple languages, natural language generation of SQL, and structured data output, all with an equivalent parameter count.

羽人 13B 是在 Llama 2 13B 基础上进行持续训练的大语言模型，聚焦于信息合成领域并建立在 Pleisto 以数据为中心的工作上。该模型在以中英文为主的多种语言的信息抽取、自然语言生成 SQL、结构化数据输出等数据合成类场景下实现了同等参数量下的 SOTA 水平。

Quick Start/快速开始

from transformers import LlamaTokenizer, LlamaForCausalLM
import torch

device = torch.device("cuda")
model = LlamaForCausalLM.from_pretrained(
    "pleisto/yuren-13b-chatml", torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = LlamaTokenizer.from_pretrained("pleisto/yuren-13b-chatml", use_fast=False)
system_prompt = "You are an AI model capable of translating natural language queries into SQL statements. Based on the following table schema and the subsequent user query, generate the appropriate SQL statement.\nTable schema: CREATE TABLE table_name_86 (name VARCHAR, score VARCHAR, song_type VARCHAR)\nScoreTypeEnum: [\"folk\",\"rock\",\"other\"]"
query = "8分以上的民谣有哪些？"
inputs = f"<|im_start|>system\n{system_prompt}<|im_end|>\n<|im_start|>user\n{query}<|im_end|>\n<|im_start|>assistant\n"
input_ids = tokenizer(inputs, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(
    input_ids,
    max_new_tokens=4096,
    do_sample=True,
    top_p=1.0,
    temperature=0.32,
    eos_token_id=36845,
)
output = tokenizer.batch_decode(generate_ids)[0]
print(output)
# <s> <|im_start|> system
# You are an AI model capable of translating natural language queries into SQL statements. Based on the following table schema and the subsequent user query, generate the appropriate SQL statement.
# Table schema: CREATE TABLE table_name_86 (name VARCHAR, score VARCHAR, song_type VARCHAR)
# ScoreTypeEnum: ["folk","rock","other"] <|im_end|> <|im_start|> user
# 8分以上的民谣有哪些？ <|im_end|> <|im_start|> assistant
# SELECT name FROM table_name_86 WHERE score > 8 AND song_type = "folk" <|im_end|>

Example Dialogue/示例对话

Text2SQL/自然语言转SQL查询

Structured Data Extraction/结构化数据抽取

ReAct

Other/其他场景

Details of model training/模型训练细节

Extending vocabulary/词表扩充

In the original Llama vocabulary, only a few hundred Chinese characters were included, and the remaining Chinese characters had to be generated by concatenating multiple Unicode bytes. This issue not only obviously affects the Chinese inference performance (generation speed), but also significantly creates a performance bottleneck in Chinese semantic understanding.

We conducted a series of comparative experiments on different vocabulary expansion approaches and found the following:

• Compared to the prevailing strategy of adding a large number of commonly used Chinese character words to the vocabulary, simply adding Chinese character characters to the vocabulary can achieve better semantic understanding performance with a smaller scale of pretraining data. In our experiments, we found that the existing BPE-based tokenizers for Chinese word segmentation inevitably lead to token segmentation that is difficult to align with the true semantics due to the inherent ambiguity in word segmentation. Although increasing the model's parameter size and diversifying the training data can enable the model to have the ability to correctly understand incorrectly segmented tokens during the pretraining process, this understanding always comes with additional costs.

• The number of newly added tokens during vocabulary expansion is directly proportional to the perturbation of the original token distribution, so the fewer new tokens added, the less impact it will have on the semantic disturbance of existing tokens.

Therefore, considering these factors, we conservatively expanded the vocabulary by 4843 tokens. Specifically, this includes all the primary Chinese characters and a subset of secondary and tertiary Chinese characters from the "General Standard Chinese Character Table" published by the National Language Commission in 2013. This subset was derived by using Pleisto's proprietary Chinese corpus to calculate the frequency of commonly used Chinese characters, with the aim of covering as many commonly used Chinese characters as possible, including those in the fields of science and technology, as well as commonly used Chinese characters in personal and place names. Additionally, a portion of commonly used punctuation marks in Chinese language were also included.

原始 Llama 词表中仅含有几百个汉字，其余汉字均需要以多个 unicode 字节形式拼接生成。这一问题除了显而易见地导致中文推理性能（生成速度）受到影响之外，还在很大程度上造成了模型在中文语义理解上造成了性能瓶颈。

我们对于不同的词表扩充方案进行了一系列对比实验并发现：

• 相较于目前主流的在词表中加入大量的常用汉字词语的策略而言，仅在词表中添加汉字字符的方案可以在更少的预训练数据规模下实现更佳的语义理解性能。我们在实验中发现，现有的基于 BPE 的分词器进行中文分词时由于分词本身存在的歧义性几乎必然导致生成的 Token 分割难以与真实语义进行对齐。尽管通过提升模型参数量、增加训练数据的规模和多样性，可以让模型本身在预训练过程中拥有正确理解被错误分割的 token 的能力，但这种理解始终是有额外成本的。

• 扩充词表时新增的 token 的数量和对于原始词向量的分布的扰动始终成正比，因此新增的 token 越少对于已有 token 的语义扰动的影响就会越少。

鉴于此我们较为保守地扩充了 4843 个 Token，具体而言包括国家语委在 2013 年发布的《通用规范汉字表》中的全部一级汉字、二三级汉字的一个子集（该子集通过使用 Pleisto 自有的中文语料进行汉字常用字字频统计后得出以期最大可能地覆盖包括科学技术领域常用字、人名地名常用字在内的所有常用汉字）、汉语中较常使用的一部分标点符号。

Training an embedding layer/词向量嵌入层的训练

Although the mainstream approach to extending the vocabulary typically does not involve separate pretraining of the embedding layer due to cost considerations, and instead relies on updating the embedding layer during continuous pretraining to achieve alignment, our research has revealed the following:

• Completely initializing the newly added word embeddings randomly and then achieving semantic alignment during the continuous pretraining phase can result in the model struggling to effectively learn a portion of the pretraining data. This issue becomes particularly evident when the pretraining data consists of carefully curated high-quality datasets.

• Freezing the other layers and training only the embedding layer using diverse datasets, especially those containing multilingual parallel corpora, different from the ones used in continuous pretraining, helps enhance the model's semantic understanding capabilities. This approach proves beneficial, particularly when the pretraining phase employs small-scale high-quality datasets. Pretraining the embedding layer with a more diverse and unfiltered dataset before continuous pretraining improves the model's semantic understanding and resilience.

Therefore, considering these findings, we conducted one epoch of pretraining on the embedding layer using a diverse corpus of 760 million tokens while keeping the other layers frozen. The training was performed with a global batch size of 128, and the training loss decreased from 5.907 before training to 3.429.

尽管目前主流的词表扩充方案基于成本考虑通常不再单独针对词向量嵌入层进行预先训练，而是依靠在进行持续预训练时对于词向量嵌入层的更新来实现词向量的对齐。但我们的研究发现：

• 完全将新增词向量进行随机初始化，而后在持续预训练阶段进行语义对齐的方案会导致一部分预训练数据难以被模型真正学到，该问题在预训练数据是经过精心清洗的高质量数据集的情况下尤其明显。

• 冻结其他层，使用不同于持续预训练阶段的多样性数据集（尤其是包含多语平行语料的数据集）仅针对词向量嵌入层进行训练有助于提升模型的语义理解能力。尤其是当预训练阶段使用高质量小规模数据集的情况下，使用更具多样性的、未经人工过滤的数据集预先训练词向量嵌入层有助于提升模型的语义理解能力和抗毒性能力。

鉴于此我们在冻结其他层的情况下，使用了 760M Token 的多样性语料进行了 1 个 Epoch 的词向量嵌入层预训练。训练中使用了 128 的全局 Btach Size，train/loss 从训练前的 5.907 降低到 3.429。

Pre Traning/预训练

We performed continuous pretraining on the model using a high-quality corpus of 2.45 billion tokens. The English portion of the corpus was derived from a carefully curated diverse subset of the Falcon RefinedWeb dataset, while the code data came from a specific subset of the bigcode/the-stack dataset. The Chinese portion of the corpus consisted of a specific subset of the mc4 dataset, a curated subset of Chinese Wikipedia, and Pleisto's proprietary collection of public books and papers. During the data preprocessing stage, we employed a series of heuristic methods to clean and deduplicate the data, and used an in-house proprietary model to score the quality of the corpus and align the diversity distribution.

Furthermore, our experiments revealed that the order of the training data significantly impacts the final model performance. Therefore, we sorted the training data using a heuristic algorithm based on the principle of "easy first, difficult later," rather than employing a random shuffling strategy.

During continuous pretraining, we used a sequence length of 4096 and a global batch size of 128. We utilized the 32-bit Lion optimizer with a constant learning rate of 7x10-5. Due to hardware resource limitations, we conducted the training using Lora with a rank of 64. In addition to training all the linear layers, we also trained the embed_token and lm_head layers. However, unlike QLora, we used fp16 precision for training.

我们仅使用了 2.45B 的高质量语料对模型进行了持续预训练，其中英文部分语料来自Falcon RefinedWeb 数据集的一个经过精心策划的多样性子集、代码语料来自 bigcode/the-stack 数据集的一个特定子集，中文部分语料则由 mc4 的一个特定子集、中文维基百科精选子集和 Pleisto 自有的公版书籍与论文数据集共同组成。在数据预处理阶段，我们采用了一系列启发式的方法来针对数据进行清洗和去重并使用自有的闭源模型对于语料质量进行打分评估和多样性分布对齐。

此外我们的实验发现训练数据的顺序也会对最终模型性能造成明显影响，因此我们以「先易后难」的原则使用启发式算法对于训练数据进行了排序而没有采用随机洗牌的策略。

我们在 4096 的序列长度下，使用 128 的全局 Btach Size 进行了持续预训练并使用了 32bit Lion 优化器和7x10-5 的恒定学习率。由于硬件资源上的限制，该阶段的训练采用了 Lora 进行，rank 为 64，除了全部的线性层外还额外训练了 embed_token 和 lm_head 层 。但不同于 QLora，我们采用了 fp16 精度进行训练。

Supervise Fine-tuning/有监督微调

We conducted supervised fine-tuning in two stages. In the first stage, we trained the model for one epoch using a more diverse dataset consisting of 1.8 million samples. A significant portion of this dataset was constructed from a subset of the Orca-style instruction dataset, which is based on flan2021 and COIG (inspired by the Microsoft Orca paper). Additionally, it included subsets from the following publicly available datasets:

• GSM-8k
• OpenAssistant/oasst1
• b-mc2/sql-create-context
• flan2021
• niv0
• COIG
• TheoremQA

In the second stage, we performed an additional two epochs of training using a highly curated subset of 500,000 samples that underwent multiple verification steps.

我们的有监督微调分 2 个阶段进行，首先使用了一个更具多样性的含有 180 万条数据的数据集训练了 1 个 epoch。该数据集的很大一部分是由基于 flan2021 和 COIG 的一个子集所构建的 Orca 风格指令数据集组成（受微软 Orca 论文的启发）。此外还涵盖了如下的公开数据集的子集：

• GSM-8k
• OpenAssistant/oasst1
• b-mc2/sql-create-context
• flan2021
• niv0
• COIG
• TheoremQA

在第二阶段我们使用了一个由 50 万条经过多重校验的高质量子集进行了额外的 2 个 epoch 的训练。

Benchmark Evaluation/性能评测

GSM8k

AGIEval-English

C-Eval 中文能力

Limitations and Biases/局限性

YuRen 13B model is primarily designed for the field of information synthesis, including building intelligent agents, natural language understanding, generating SQL, and other business scenarios, rather than directly providing services to the public. We strongly recommend applying this model to internal data processing scenarios within enterprises, rather than public environments.

While we have made every effort to ensure the compliance of the data used during the model training process, unforeseen issues may arise due to the complexity of the model and data. We disclaim any responsibility for any problems caused by the use of the YuRen 13B open-source model, including but not limited to data security issues, public opinion risks, or any risks and issues arising from the model being misled, abused, disseminated, or improperly utilized.

We strongly advise implementing additional security measures when using the model, such as filtering, reviewing, or restricting the inputs and outputs of the model, to prevent harm to users. Technological development should take place in a regulated and lawful environment, and we hope that all users uphold this principle. We will continue to improve the training and use of the model to enhance its security and effectiveness.

羽人 13B 模型主要设计用于信息合成领域，包括构建智能代理、自然语言理解、生成SQL等业务场景，而并非直接用于向公众提供服务。我们强烈建议将此模型应用于企业内部的数据处理场景，而不是公开环境。

虽然我们已经尽可能确保模型训练过程中的数据合规性，但由于模型和数据的复杂性，可能存在无法预见的问题。如果由于使用羽人 13B 开源模型而导致的任何问题，包括但不限于数据安全问题、公共舆论风险，或模型被误导、滥用、传播或不当利用所带来的任何风险和问题，我们不承担任何责任。

我们强烈建议在使用模型时采用额外的安全措施，如对模型的输入输出进行过滤、审查或限制，以免对用户造成伤害。科技的发展应在规范和合法的环境下进行，我们希望所有使用者都能秉持这一原则。我们将持续改进模型的训练和使用，以提升其安全性和有效性。

Open LLM Leaderboard Evaluation Results

Detailed results can be found here