yuyan-10b

YuYan-10b

YuYan is a series of natural language processing models developed by Fuxi AI lab, Netease.Inc, including text generation models, natural language understanding models, and more. YuYan-10b is a natural language understanding model trained on high-quality Chinese corpus.

YuYan-10b is similar to BERT in that it is trained on large-scale pre-training corpora using unsupervised learning. However, it differs in that it incorporates various tasks such as sentence order and word deletion in addition to the MLM task during training to enhance the model's semantic representation ability and improve its understanding of Chinese.

CLUE result

How to use

Our model is trained based on the Megatron. As a result, the inference and finetuning depend on it.

Below are the install tutorial. We have packaged all the required dependencies for the model. Use the following command to obtain the model running environment.

pip install -r requirements.txt

Finetuning script

We provide multiple scripts for finetuning on the CLUE benchmark, which is a Chinese language understanding evaluation leaderboard that covers various tasks such as natural language understanding, reading comprehension, and semantic understanding. For any given CLUE task, use the following command to start finetuning.

# finetuning afqmc task
sh finetune_afqmc_distributed.sh

# finetuning csl task
sh finetune_csl_distributed.sh