BELLE-LLaMA-EXT-7B

Considering LLaMA's license constraints, the model is for research and learning only. Please strictly respect LLaMA's usage policy. We are not allowed to publish weights for LLaMA, of course, even finetuned, but there is no problem publishing the difference, a patch that we suggest to apply to the files. The encryption is a simple XOR between files, ensuring that only the people that have access to the original weights (from completely legal sources, of course) can transform them into finetuned weights. You can find the decrypt code on https://github.com/LianjiaTech/BELLE/tree/main/models .

Model Card for Model ID

Welcome

If you find this model helpful, please like this model and star us on https://github.com/LianjiaTech/BELLE !

Model description

We release our base model described in the paper Towards Better Instruction Following Language Models for Chinese

We extend original LLaMA's vocabulary for an efficiency tokenization of Chinese. This model is derived through the following steps:

1. Train a tokenizer with a vocabulary of 50K tokens on 12M lines of Chinese text.
2. Merge the trained vocabulary with the original LLaMA vocabulary, resulting in a new vocabulary of 79,458 tokens.
3. Resize word embeddings and further pretrain LLaMA on 3.4B Chinese words with other parameters fixed.

We test the extended tokenizer and the original tokenizer on 5,000 lines of Chinese text, and the average tokens of a line reduces from 733 to 291.

Download, Convert & Check

1. After you git clone this model

md5sum ./*
228a21b7bf927f7ffd44c16c88256684  ./config.json.fb090219f6fed69687ab8f9c902f7802cff8060b08007ca0e5af177a8f9613d5.enc
f9b33d359f17a437f6c24b4de6f2272e  ./generation_config.json.fd7ff399e5568cc21a0a8414f43df88ef7c424995b9b97a90563165d2cf79efd.enc
1c12c5bb95b1d191779ef160624a622a  ./pytorch_model-00001-of-00002.bin.3b0666c50d7fd55d5116e788ec51aa96a34ba6816e86ffbee1dbe983bf511b4b.enc
1a67804dbdfd2168ef30ec077b73e90d  ./pytorch_model-00002-of-00002.bin.763b336a89ef37327716d9c097835720662da656bdc27afde27daec9d0873284.enc
0d6db7f247a51589f3dd6d08dbfe64ce  ./pytorch_model.bin.index.json.4f08b269e18619675bc3fd62f6efb3a8d59f9d54fa50f5625d0bba7adabaf90e.enc
34696bfce7b27548cfc2410e2b55762e  ./special_tokens_map.json.96bdbb8504d9967606e5f661ccc7cbbac44a3661af863a7a58614670a0ccab33.enc
6014cf2235521f974c8d9fb69b6cf07e  ./tokenizer_config.json.7078cc180b3d35e7ccd06b49ede4a7fef85f2572bda40c1fe2fc8f9ab25418d3.enc
56724a79091f3d1877cca65c6412d646  ./tokenizer.model.0b716a618c9e7c45648f91d997431eba3b0ff111b17ce7b777280ed771a49f95.enc

1. Decrypt the files using the scripts in https://github.com/LianjiaTech/BELLE/tree/main/models

You can use the following command in Bash. Please replace "/path/to_encrypted" with the path where you stored your encrypted file, replace "/path/to_original_llama_7B" with the path where you stored your original llama7B file, and replace "/path/to_finetuned_model" with the path where you want to save your final trained model.

mkdir /path/to_finetuned_model
for f in "/path/to_encrypted"/*; \
    do if [ -f "$f" ]; then \
       python3 decrypt.py "$f" "/path/to_original_llama_7B/consolidated.00.pth" "/path/to_finetuned_model/"; \
    fi; \
done

After executing the aforementioned command, you will obtain the following files.

./config.json
./generation_config.json
./pytorch_model-00001-of-00002.bin
./pytorch_model-00002-of-00002.bin
./pytorch_model.bin.index.json
./special_tokens_map.json
./tokenizer_config.json
./tokenizer.model

1. Check md5sum

You can verify the integrity of these files by performing an MD5 checksum to ensure their complete recovery. Here are the MD5 checksums for the relevant files:

md5sum ./*
df363050c4ded5c3136270cef715a7d1  ./config.json
2917a1cafb895cf57e746cfd7696bfe5  ./generation_config.json
a88865ce42f45c0c88cd4f7f8ecd75ea  ./pytorch_model-00001-of-00002.bin
ce23ee57ecc73a78b0117e38a68f8d84  ./pytorch_model-00002-of-00002.bin
e5385004e4876ea6b93d6126e845a82f  ./pytorch_model.bin.index.json
15f7a943faa91a794f38dd81a212cb01  ./special_tokens_map.json
08f6f621dba90b2a23c6f9f7af974621  ./tokenizer_config.json
6ffe559392973a92ea28032add2a8494  ./tokenizer.model

Use model

This model is a pre-trained language model and has not been instruction-tuned. To obtain well instruction-following capabilities, please finetune it with your instruction data.

Limitations

There still exists a few issues in the model trained on current base model and data:

1. The model might generate factual errors when asked to follow instructions related to facts.

2. Occasionally generates harmful responses since the model still struggles to identify potential harmful instructions.

3. Needs improvements on reasoning and coding.

Since the model still has its limitations, we require developers only use the open-sourced code, data, model and any other artifacts generated via this project for research purposes. Commercial use and other potential harmful use cases are not allowed.

Citation

Please cite our paper and github when using our code, data or model.

@misc{ji2023better,
      title={Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation}, 
      author={Yunjie Ji and Yan Gong and Yong Deng and Yiping Peng and Qiang Niu and Baochang Ma and Xiangang Li},
      year={2023},
      eprint={2304.07854},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
@misc{BELLE,
  author = {Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Baochang Ma, Xiangang Li},
  title = {BELLE: Be Everyone's Large Language model Engine},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/LianjiaTech/BELLE}},
}