ocisd4

llama-2-tokenizer-dataprep

ocisd4/llama-2-tokenizer-dataprep

llama-2-tokenizer-dataprep is a machine learning model from ocisd4. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

- 關閉自動添加<s，方便產生megatron-deepspeed訓練用檔案 - 指定pad token為<unk，訓練時token數才會正確，以及finetunet0.py才能正確packsample