vicuna-baichuan-13b-chat

Vicuna Baichuan-13B Model

基于Baichaun-13B-chat的Vicuna微调对话模型

shibing624/vicuna-baichuan-13b-chat evaluate test data：

The overall performance of model on QA test:

在中文开放测试集中的表现优异，继承了两方面的优势： 1）微调训练的底座是 baichuan-inc/Baichuan-13B-Chat 模型，是较强的中英文底座模型， 2）微调使用的是高质量10万条多语言GPT4对话数据集，微调后的模型在通用对话，包括医疗行业答复能力达到领先水平，在通用问题上的答复能力不弱于Baichuan-13B-Chat。

Training details

training args:

{"per_device_train_batch_size": 2,
"per_device_eval_batch_size": 2,
"gradient_accumulation_steps": 1, 
"learning_rate": 2e-05, 
"num_train_epochs": 3.0,
"max_steps": -1,
"lr_scheduler_type": "linear",
"warmup_ratio": 0.0,
"warmup_steps": 50, 
"save_strategy": "steps",
"save_steps": 500,
"save_total_limit": 13,
"seed": 42,
"bf16": true,
"fp16": false, 
"eval_steps": 50,
"remove_unused_columns": false,
"load_best_model_at_end": true,
"metric_for_best_model": "loss", 
"optim": "paged_adamw_32bit",
"report_to": ["tensorboard"], 
}

• train loss: 1.3209
• evaluate loss: 1.4953

The following bitsandbytes quantization config was used during training:

• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16

Usage

本项目开源在 github repo:

• shibing624/textgen
• shibing624/MedicalGPT

使用textgen库：textgen，可调用Baichuan/LLaMA模型：

Install package:

pip install -U textgen

from textgen import GptModel

def generate_prompt(instruction):
    return f"""A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.USER: {instruction} ASSISTANT: """

model = GptModel("baichuan", "shibing624/vicuna-baichuan-13b-chat")
predict_sentence = generate_prompt("一岁宝宝发烧能吃啥药?")
r = model.predict([predict_sentence])
print(r) # ["1、首先大多数小儿退热药中含有解热镇痛成分阿司匹林或布洛芬等，这类药品虽然副作用较少..."]

Usage (HuggingFace Transformers)

Without textgen, you can use the model like this:

First, you pass your input through the transformer model, then you get the generated sentence.

Install package:

pip install transformers 

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig


model = AutoModelForCausalLM.from_pretrained("shibing624/vicuna-baichuan-13b-chat", device_map='auto', torch_dtype=torch.float16, trust_remote_code=True)
model.generation_config = GenerationConfig.from_pretrained("shibing624/vicuna-baichuan-13b-chat", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("shibing624/vicuna-baichuan-13b-chat", trust_remote_code=True)
device = torch.device(0) if torch.cuda.is_available() else torch.device("cpu")

def generate_prompt(instruction):
    return f"""A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.USER: {instruction} ASSISTANT: """


sents = ['一岁宝宝发烧能吃啥药', "who are you?"]
for s in sents:
    q = generate_prompt(s)
    inputs = tokenizer(q, return_tensors="pt")
    inputs = inputs.to(device)

    generate_ids = model.generate(
        **inputs,
        max_new_tokens=512, 
    )

    output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True)[0]
    print(output)
    print()

output:

一岁宝宝发烧能吃啥药
孩子的身体是很脆弱的，如果不注意保护好自己就会出现各种疾病。有些家长在看到孩子生病后都非常着急，想要立刻给他们用药治疗，但其实并没有那么简单，因为每个人对于药物的敏感度和耐受性都是不同的，所以我们需要根据具体情况来选择合适的方法进行治疗才可以达到最佳效果。1、首先大多数小儿退热药中含有解热镇痛成分阿司匹林或布洛芬等，这类药品虽然副作用较少，但也存在过量使用引起胃肠道损伤甚至消化道溃疡出血的风险；2、还有部分小儿退热药中含有扑尔敏等抗组胺药，它们本身属于处方药，应该严格按照医嘱服用，切勿擅自加减剂量或者延长用药时间;3、此外，某些小儿退热药中含有维生素C及葡萄糖酸钙等营养补充剂，只有当患儿伴随脱水症状（如口渴、尿色深）时才考虑添加，否则容易造成电解质紊乱。

模型文件组成：

vicuna-baichuan-13b-chat
├── config.json
├── configuration_baichuan.py
├── generation_config.json
├── modeling_baichuan.py
├── pytorch_model-00001-of-00003.bin
├── pytorch_model-00002-of-00003.bin
├── pytorch_model-00003-of-00003.bin
├── pytorch_model.bin.index.json
├── quantizer.py
├── special_tokens_map.json
├── tokenization_baichuan.py
├── tokenizer_config.json
└── tokenizer.model

• Inference GPU: 27G

Inference Examples

训练数据集

• 10万条多语言ShareGPT GPT4多轮对话数据集：shibing624/sharegpt_gpt4

如果需要训练ChatGLM/LLAMA/BLOOM模型，请参考shibing624/MedicalGPT

• training script:

CUDA_VISIBLE_DEVICES=0,1,2 torchrun --nproc_per_node 3 supervised_finetuning.py \
    --model_type baichuan \
    --model_name_or_path baichuan-inc/Baichuan-13B-Chat \
    --train_file_dir path_of_sharegpt_gpt4 \
    --validation_file_dir data/finetune/ \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --template_name vicuna \
    --do_train \
    --do_eval \
    --use_peft True --bf16 \
    --max_train_samples -1 \
    --max_source_length 1024 --max_target_length 1024 \
    --max_eval_samples 50 \
    --num_train_epochs 3 \
    --learning_rate 2e-5 \
    --warmup_ratio 0.05 \
    --weight_decay 0.05 \
    --logging_strategy steps \
    --logging_steps 10 \
    --eval_steps 50 \
    --evaluation_strategy steps \
    --save_steps 1000 \
    --save_strategy steps \
    --save_total_limit 13 \
    --preprocessing_num_workers 10 \
    --output_dir outputs-sft--baichuan-v3-production \
    --overwrite_output_dir \
    --ddp_timeout 300000 \
    --logging_first_step True \
    --target_modules all \
    --lora_rank 8 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --torch_dtype bfloat16 --optim paged_adamw_32bit \
    --device_map auto \
    --report_to tensorboard \
    --ddp_find_unused_parameters False \
    --gradient_checkpointing True --qlora True --cache_dir ./cache

• 机器：3*A100(40G), 36hours

Citation

@software{textgen,
  author = {Ming Xu},
  title = {textgen: Implementation of language model finetune},
  year = {2023},
  url = {https://github.com/shibing624/textgen},
}