tianpeng-lora-30B

TianPeng is a chat model that is fine-tuned using LoRA on top of the LLaMA-30B model. The training corpus includes both English and Chinese language.

TianPeng 是在 LLaMA-30B的基础上使用 LoRA 进行 FineTuning 的聊天模型，语料同时涵盖英文和中文。

Training Parameters

• Base Model: LLaMA-30B
• Training Epoch: 1
• Batch Size: 40
• Maximum Input Length: 2048
• Learning Rate: 5e-5
• LoRA Rank: 8
• Updated Modules: All Linears
• Steps: 30091

Performance

由于费用限制，当前训练仅跑了1个epoch，仍有很大的提升空间。

Demo

更建议直接使用Baize-chatbot/demo作为WebUI进行部署，它适用于所有基于PEFT训练的LLaMA LoRA权重。

# pip install git+https://github.com/huggingface/transformers
 
from peft import PeftModel
from transformers import GenerationConfig, LlamaForCausalLM, LlamaTokenizer
import torch

base_model = "decapoda-research/llama-30b-hf"
adapter_model ="pleisto/tianpeng-lora-30B"
tokenizer = LlamaTokenizer.from_pretrained(base_model)
model = LlamaForCausalLM.from_pretrained(
            base_model,
            load_in_8bit=True,
            torch_dtype=torch.float16,
            device_map="auto",
        )
model = PeftModel.from_pretrained(
            model,
            adapter_model,
            torch_dtype=torch.float16,
        )
model.eval()
prompt = "Hey, are you consciours? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate
generate_ids = model.generate(inputs.input_ids, max_length=30)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
"Hey, are you consciours? Can you talk to me?\nI'm not consciours, but I can talk to you."

Training Dataset

• alpaca.en.json, quora.en.json and stackoverflow.en.json are extracted from BaiZe, released under the GPL-3.0 License, authored by Xu, Canwen and Guo, Daya and Duan, Nan and McAuley, Julian.
• 'guanaco.json' and guanaco-paper-answers.json are extracted from GuanacoDataset
• 'translation2019.json' are extracted from nlp_chinese_corpus
• 'belle1m.cn.json' are extracted from BelleGroup Dataset
• 'unnatural_instruction_gpt4.en.json' are extractd from Instruction Tuning with GPT-4

Notes

本模型仅供学术研究之用，根据 LLaMA 的license 您不应该将本模型用于商业用途。

This model is intended for academic research purposes only. According to the license of LLaMA, you should not use this model for commercial purposes.

Citation

@misc{tianpeng,
  author = {Li, Ding and Xian, Zhang},
  title = {TianPeng: A Chinese chat model that is fine-tuned using LoRA on top of the LLaMA-30B model},
  howpublished = {\url{https://huggingface.co/pleisto/tianpeng-lora-30B}},
  year = {2023},
  doi = { 10.57967/hf/0528 }
}