Downloads · 30 days
11
9% of all-time downloads
weiren119/traditional_chinese_qlora_llama2_13b_merged
traditional_chinese_qlora_llama2_13b_merged is a text generation model from weiren119. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
- Github repo: https://github.com/MIBlue119/traditionalchinesellama2/ - This is a practice to finetune Llama2 on traditional chinese instruction dataset at Llama2 chat model. - Use qlora and the alpaca translated data…
Downloads · 30 days
11
9% of all-time downloads
All-time downloads
124
Public
Parameters
13B
26 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors26 GB · 100%
How the weights are stored.
F1613B · 100%
From the Hugging Face model README
Thanks for these references:
The following bitsandbytes quantization config was used during training:
$pip install transformers torch peft
import transformers
import torch
from transformers import AutoTokenizer, TextStreamer
# Use the same tokenizer from the source model
model_id="weiren119/traditional_chinese_qlora_llama2_13b_merged"
tokenizer = AutoTokenizer.from_pretrained(original_model_path, use_fast=False)
# Load fine-tuned model, you can replace this with your own model
model = AutoPeftModelForCausalLM.from_pretrained(
model_id,
load_in_4bit=model_id.endswith("4bit"),
torch_dtype=torch.float16,
device_map='auto'
)
system_prompt = """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."""
def get_prompt(message: str, chat_history: list[tuple[str, str]]) -> str:
texts = [f'[INST] <<SYS>>\n{system_prompt}\n<</SYS>>\n\n']
for user_input, response in chat_history:
texts.append(f'{user_input.strip()} [/INST] {response.strip()} </s><s> [INST] ')
texts.append(f'{message.strip()} [/INST]')
return ''.join(texts)
print ("="*100)
print ("-"*80)
print ("Have a try!")
s = ''
chat_history = []
while True:
s = input("User: ")
if s != '':
prompt = get_prompt(s, chat_history)
print ('Answer:')
tokens = tokenizer(prompt, return_tensors='pt').input_ids
#generate_ids = model.generate(tokens.cuda(), max_new_tokens=4096, streamer=streamer)
generate_ids = model.generate(input_ids=tokens.cuda(), max_new_tokens=4096, streamer=streamer)
output = tokenizer.decode(generate_ids[0, len(tokens[0]):-1]).strip()
chat_history.append([s, output])
print ('-'*80)