llm-jp-3-13b-ft2

llm-jp-3-13b-ft2

This repository provides large language model developed for Matsuo-Lab LLM2024 Final Project by Hiroya Chiba (hiroyachiba).

Usage

!pip install unsloth
!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install -U torch
!pip install -U peft

from datasets import load_dataset
from unsloth import FastLanguageModel
from peft import PeftModel
import torch
import json
import re
from transformers import set_seed
set_seed(0)

base_model_id = "llm-jp/llm-jp-3-13b"
adapter_id = "hiroyachiba/llm-jp-3-13b-ft2"

dtype = None # Noneにしておけば自動で設定
load_in_4bit = True # 今回は13Bモデルを扱うためTrue
base_model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=base_model_id,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)

INSTRUCTION_MESSAGE = '''以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。
タスクを説明する指示とAIが生成した応答を <USER_INPUT> と <AI_OUTPUT> セクションで提供します。'''
tokenizer.chat_template = "{{bos_token}}{% for message in messages %}{% if message['role'] == 'user' %}{{ '\n\n### 指示:\n' + '<USER_INPUT>\n' + message['content'] + '\n</USER_INPUT>' }}{% elif message['role'] == 'system' %}{{ '" + INSTRUCTION_MESSAGE + "' }}{% elif message['role'] == 'assistant' %}{{ '\n\n### 応答:\n' + '<AI_OUTPUT>\n' + message['content'] + '\n</AI_OUTPUT>' + eos_token }}{% endif %}{% if loop.last and add_generation_prompt %}{{ '\n\n### 応答:\n' + '<AI_OUTPUT>\n' }}{% endif %}{% endfor %}"

dataset = load_dataset('json', data_files='data/elyza-tasks-100-TV_0.jsonl', split='train')

model.eval()
FastLanguageModel.for_inference(model)
model.config.use_cache = True
torch.cuda.empty_cache()

def gen_output(input):
  messages = [
    {"role": "system", "content": 'あなたは優秀なアシスタントです。'},
    {"role": "user", "content": input}
  ]
  inputs = tokenizer.apply_chat_template(messages,
                                          return_tensors="pt",
                                          add_generation_prompt=True,
                                          return_dict=True).to(model.device)
  del inputs['token_type_ids']

  with torch.no_grad():
    outputs = model.generate(**inputs,
                            max_new_tokens = 512,
                            pad_token_id=tokenizer.pad_token_id,
                            do_sample=False,
                            repetition_penalty=1.2,
                            )
  output = tokenizer.decode(outputs[0][inputs.input_ids.size(1):], skip_special_tokens=True)
  return re.sub('\n?</?AI_OUTPUT>', '', output)

results = []
for i, data in enumerate(dataset):
  input = data["input"]
  output = gen_output(input)
  results.append({"task_id": i, "input": input, "output": output})

json_file_id = re.sub(".*/", "", adapter_id)
with open(f"/content/{json_file_id}_output.jsonl", 'w', encoding='utf-8') as f:
    for result in results:
        json.dump(result, f, ensure_ascii=False)
        f.write('\n')

Model Details

The model was fine-tuned based on llm-jp/llm-jp-3-13b.

• Model type: Transformer-based Language Model

Datasets

Instruction tuning

The model was fine-tuned on the following datasets.

Evaluation

elyza-tasks-100

Risks and Limitations

The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.

Send Questions to

[email protected]

License

Apache License, Version 2.0

Model Card Authors

Hiroya Chiba