llm-jp-3-13b_ft_nf4

llm-jp-3-13b_ft_nf4

This repository provides a model based on llm-jp-3-13b with SFT using ichikara-instruction.

Usage

import os, torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

HF_TOKEN = os.environ.get('HF_LLM_API_KEY')

base_model_id = "llm-jp/llm-jp-3-13b"
adapter_id = "5h11Vj0/llm-jp-3-13b_ft_nf4"

bnb_config = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    token = HF_TOKEN
)
model = PeftModel.from_pretrained(model, adapter_id, token=HF_TOKEN)
tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True, token=HF_TOKEN)

input_text = "連濁とはなんですか？"
prompt = f"""### 指示
{input_text}
### 回答
"""

tokenized_input = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
attention_mask = torch.ones_like(tokenized_input)

with torch.no_grad():
    outputs = model.generate(
    tokenized_input,
    attention_mask=attention_mask,
    max_new_tokens=100,
    do_sample=False,
    repetition_penalty=1.2,
    pad_token_id=tokenizer.eos_token_id
    )[0]
output = tokenizer.decode(outputs[tokenized_input.size(1):], skip_special_tokens=True)
print(output)

Training Data

Pre-training

See llm-jp-3-13b.

Instruction tuning

Use ichikara-instruction.