llm-jp-3-13b-finetune-2

Overview

This repository demonstrates how to use a pre-trained language model to perform task inference using given input datasets. The code iterates over a dataset, processes each entry through the model, and generates an output for each task.

Features

• Processes input datasets using a pre-trained language model.
• Generates textual outputs based on task-specific inputs.
• Supports customizable parameters for tokenization and generation.

Usage

Dataset Format

The dataset should be a list of dictionaries, where each dictionary contains:

• input: The task instruction as a string.
• task_id: A unique identifier for the task.

Code

from tqdm import tqdm

results = []
for data in tqdm(datasets):

  input = data["input"]

  prompt = f"""### 指示
  {input}
  ### 回答
  """

  tokenized_input = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
  attention_mask = torch.ones_like(tokenized_input)

  with torch.no_grad():
      outputs = model.generate(
          tokenized_input,
          attention_mask=attention_mask,
          max_new_tokens=100,
          do_sample=False,
          repetition_penalty=1.2,
          pad_token_id=tokenizer.eos_token_id
      )[0]
  output = tokenizer.decode(outputs[tokenized_input.size(1):], skip_special_tokens=True)

  results.append({"task_id": data["task_id"], "input": input, "output": output})