question-to-tagged-question-llama3-lora

Model Overview

This model is a fine-tuned version of llama3 using the QueryBridge dataset. We utilized Low-Rank Adaptation (LoRA) to train it for tagging question components using the tags in the table below. The demo video shows how the mapped question appears and, after converting it to a graph representation, how we visualized it as shown in the video.

The tagged questions in the QueryBridge dataset are designed to train language models to understand the components and structure of a question effectively. By annotating questions with specific tags such as <qt>, <p>, <o>, and <s>, we provide a detailed breakdown of each question's elements, which aids the model in grasping the roles of different components.

<a href="https://youtu.be/J_N-6m8fHz0"> <img src="https://cdn-uploads.huggingface.co/production/uploads/664adb4a691370727c200af0/sDfp7DiYrGKvH58KdXOIY.png" alt="Training Model with Tagged Questions" width="400" height="300" /> </a>

Tags Used in Tagged Questions

How to use the model?

There are two main steps

1- Download the model from Huggingface

To use the model, you can run it with TorchTune commands. I have provided the necessary Python code to automate the process. Follow these steps to get started:

• Download the fintuned version including the meta_model_0.pt file and the tokenizer. (see the files and versions tap in this page).
• Save the model file in the following directory: /home/USERNAME/Meta-Llama-3-8B/

2- Using the model

<details> <summary>Steps</summary>

• Note: Replace each USERNAME with your username.

Step 1: Create a Configuration File

First, save a file named custom_generation_config_bigModel.yaml in /home/USERNAME/ with the following content:

# Config for running the InferenceRecipe in generate.py to generate output from an LLM

# Model arguments
model:
  _component_: torchtune.models.llama3.llama3_8b

checkpointer:
  _component_: torchtune.utils.FullModelMetaCheckpointer
  checkpoint_dir: /home/USERNAME/Meta-Llama-3-8B/
  checkpoint_files: [
    meta_model_0.pt
  ]
  output_dir: /home/USERNAME/Meta-Llama-3-8B/
  model_type: LLAMA3

device: cuda
dtype: bf16

seed: 1234

# Tokenizer arguments
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  path: /home/USERNAME/Meta-Llama-3-8B/original/tokenizer.model

# Generation arguments; defaults taken from gpt-fast
prompt: "### Instruction: \nYou are a powerful model trained to convert questions to tagged questions. Use the tags as follows: \n<qt> to surround question keywords like 'What', 'Who', 'Which', 'How many', 'Return' or any word that represents requests. \n<o> to surround entities as an object like person name, place name, etc. It must be a noun or a noun phrase. \n<s> to surround entities as a subject like person name, place name, etc. The difference between <s> and <o>, <s> only appear in yes/no questions as in the training data you saw before. \n<cc> to surround coordinating conjunctions that connect two or more phrases like 'and', 'or', 'nor', etc. \n<p> to surround predicates that may be an entity attribute or a relationship between two entities. It can be a verb phrase or a noun phrase. The question must contain at least one predicate. \n<off> for offset in questions asking for the second, third, etc. For example, the question 'What is the second largest country?', <off> will be located as follows. 'What is the <off>second</off> largest country?' \n<t> to surround entity types like person, place, etc. \n<op> to surround operators that compare quantities or values, like 'greater than', 'more than', etc. \n<ref> to indicate a reference within the question that requires a cycle to refer back to an entity (e.g., 'Who is the CEO of a company founded by himself?' where 'himself' would be tagged as <ref>himself</ref>). \nInput: Which films directed by a dirctor died in 2014 and starring both Julia Roberts and Richard Gere?\nResponse:"
max_new_tokens: 100
temperature: 0.6
top_k: 1

quantizer: null

Step 2: Set Up the Environment

Create a virtual environment:

/home/USERNAME/myenv

Install TorchTune with:

pip install torchtune

Step 3: Create the Python File

Next, create a Python file called command.py with the following content:

import subprocess
import os
import re
import shlex  # For safely handling command line arguments

def _create_config_file(question):
    # Path to the template and output config file
    template_path = "/home/USERNAME/custom_generation_config_bigModel.yaml"
    output_path = "/tmp/dynamic_generation.yaml"
    
    # Load the template from the file
    with open(template_path, 'r') as file:
        config_template = file.read()

    # Replace the placeholder in the template with the actual question
    updated_prompt = config_template.replace("Input: Which films directed by a dirctor died in 2014 and starring both Julia Roberts and Richard Gere?", f"Input: {question}")
    maxLen = int(1.3*len(question))
    print(f"maxLen: {maxLen}")
    updated_prompt = updated_prompt.replace("max_new_tokens: 100", f"max_new_tokens: {maxLen}")

    # Write the updated configuration to a new file
    with open(output_path, 'w') as file:
        file.write(updated_prompt)
    
    print(f"Configuration file created at: {output_path}")

def get_tagged_question(question):
    # Define the path to the virtual environment's activation script
    activate_env = "/home/USERNAME/myenv/bin/activate"

    # Create configuration file with the question
    _create_config_file(question)

    print('get_tagged_question')
    
    # Command to run within the virtual environment
    command = f"tune run generate --config /tmp/dynamic_generation.yaml"
    
    # Full command to activate the environment and run your command
    full_command = f"source {activate_env} && {command}"
    
    # Run the full command in a shell
    try:
        result = subprocess.run(full_command, shell=True, check=True, text=True, capture_output=True, executable="/bin/bash")
        print("Command output:", result.stdout)
        print("Command error output:", result.stderr)

        output = result.stdout + result.stderr
        # Extract the input and response using modified regular expressions
        input_match = re.search(r'Input: (.*?)(?=Response:)', output, re.S)
        response_match = re.search(r'Response: (.*)', output)

        response_match = response_match.group(1).strip()

        if input_match and response_match:
            print("Input Question: ", question)
            print("Extracted Response: ", response_match)
        else:
            print("Input or Response not found in the output.")
        
    except subprocess.CalledProcessError as e:
        print("An error occurred:", e.stderr)
    return response_match

if __name__ == "__main__":
    # Call the function with a sample question
    get_tagged_question("Who is the president of largest country in Africa?")

Step 4: Run the Script

To run the script and generate tagged questions, execute the following command in your terminal:

python command.py

</details>

How We Fine-Tuned the Model

We fine-tuned the Meta-Llama-3-8B model by two key steps: preparing the dataset and executing the fine-tuning process.

1- Prepare the Dataset

For this fine-tuning, we utilized the QueryBridge dataset, specifically the pairs of questions and their corresponding tagged questions. However, before we can use this dataset, it is necessary to convert the data into instruct prompts suitable for fine-tuning the model. You can find these prompts at this link. Download the prompts and save them in the directory: /home/YOUR_USERNAME/data

2- Fine-Tune the Model

To fine-tune the Meta-Llama-3-8B model, we leveraged Torchtune. Follow these steps to complete the process:

<details> <summary>Steps</summary>

Step 1: Download the Model

Begin by downloading the model with the following command. Replace <ACCESS TOKEN> with your actual Huggingface token and adjust the output directory as needed:

tune download \
  meta-llama/Meta-Llama-3-8B \
  --output-dir /home/YOUR_USERNAME/Meta-Llama-3-8B \
  --hf-token <ACCESS TOKEN>

Step 2: Prepare the Configuration File

Next, you need to set up a configuration file. Start by downloading the default configuration:

tune cp llama3/8B_lora_single_device custom_config.yaml

Then, open custom_config.yaml and update it as follows:

# Config for single device LoRA finetuning in lora_finetune_single_device.py
# using a Llama3 8B model
#
# Ensure the model is downloaded using the following command before launching:
#   tune download meta-llama/Meta-Llama-3-8B --output-dir /tmp/Meta-Llama-3-8B --hf-token <HF_TOKEN>
#
# To launch on a single device, run this command from the root directory:
#   tune run lora_finetune_single_device --config llama3/8B_lora_single_device
#
# You can add specific overrides through the command line. For example,
# to override the checkpointer directory, use:
#   tune run lora_finetune_single_device --config llama3/8B_lora_single_device checkpointer.checkpoint_dir=<YOUR_CHECKPOINT_DIR>
#
# This config is for training on a single device.

# Model Arguments
model:
  _component_: torchtune.models.llama3.lora_llama3_8b
  lora_attn_modules: ['q_proj', 'v_proj']
  apply_lora_to_mlp: False
  apply_lora_to_output: False
  lora_rank: 8
  lora_alpha: 16

# Tokenizer
tokenizer:
  _component_: torchtune.models.llama3.llama3_tokenizer
  path: /home/YOUR_USERNAME/Meta-Llama-3-8B/original/tokenizer.model

checkpointer:
  _component_: torchtune.utils.FullModelMetaCheckpointer
  checkpoint_dir: /home/YOUR_USERNAME/Meta-Llama-3-8B/original/
  checkpoint_files: [
    consolidated.00.pth
  ]
  recipe_checkpoint: null
  output_dir: /home/YOUR_USERNAME/Meta-Llama-3-8B/
  model_type: LLAMA3
resume_from_checkpoint: False

# Dataset and Sampler
dataset:
  _component_: torchtune.datasets.instruct_dataset
  split: train
  source: /home/YOUR_USERNAME/data
  template: AlpacaInstructTemplate
  train_on_input: False
seed: null
shuffle: True
batch_size: 1

# Optimizer and Scheduler
optimizer:
  _component_: torch.optim.AdamW
  weight_decay: 0.01
  lr: 3e-4
lr_scheduler:
  _component_: torchtune.modules.get_cosine_schedule_with_warmup
  num_warmup_steps: 100

loss:
  _component_: torch.nn.CrossEntropyLoss

# Training
epochs: 1
max_steps_per_epoch: null
gradient_accumulation_steps: 64
compile: False

# Logging
output_dir: /home/YOUR_USERNAME/lora_finetune_output
metric_logger:
  _component_: torchtune.utils.metric_logging.DiskLogger
  log_dir: ${output_dir}
log_every_n_steps: null

# Environment
device: cuda
dtype: bf16
enable_activation_checkpointing: True

# Profiler (disabled)
profiler:
  _component_: torchtune.utils.profiler
  enabled: False

Step 3: Run the Finetuning Process

After configuring the file, you can start the finetuning process with the following command:

tune run lora_finetune_single_device --config /home/YOUR_USERNAME/.../custom_config.yaml

The new model can be found in /home/YOUR_USERNAME/Meta-Llama-3-8B/ directory.

</details>