PEFT_TuringQ_llama3_FT

Llama3-8B-ft-TuringQ

Llama3-8B-ft-TuringQ is a fine-tuned version of the meta-llama/Meta-Llama-3-8B-Instruct model on the TuringQ dataset. it is designed to enhance reasoning capabilities in theoretical computer science, particularly in the theory of computation.

Key Features:

• Specialization: Theory of computation and related concepts
• Training Approach: Combination of QLoRA, PEFT, and Supervised Fine-Tuning
• Lightweight Adapter: Easily integrates with the base model for flexible updates.

Usage

• Solving complex problems in theoretical computer science
• Assisting in educational contexts for undergraduate and graduate-level computer science courses

To use Llama3-8B-ft-TuringQ, you can load the model and make predictions using the Hugging Face transformers library. Below is an example of how to do this:

from torch.utils.data import Dataset, DataLoader
from time import time
from datasets import load_dataset
from peft import LoraConfig, PeftModel, prepare_model_for_kbit_training, PeftConfig
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
    pipeline,
)
from trl import SFTTrainer, setup_chat_format
from huggingface_hub import login
from tqdm import tqdm
import pandas as pd
import torch
import os

# Load the base model
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)
base_model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Set up chat format
base_model, tokenizer = setup_chat_format(base_model, tokenizer)

# Load the PEFT adapter
peft_model_id = "PardisSzah/PEFT_TuringQ_llama3_FT"
peft_config = PeftConfig.from_pretrained(peft_model_id)
model = PeftModel.from_pretrained(base_model, peft_model_id)

# Merge the base model and PEFT adapter
model = model.merge_and_unload()

# Set the model to evaluation mode
model.eval()

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.float16,
    device_map="auto",
)

# You can change the parameters to match your specifications
def get_answer(system_message, user_message, temperature=0.8, max_length=1024):
    user_message = "Question: " + user_message + " Answer:"
    messages = [
        {"role": "system", "content": system_message},
        {"role": "user", "content": user_message},
    ]
    prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    terminators = [
        pipeline.tokenizer.eos_token_id,
        pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
    ]
    sequences = pipeline(
        prompt,
        do_sample=True,
        top_p=0.9,
        temperature=temperature,
        eos_token_id=terminators,
        max_new_tokens=max_length,
        return_full_text=False,
        pad_token_id=pipeline.model.config.eos_token_id
    )
    answer = sequences[0]['generated_text']
    return answer 

# Use your own prompt
system_message =
"""
You are an AI assistant designed to answer theory of computation and math questions.
Please provide a step-by-step solution to the problem.
"""

answer = query_model(system_message, question, temperature=0.1, max_length=600)
print(answer)

Question Prealgebra MATH Level 2

If each 1/2 inch on a map represents 20 miles, how many inches apart on the map are two cities whose actual distance apart is 120 miles?

Question Intermediate Algebra MATH Level 5

Let f: R → R be a function satisfying f(x)f(y) = f(x-y) . Find all possible values of f(2017). Enter all the possible values, separated by commas.

Question Computability Theory TuringQ Level 2

INFC = {(M): M is a computer in C and L(M) contains infinitely many strings} True or false with justification: The language INFCFG is Turing decidable.

Limitations

• Specialized for theoretical computer science; may not generalize well to other domains
• Performance may vary depending on the complexity and specificity of the queries

Training Hyperparameters

PEFT Configuration**:

• lora_alpha: 64
• lora_dropout: 0.05
• r: 4
• bias: "none"
• task_type: "CAUSAL_LM"
• target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

Training Arguments**:

• evaluation_strategy: "steps"
• optim: "paged_adamw_8bit"
• per_device_train_batch_size: 4
• gradient_accumulation_steps: 2
• per_device_eval_batch_size: 4
• learning_rate: 5e-6
• eval_steps: 500
• max_steps: 4000
• num_train_epochs: 3
• warmup_steps: 100
• lr_scheduler_type: "cosine"
• weight_decay: 0.01
• fp16: True
• metric_for_best_model: "eval_loss"

Evaluation:

Score Distribution Across Models on the Test Split of the TuringQ Dataset assigned by LLM evaluator:

Human Evaluation of Llama3-8B vs. Llama3-8B-ft on MATH Test: