llama-lora-predictive-modeling

Model Card

<!-- Provide a quick summary of what the model is/does. -->

Introduction

This model is tuned to generate predictive model code workflows. The base model (Llama-3.2-1B-Instruct) is fine-tuned with LoRA adapters. The adapted model outpeformed the base model on various benchmarks, including a custom benchmark evaluating the text similarity of the generated response to the desired response.

The goal of this model is to help everyone from non-experts to professional data scientists write predictive model workflows from a simple prompt. Functions like this do exist such as GoogleAutoML and H20.ai; however, many of these solutions are “black-box” and do not iteratively let the user conversationally comment on and update the model to fit their needs. This model will help users write their own code workflows and refine the code to their specifications.

Training Data

The data used to train this model is available in the repo. The data was synthetically generated using Llama-3.3-70B-Instruct. It contains predictive code workflows across various domains (Healthcare, Finance, etc.) and modeling tasks (regression, classification, etc.).

The data was shuffled using sample() from the Pandas library with 100% of the dataset and random_state=42. The first 8 rows were kept for eight-shot prompting, rows 8-299 were used for training, and rows 300-375 were used for testing.

The data can be imported like this:

import pandas as pd

data = pd.read_csv("synthetic_code_workflows_combined.csv")

data = data[['instruction', 'workflow_code']].rename(
    columns={'instruction': 'Instruction', 'workflow_code': 'Response'}
)

data = data.sample(frac=1, random_state=42)

few_shot = data[:8]
train_raw = data[8:300]
val_raw = data[300:]

Training Method

The model was fine-tuned with LoRA adapters. LoRA performed the best in terms of generalizability and reliability as compared to prompt tuning and model editing techniques (ROME, WISE, and EMMET) when separetely testing the base model with the GSM8K dataset on the GSM8K_COT benchmark.

The LoRA model configuration was:

• r: 64
• lora_alpha: 64
• lora_dropout: .03

The LoRA model was trained with a step strategy and early stopping with the following parameters:

• num_train_epochs: 10
• per_device_train_batch_size: 4
• learning_rate: 0.00005
• early_stopping_patience: 2

Evaluation

The LoRA Adapted Model was trained with the base model of Llama-3.2-1B-Instruct. The Custom Text Match metric was created using SequenceMatcher to return a ratio between 0 and 1 that represents how close the generated response is to the desired reponse. The responses were normalized before comparison (i.e., remove comments, doc strings, whitespace, etc.). Additionally, the GSM8K COT benchmark was used to test the models' mathematical COT reasoning, the HellaSwag benchmark was used to test the models' commonsense reasoning, and HumanEval was used to test the models' code generation capabilities.

The LoRA adapted model and base model are compared along with two similar sized models. The performances of Llama-3.2-1B without instruction-tuning along with a slightly larger instruction-tuned Qwen model are also compared. The results of these models are shown below:

The LoRA adapted model improved on the custom text match metrics and HellaSwag and HumanEval benchmarks as compared to the base model. However, the larger Qwen model outperforms each Llama model highlighting that accuracies can increase if willing to use larger models that increase runtime.

Usage and Intended Uses

This model is intended to write predictive code workflows. It has been fine-tuned using Python examples so it is most effective in that language. The model can be loaded using the following code:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained('SamKnisely/llama-lora-predictive-modeling')
model = AutoModelForCausalLM.from_pretrained('SamKnisely/llama-lora-predictive-modeling', device_map="auto", torch_dtype=torch.bfloat16)

Prompt Format

The model is most effective using eight-shot prompting and then prompting the model. The training dataset is available in the repo and the first eight rows were used for few-shot prompting.

The model pipeline should be created after intializing the model earlier:

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device_map="auto",
    max_new_tokens = 1500,
    torch_dtype=torch.bfloat16
)

Next, a few-shot prompting function can be created:

import pandas as pd
from tqdm import tqdm

def few_shot_prompt(instruction, df, n=8):
    """
    Builds a few-shot prompt using `n` demo examples from df (starting at index 2).
    Adds a clear separator for the model to continue from.
    """
    # Build demonstrations
    demos = []
    for i in range(n):
        demo_instr = data.iloc[i]['Instruction']
        demo_resp = data.iloc[i]['Response']
        demos.append(f"Instruction: {demo_instr}\nResponse: {demo_resp}")
    demo_text = "\n\n".join(demos)
    
    # Hide demos
    hidden_context = f"[BEGIN HIDDEN CONTEXT]\n{demo_text}\n[END HIDDEN CONTEXT]\n\n"
    
    # Build final prompt for the current instruction
    prompt = (
        hidden_context +
        "Below is your instruction. Provide only your answer after '### Answer:' without including the hidden context above.\n\n"
        f"Instruction: {instruction}\n### Answer:"
    )
    return prompt

Finally, the model can be prompted. An example prompt and code are provided below:

instruction = "Train a classification model to predict building_stability in Construction based on material_used, building_age, structural_integrity, and weather_resistance."

# Build prompt using 8-shot examples
prompt = few_shot_prompt(instruction, few_shot, n=8)

# Generate a response from the model pipeline
result = pipe(prompt)
generated_text = result[0]['generated_text']

Expected Output

The output is expected to be written after "###Answer:". The output can be extracted using the following code:

# Post-process: Extract the answer after the marker
if "### Answer:" in generated_text:
    answer = generated_text.split("### Answer:")[-1].strip()
else:
    answer = generated_text.strip()

print(answer)

The output from printing answer should look something like this:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# Load data
df = pd.read_csv("construction_data.csv")

# Preprocessing
X = df[['material_used', 'building_age','structural_integrity', 'weather_resistance']]
y = df['building_stability']

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# One-hot encoding for categorical variables
X_train = pd.get_dummies(X_train, columns=['material_used'])
X_test = pd.get_dummies(X_test, columns=['material_used'])

# Align the training and testing data
X_test = X_test.reindex(columns=X_train.columns, fill_value=0)

# Initialize and train the model
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Make predictions on the test set
y_pred = model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print("Model Accuracy:", accuracy)
print("Classification Report:\n", classification_report(y_test, y_pred))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))

Limitations

The model does not always produce a full response as shown above in the example above. Sometimes, the model may only write the first few lines of code or final model diagnostics. However, generally the model does produce the desired results. Larger models with more parameters may perform more consistently at writing predictive models.