sql-llm-wikisql-lora

SQL Assistant – TinyLlama Fine-Tuned on WikiSQL (QLoRA) Model Overview

This model is a schema-aware Text-to-SQL generator built by fine-tuning TinyLlama-1.1B using QLoRA on the WikiSQL dataset.

It converts natural language questions into structured SQL queries given a database schema.

The model has been adapted specifically for:

SQL generation

Schema-conditioned reasoning

Structured query formatting

Reduced hallucination compared to base model

Model Details

Developed by: Ruben S

Model type: Causal Language Model (LoRA fine-tuned adapter)

Base model: TinyLlama-1.1B-Chat-v1.0

Task: Text-to-SQL generation

Language: English

Training dataset: WikiSQL (10,000 samples subset)

Fine-tuning method: QLoRA (4-bit quantization + LoRA adapters)

Epochs: 3

Final training loss: 0.52

Hardware: Google Colab T4 GPU

License: Apache 2.0 (inherits from base model)

This repository contains only the LoRA adapter weights. The base model must be loaded separately.

Intended Use Direct Use

This model is intended for:

Converting natural language queries into SQL

Educational and research use

SQL assistant systems

Demonstrations of parameter-efficient fine-tuning

Example use cases:

"Find employees with salary greater than 50000"

"What is the average price of products in Electronics category?"

Downstream Use

The model can be integrated into:

Database query assistants

Data analytics dashboards

Backend services that translate user questions into SQL

Chat-based data exploration tools

Out-of-Scope Use

This model is not suitable for:

Production-grade database security systems

Financial or safety-critical systems

Complex multi-table join reasoning (not trained on Spider)

SQL injection protection

It was trained on single-table WikiSQL-style queries.

Training Details Training Data

Dataset: WikiSQL (publicly available dataset)

10,000 training samples

1,000 validation samples

Single-table SQL queries

Aggregations: MAX, MIN, COUNT, SUM, AVG

WHERE clause conditions

SQL queries were reconstructed from parsed format into full SQL strings before training.

Training Procedure

Base Model: TinyLlama-1.1B-Chat-v1.0

Quantization: 4-bit (bitsandbytes)

Fine-tuning: LoRA (Parameter Efficient Fine-Tuning)

Trainable parameters: ~1% of total model parameters

Objective: Causal Language Modeling (next-token prediction)

Labels set equal to input_ids

Training Hyperparameters

Epochs: 3

Batch size: 4

Gradient accumulation steps: 4

Learning rate: 1e-4

Precision: FP16

Optimizer: AdamW (default Trainer optimizer)

Evaluation Qualitative Evaluation

The fine-tuned model was compared against the base TinyLlama model.

Improvements observed:

Removal of chat-style explanations

No markdown formatting

Reduced hallucinated table names

Improved aggregation selection (AVG, COUNT, etc.)

Better multi-condition WHERE clauses

Example:

Input:

Find the average price of products in Electronics category.

Output:

SELECT AVG(price) FROM table WHERE category = 'Electronics'

Limitations

Trained only on WikiSQL (single-table queries)

Limited support for JOIN operations

Numeric formatting inconsistencies may occur (e.g., quoting numbers)

Sensitive to schema formatting structure

How to Use from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch

base_model = AutoModelForCausalLM.from_pretrained( "TinyLlama/TinyLlama-1.1B-Chat-v1.0", torch_dtype=torch.float16, device_map="auto" )

model = PeftModel.from_pretrained( base_model, "YOUR_USERNAME/sql-assistant-tinyllama-wikisql-qlora" )

tokenizer = AutoTokenizer.from_pretrained( "TinyLlama/TinyLlama-1.1B-Chat-v1.0" )

prompt = """### Instruction: Convert natural language to SQL using the given schema.

Schema:

Table columns: product, price, category, rating

Question:

Find the average price of products in Electronics category.

SQL:

"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate( **inputs, max_new_tokens=80, temperature=0.1, do_sample=True )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Environmental Impact

Hardware: NVIDIA T4 GPU

Training Time: ~2 hours total

Cloud Provider: Google Colab

Precision: FP16 + 4-bit quantization

Parameter-efficient fine-tuning significantly reduces compute and memory usage compared to full fine-tuning.

Future Work

Extend training to Spider dataset (multi-table joins)

Add execution-based evaluation

Improve numeric formatting consistency

Add schema-aware table naming

Citation

If you use this model, please cite:

TinyLlama-1.1B-Chat-v1.0 WikiSQL Dataset

Contact: [email protected]