schema-aware-text-to-sql-codet5p-770m

Schema-Aware Text-to-SQL CodeT5+ 770M

This repository contains a schema-aware Text-to-SQL encoder-decoder model based on Salesforce/codet5p-770m.

The model converts a natural-language question and a serialized relational database schema into one read-only SQLite query.

It was fine-tuned using PEFT and LoRA on Spider 1.0 together with a small set of curated rule-based and synthetic portfolio examples. The final LoRA adapter was merged into the base model so that the model can be loaded directly with Hugging Face Transformers without requiring PEFT during inference.

Model summary

Final evaluation results

Validation results

The selected model was evaluated on 628 validation examples.

Held-out test results

The final held-out evaluation used 1,040 examples.

Portfolio quality gate

The final project quality gate required:

The model passed all required portfolio-readiness checks.

Dataset

The complete training corpus contained 8,070 examples.

The corpus included:

• Spider 1.0 examples;
• curated rule-based examples;
• synthetic portfolio examples.

The evaluation process used database-aware splitting and leakage checks.

The completed leakage audit found:

• zero exact-record leakage;
• zero question leakage;
• no detected overlap between the training and held-out evaluation records.

Spider database files and private database files are not redistributed with this model repository.

Input format

The model expects one prompt containing:

• a task instruction;
• the database schema;
• table names;
• column names and data types;
• primary-key indicators;
• foreign-key relationships;
• the natural-language question;
• read-only SQL generation rules.

Example input:

Task:
Generate one valid SQLite query for the given business question.

Database schema:
Table: sales

Columns:
- sale_id INTEGER PRIMARY KEY
- region TEXT
- sale_date TEXT
- sales_amount REAL

Question:
What is the total sales amount for each region?

Rules:
- Use only tables and columns present in the schema.
- Generate exactly one read-only SELECT or WITH query.
- Return SQL only.

Output format

The expected output is SQL text without an explanation:

SELECT region,
       SUM(sales_amount) AS total_sales
FROM sales
GROUP BY region;

Ablation study

The final ablation study was performed on the complete 1,040-example held-out test set.

The experiment demonstrates that both domain fine-tuning and structured schema conditioning were necessary for reliable Text-to-SQL generation.

Repair-layer interpretation

The conservative repair condition produced the same results as the condition without repair:

• repair attempt rate: 0.00%;
• repair success rate: 0.00%;
• execution-accuracy improvement: 0 percentage points.

The repair layer should therefore be described as a conservative SQL formatting and safety mechanism rather than as an accuracy-improvement component.

No repair-related performance gain is claimed for this experiment.

Basic usage

from __future__ import annotations

import torch
from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer,
)

MODEL_ID = (
    "anmol-unitmole/"
    "schema-aware-text-to-sql-codet5p-770m"
)

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    use_fast=True,
)

dtype = (
    torch.bfloat16
    if torch.cuda.is_available()
    and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    MODEL_ID,
    torch_dtype=dtype,
)

device = torch.device(
    "cuda"
    if torch.cuda.is_available()
    else "cpu"
)

model = model.to(device)
model.eval()

prompt = """
Task:
Generate one valid SQLite query for the given business question.

Database schema:
Table: sales

Columns:
- sale_id INTEGER PRIMARY KEY
- region TEXT
- sale_date TEXT
- sales_amount REAL

Question:
What is the total sales amount for each region?

Rules:
- Use only tables and columns present in the schema.
- Generate exactly one read-only SELECT or WITH query.
- Return SQL only.
""".strip()

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    truncation=True,
    max_length=768,
)

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}

with torch.no_grad():
    generated = model.generate(
        **inputs,
        max_new_tokens=256,
        num_beams=4,
        do_sample=False,
        early_stopping=True,
    )

sql = tokenizer.decode(
    generated[0],
    skip_special_tokens=True,
)

print(sql)

CPU usage

The model can be loaded on CPU, but inference will be significantly slower:

import torch
from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer,
)

MODEL_ID = (
    "anmol-unitmole/"
    "schema-aware-text-to-sql-codet5p-770m"
)

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float32,
)

model = model.to("cpu")
model.eval()

For interactive inference, a CUDA-capable GPU is recommended.

SQL validation and execution safety

The model itself generates text and does not independently guarantee safe SQL.

The accompanying project applies a separate SQL safety layer that includes:

• destructive-keyword rejection;
• read-only query enforcement;
• single-statement enforcement;
• SQL comment rejection;
• schema-aware table validation;
• schema-aware column validation;
• handling of quoted SQLite string literals;
• read-only SQLite execution;
• query timeouts;
• output-row limits;
• structured execution-error reporting.

Generated SQL should always be validated before execution.

Intended uses

This model is intended for:

• educational demonstrations;
• machine-learning portfolio projects;
• Text-to-SQL research;
• schema-aware generation experiments;
• public and synthetic SQLite databases;
• human-reviewed analytics assistance;
• encoder-decoder model demonstrations;
• LoRA fine-tuning demonstrations.

Out-of-scope uses

The model is not intended for:

• autonomous execution on production databases;
• unrestricted access to private databases;
• destructive SQL operations;
• unsupervised business-critical analytics;
• financial decision automation;
• medical decision automation;
• legal decision automation;
• compliance or safety-critical applications;
• database administration.

Limitations

The model can still generate SQL that is:

• syntactically invalid;
• valid but semantically incorrect;
• based on an incorrect table;
• based on an incorrect join relationship;
• missing a required filter;
• using an incorrect aggregation;
• using an incorrect grouping condition;
• using an incorrect ordering or limit;
• incompatible with SQL dialects other than SQLite.

Complex multi-table joins, correlated subqueries, nested aggregations and semantically ambiguous questions remain challenging.

Exact-match accuracy is lower than execution accuracy because multiple SQL queries can be textually different while returning equivalent results.

Human review is required before using generated SQL for real decisions.

Evaluation notes

Execution accuracy compares the shape and normalized returned values of the generated and reference queries.

Output aliases and SQLite-generated column labels are not required to match when the returned values are equivalent.

The SQL validator masks quoted string literals before applying schema-aware identifier checks. This prevents values such as "JetBlue Airways" or "Presentation" from being incorrectly classified as column names.

Training configuration

The selected experiment used approximately the following configuration:

Training environment

Reproducibility

The complete project includes:

• dataset preparation;
• corpus construction;
• database-aware data splitting;
• leakage auditing;
• schema serialization;
• prompt construction;
• LoRA training;
• full fine-tuning comparison;
• candidate-model evaluation;
• execution-accuracy measurement;
• SQL safety validation;
• ablation studies;
• error analysis;
• reporting;
• model merging;
• deployment preparation.

Source repository:

https://github.com/unit-mole/encoder-decoder-projects

Project directory:

01-schema-aware-text-to-sql-encoder-decoder

Model selection

Five final candidates were evaluated:

1. CodeT5+ 770M LoRA rank 32;
2. CodeT5-base full fine-tuning;
3. CodeT5-base LoRA rank 64;
4. CodeT5-base LoRA rank 32;
5. FLAN-T5-base LoRA rank 32.

Candidate ranking used:

1. execution accuracy;
2. valid-SQL rate;
3. exact match;
4. average latency.

CodeT5+ 770M LoRA rank 32 was selected as the final model.

Base-model attribution

This model is derived from:

Salesforce/codet5p-770m

The base model and this merged derivative use the BSD 3-Clause license.

Users should also review the original base-model documentation and comply with all applicable dataset, model and software licenses.

Citation

A formal research-paper citation is not currently associated with this portfolio model.

When referencing the implementation, cite the GitHub repository and this Hugging Face model page.

Disclaimer

This model is provided for research, educational and portfolio-demonstration purposes.

The model authors do not guarantee the correctness, completeness, safety or business suitability of generated SQL. Users are responsible for validating queries and protecting all connected databases.