insightsql-qwen3-8b-lora

InsightSQL — Qwen3-8B Fine-tuned for Text-to-SQL

LoRA adapter fine-tuned on top of unsloth/Qwen3-8B-Base for natural-language to SQL translation, trained on the BIRD-SQL benchmark.

This is Phase 1 of the InsightSQL project — an Agentic Business Intelligence System that combines a fine-tuned model, schema RAG, and a self-correcting agent loop.

Results

Evaluation on BIRD Mini-Dev (50 samples, string-match accuracy):

Note: This is string-match accuracy, a proxy metric. The official BIRD-SQL metric is Execution Accuracy (runs both predicted and gold SQL against real databases and compares result sets). String match underestimates true execution accuracy by ~10-20pp.

Training Details

• Base model: unsloth/Qwen3-8B-Base
• Method: QLoRA (4-bit quantized base + LoRA adapter in FP16)
• LoRA config: r=16, alpha=16, target_modules=all-linear (q, k, v, o, gate, up, down)
• Dataset: birdsql/bird23-train-filtered (~6.6k examples)
• Hardware: Kaggle Tesla T4 (16GB VRAM)
• Framework: Unsloth + TRL SFTTrainer
• Epochs: 1
• Effective batch size: 8 (per_device=1 × grad_accum=8)
• Learning rate: 2e-4, linear schedule with 3% warmup
• Optimizer: AdamW 8-bit
• Training time: 81.3 minutes
• Final training loss: 0.6981

Prompt Format

⚠️ Critical: This adapter expects the exact prompt format below. Using a different format will significantly degrade performance. Database schema: {schema} Hint: {hint or "(no hint)"} Question: {question} SQL: {generated_sql_here}

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model + adapter
base = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3-8B-Base",
    load_in_4bit=True,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "LineLuLan/insightsql-qwen3-8b-lora")
tokenizer = AutoTokenizer.from_pretrained("LineLuLan/insightsql-qwen3-8b-lora")

# Format prompt
prompt = '''### Database schema:
CREATE TABLE customers (id INT, name TEXT, total_spend REAL);

### Hint:
(no hint)

### Question:
List the top 3 customers by total spend.

### SQL:
'''

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Limitations

• Trained on English BIRD-SQL only. Performance on Vietnamese questions may be limited; future work would include localized augmentation.
• String-match eval is loose. Real Execution Accuracy may differ.
• Schema RAG required for production. This adapter alone doesn't handle database schemas with hundreds of tables — use it with a retrieval layer (see Phase 2 of the project).
• No reasoning capability preserved. Trained from base model, not instruct, so does not handle multi-turn chat well.

Citation

@misc{insightsql2026,
  author       = { LineLuLan },
  title        = {InsightSQL: Agentic Business Intelligence with Fine-tuned LLMs},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/LineLuLan/insightsql-qwen3-8b-lora}}
}

Acknowledgements

• Base model by Alibaba Qwen team
• Training framework by Unsloth
• Dataset by BIRD-SQL benchmark