xorazm-text2sql-0.8b

xorazm-text2sql-0.8b (GRPO Checkpoint)

This directory contains the final checkpoint files for xorazm-text2sql-0.8b, a lightweight SQLite text-to-SQL translation model. The model was fine-tuned using Group Relative Policy Optimization (GRPO) reinforcement learning on a unified dataset.

Developer & Organization

• Developer: Shakhriyor Kadamboev
• Email: [email protected]
• Hugging Face Repository: shakha-de/xorazm-text2sql-0.8b

Training Setup

The training pipeline was executed via Unsloth.

Hyperparameters:

• Base Model: Qwen/Qwen3.5-0.8B
• Frameworks: Unsloth (for memory-efficient fast training) & TRL GRPOTrainer
• Loss Type: dr_grpo
• Optimizer: adamw_8bit (8-bit AdamW)
• Learning Rate: 5e-6 with Cosine scheduler
• Warmup Ratio: 0.03
• Weight Decay: 0.1
• Max Prompt Length: 6,140 tokens (filtered from maximum 6,144 context window)
• Max Completion Length: 512 tokens
• Generations per Prompt (num_generations): 8
• Gradient Accumulation Steps: 2

---

Dataset

• Training Dataset: BIRD; SPIDER; SYNSQL-2.5M
• Original Examples: 20,000 for SFT and 16,676 for GRPO (pruned down to 15,566 after filtering prompts exceeding 6,140 tokens using tokenizer.apply_chat_template check)
• Data Sources: Unified blend of:
  • BIRD Bench (Spider / Bird style SQL DDL + sample rows)
  • Spider 1.0
  • synSQL-2.5M

---

Reward Metrics & Weights

Three independent reward signals were utilized:

1. Execution Accuracy Reward (execution_reward_func): Binary check on whether the query successfully executes and matches the output rows of the gold query.
   • Weight: 128.0
2. Format Compliance Reward (format_reward_func): Checks for proper </think>\n ```sql ... ``` tag adherence.
   • Weight: 2.0
3. Syntax Reward (syntax_reward_func): Verifies SQL syntax correctness.
   • Weight: 4.0

---

Compute & Training Statistics

• Hardware: 1x NVIDIA L40S GPU (44.394 GB Max VRAM)
• VRAM Before Training: 1.617 GB reserved
• Epochs: 1