Qwen3.5-4B-neo

<div align="center">

Qwen3.5-4B-neo

unsloth/Qwen3.5-4B fine-tuned for Text-to-SQL generation via LoRA SFT

</div>

---

Overview

This model was fine-tuned from unsloth/Qwen3.5-4B on the Shumatsurontek/neo-sql-reasoning-combined dataset using Supervised Fine-Tuning (SFT) with LoRA adapters (merged into base weights for easy deployment).

• Developed by: Shumatsurontek
• Fine-tuning pipeline: neo-deep-agent-lab
• Engine: Unsloth
• Compute: NVIDIA L40S on Modal serverless GPUs
• Precision: bf16
• License: Apache 2.0

Training Details

Configuration

Hyperparameters

$$ \begin{aligned} \eta &= 2 \times 10^{-4} & \text{(learning rate, cosine schedule)} \ r &= 16 & \text{(LoRA rank)} \ \alpha &= 32 & \text{(LoRA scaling, } \alpha/r = 2 \text{)} \ B &= 4 & \text{(batch size per GPU)} \ T &= 2048 & \text{(max sequence length)} \ E &= 2 & \text{(epochs)} \ \end{aligned} $$

Results

Dataset

Shumatsurontek/neo-sql-reasoning-combined

Each sample follows a 3-turn chat format:

System: You are a SQL expert. Given a database schema and a
        natural language question, generate the correct SQL query.
User:   Schema: CREATE TABLE orders (id INT, total DECIMAL);
        Question: What is the total revenue?
Assistant: SELECT SUM(total) FROM orders;

Quickstart

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Shumatsurontek/Qwen3.5-4B-neo")
tokenizer = AutoTokenizer.from_pretrained("Shumatsurontek/Qwen3.5-4B-neo")

messages = [
    {"role": "system", "content": "You are a SQL expert. Given a database schema and a natural language question, generate the correct SQL query."},
    {"role": "user", "content": "Schema: CREATE TABLE orders (id INT, user_id INT, total DECIMAL);\nQuestion: Total revenue per user?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

vLLM (OpenAI-compatible server)

vllm serve Shumatsurontek/Qwen3.5-4B-neo --trust-remote-code

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Shumatsurontek/Qwen3.5-4B-neo",
  "messages": [
    {"role": "system", "content": "You are a SQL expert."},
    {"role": "user", "content": "Schema: CREATE TABLE users (id INT, name TEXT);\nQuestion: List all users?"}
  ]
}'

Intended Use

This model is designed for text-to-SQL tasks: given a database schema and a natural-language question, it generates the corresponding SQL query. Best suited for analytical and read-only queries.

Out of scope: DDL/DML generation (CREATE, DROP, INSERT, UPDATE, DELETE), multi-database queries, or production use without human review of generated SQL.

Benchmark Results

Evaluated against baseline unsloth/Qwen3.5-4B using lm-eval-harness on NVIDIA L40S.

Evaluated on 50 samples per task.

Citation

@misc{Qwen3_5_4B_neo,
  title  = {Qwen3.5-4B-neo},
  author = {Shumatsurontek},
  year   = {2026},
  url    = {https://huggingface.co/Shumatsurontek/Qwen3.5-4B-neo}
}

License

Apache 2.0