cap-sft-qwen3-lora-30k

CAP-SFT-Qwen3-LoRA-30K

Model Description

This is the optimal 30,000-sample supervised fine-tuned (SFT) version of Qwen3-14B trained on legal reasoning tasks from the Caselaw Access Project (CAP). The model uses LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning and represents the sweet spot for legal reasoning performance.

Performance

Overall Accuracy: 81.3% (67/80 samples across 5 legal tasks) - Peak Performance

Why This Model is Optimal

⭐ Proven Best Performance: This 30K model significantly outperforms both smaller (10K: 70.0%) and larger (75K: 76.2%) training runs, making it the optimal choice for downstream GRPO training.

Scaling Analysis:

• Base Model: 18.8% accuracy
• 10K SFT: 70.0% accuracy (+51.2%)
• 30K SFT: 81.3% accuracy (+11.3%) ← Optimal
• 75K SFT: 76.2% accuracy (-5.1%) ← Overfitted

Training Details

• Base Model: Qwen/Qwen3-14B
• Training Method: LoRA (Low-Rank Adaptation)
• Training Samples: 30,000 legal reasoning examples
• Training Duration: 53 minutes (468 steps)
• Final Loss: 1.347
• Trainable Parameters: 64.2M / 14.8B (0.43%)
• Hardware: 2x NVIDIA H100-80GB GPUs
• Effective Batch Size: 64

Performance Highlights

🎯 Perfect Tasks: Achieves 100% accuracy on Bluebook citation formatting and IRAC case summarization

📈 Optimal Training Point: Represents the peak of the performance curve before overfitting sets in

🚀 GRPO Ready: This model serves as the optimal base for Group Relative Policy Optimization training

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kylebrussell/cap-sft-qwen3-lora-30k")
model = AutoModelForCausalLM.from_pretrained("kylebrussell/cap-sft-qwen3-lora-30k")

# Generate legal text
prompt = "Complete this citation: Brown v. Board of Education, "
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Dataset

Trained on kylebrussell/cap-rlvr-sft - a multi-task legal reasoning dataset derived from the Caselaw Access Project.

Comparison with Other Models

Recommended Use Cases

✅ Recommended For:

• GRPO/RLHF training base model
• Legal reasoning applications
• Citation formatting tasks
• Case summarization

❌ Not Recommended:

• Use 75K model instead - this 30K model is superior
• Further SFT training - performance plateaus/degrades

License

Apache 2.0

Citation

@model{cap-sft-qwen3-lora-30k,
  title={CAP-SFT-Qwen3-LoRA-30K: Optimal Legal Reasoning with Supervised Fine-Tuning},
  author={CAP RLVR Project},
  year={2025},
  url={https://huggingface.co/kylebrussell/cap-sft-qwen3-lora-30k}
}

Next Steps

This model is currently being used as the base for GRPO (Group Relative Policy Optimization) training to further improve legal reasoning through reinforcement learning with process supervision.