Downloads · 30 days
11
22% of all-time downloads
swegym-qwen2.5-coder-7b-instruct-lora-64k
MMR115/swegym-qwen2.5-coder-7b-instruct-lora-64k
swegym-qwen2.5-coder-7b-instruct-lora-64k is a text generation model from MMR115. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
A LoRA adapter fine-tuned on Qwen2.5-Coder-7B-Instruct using SWE-agent trajectory data distilled from Qwen3-Coder-480B-A35B-Instruct on the SWE-Gym dataset.
All-time downloads
49
Public
Repo size
51.9 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors40.4 MB · 78%
At a glance
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Mar 31, 2026
- Updated
- Mar 31, 2026
- SHA
- 35a1f186
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
SWE-Gym Qwen2.5-Coder-7B-Instruct LoRA (64K Context)
A LoRA adapter fine-tuned on Qwen2.5-Coder-7B-Instruct using SWE-agent trajectory data distilled from Qwen3-Coder-480B-A35B-Instruct on the SWE-Gym dataset.
Model Details
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-Coder-7B-Instruct |
| Fine-tuning Method | LoRA (Low-Rank Adaptation) |
| LoRA Rank (r) | 8 |
| LoRA Alpha | 16 |
| LoRA Dropout | 0.0 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Max Context Length | 64K tokens |
| Precision | bfloat16 |
Training Details
| Property | Value |
|---|---|
| Training Data | 634 resolved SWE-Gym instances |
| Teacher Model | Qwen3-Coder-480B-A35B-Instruct |
| Agent Framework | OpenHands CodeActAgent |
| Epochs | 3 |
| Total Steps | 60 |
| Batch Size | 1 per device × 8 GPUs × 4 grad accum = 32 effective |
| Learning Rate | 1e-4 (cosine schedule, 10% warmup) |
| Optimizer | AdamW (β1=0.9, β2=0.999, ε=1e-8) |
| Final Training Loss | 0.379 |
| Training Runtime | ~2.5 hours |
| Framework | LLaMA-Factory + DeepSpeed |
| PEFT Version | 0.18.1 |
| Transformers | 5.2.0 |
| PyTorch | 2.6.0 |
Training Data
The training data consists of 634 resolved instances from the SWE-Gym training set. Trajectories were generated by running Qwen3-Coder-480B-A35B-Instruct (via OpenHands CodeActAgent with maxiter=100) on SWE-Gym tasks, then filtering to only resolved (successful) trajectories. Function-calling messages were converted to non-function-calling format for SFT, and trajectories exceeding 64K tokens were excluded.
Training Curve
| Step | Epoch | Loss | Learning Rate |
|---|---|---|---|
| 5 | 0.25 | 0.526 | 6.67e-05 |
| 10 | 0.50 | 0.497 | 9.92e-05 |
| 20 | 1.00 | 0.444 | 8.64e-05 |
| 30 | 1.50 | 0.402 | 6.15e-05 |
| 40 | 2.00 | 0.389 | 3.29e-05 |
| 50 | 2.50 | 0.370 | 9.89e-06 |
| 60 | 3.00 | 0.379 | 8.46e-08 |
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-Coder-7B-Instruct",
torch_dtype="auto",
device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "MMR115/swegym-qwen2.5-coder-7b-instruct-lora-64k")
tokenizer = AutoTokenizer.from_pretrained("MMR115/swegym-qwen2.5-coder-7b-instruct-lora-64k")
Citation
If you use this model, please cite SWE-Gym and OpenHands:
@article{pan2024swegym,
title={Training Software Engineering Agents and Verifiers with SWE-Gym},
author={Pan, Jiayi and Xiao, Xingyao and Wang, Jinda and Graham, Colin and Wang, Xinran and Hu, Hoang and Wang, Rui and Shi, Heng and Liu, Pengfei and Wang, Huan and Qian, Cong},
journal={ICML},
year={2025}
}
- Tokenizers 0.22.2
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Created
- Mar 31, 2026
- Updated
- Mar 31, 2026