Downloads · 30 days
8
2% of all-time downloads
LLM2025-SFT-LoRA
154teru/LLM2025-SFT-LoRA
LLM2025-SFT-LoRA is a text generation model from 154teru. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using QLoRA (4-bit, Unsloth).
All-time downloads
395
Public
Repo size
540 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors529 MB · 97%
At a glance
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Feb 8, 2026
- Updated
- Feb 8, 2026
- SHA
- 42eccaac
Try a prompt
Base models
Model card
From the Hugging Face model README
LLM2025-SFT-LoRA
This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using QLoRA (4-bit, Unsloth).
This repository contains LoRA adapter weights only. The base model must be loaded separately.
Training Objective
This adapter is trained to improve structured output accuracy (JSON / YAML / XML / TOML / CSV) for the Matsuolab LLM 2025 competition.
Loss is applied only to the final structured output (after Output: marker),
while intermediate reasoning (Chain-of-Thought) is masked.
Training Configuration
| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen3-4B-Instruct-2507 |
| Method | QLoRA (4-bit, Unsloth) |
| Dataset | u-10bei/structured_data_with_cot_dataset_512_v2 (~3.9k) |
| Max sequence length | 2048 |
| Epochs | 2 |
| Learning rate | 2e-6 |
| Per-device batch size | 4 |
| Gradient accumulation | 4 |
| Effective batch size | 16 |
| Warmup ratio | 0.1 |
| Weight decay | 0.05 |
| LoRA r | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0 |
| LoRA targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| CoT masking | Enabled (Output: marker) |
| Precision | bf16 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "154teru/LLM2025-SFT-LoRA"
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
base,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)
Sources & Terms (IMPORTANT)
Training data: u-10bei/structured_data_with_cot_dataset_512_v2
Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License. Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.
Framework versions
- PEFT 0.18.1
- Task
- Text Generation
- Library
- peft
- License
- apache-2.0
- Languages
- en
- Created
- Feb 8, 2026
- Updated
- Feb 8, 2026