qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6

qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6

This repository provides a LoRA adapter fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using QLoRA (4-bit, Unsloth).

Training Objective

This adapter is trained to improve structured output accuracy (JSON / YAML / XML / TOML / CSV).

Strategy 2 Revised: YAML/XML Clean Expansion 🔥

Training Configuration

• Base model: Qwen/Qwen3-4B-Instruct-2507
• Method: QLoRA (4-bit)
• Max sequence length: 512
• Epochs: 1
• Learning rate: 6e-06
• LoRA: r=16, alpha=32

Dataset: Cleaned StructEval + YAML/XML Boost (Revised)

Strategy 2 Revised: Three-Pronged Approach

Based on expert AI analysis, this revision addresses the root causes of YAML/XML failures:

1. u-10bei系: 「Output:」以降だけを抽出

   • Problem: u-10bei datasets contain "Approach: → Output:" structure
   • Solution: Extract only content after "Output:" marker
   • Effect: Removes "explanation before output" tendency

2. daichira系: 2倍ブースト（「Return ONLY」パターン）

   • Problem: Model needs more "Return ONLY YAML/XML" examples
   • Solution: Boost daichira datasets by 2x
   • Effect: Strengthens "direct output without explanation" pattern

3. YAML/XML: 2倍ブースト

   • Problem: YAML/XML are the weakest formats (94.3%, 85.0%)
   • Solution: Boost YAML/XML samples by 2x
   • Effect: More training data for weak formats

Data Cleaning Pipeline:

1. CoT tags removal: <thinking>...</thinking> completely removed
2. Code fence removal: yaml, json, xml, toml, ````csv removed
3. Leading phrase removal: "Here's the output:", "Sure!", etc. removed
4. 🔥 Output extraction: For u-10bei datasets, extract only content after "Output:" marker
5. Format validation: JSON/YAML/XML/TOML/CSV parsing validation
6. Deduplication: Exact duplicates removed

Format Distribution (Estimated):

• YAML: ~12,000-15,000 (40-50%) 🔥 2x boost
• XML: ~6,000-8,000 (25-30%) 🔥 2x boost
• JSON: ~4,000-5,000 (15-20%)
• TOML: ~2,500-3,000 (10%)
• CSV: ~2,500-3,000 (10%)

Total: ~25,000-30,000 samples

Source Datasets:

u-10bei系（「Output:」抽出適用）:

• u-10bei/structured_data_with_cot_dataset_512_v2
• u-10bei/structured_data_with_cot_dataset_512_v4
• u-10bei/structured_data_with_cot_dataset_512_v5
• u-10bei/structured_data_with_cot_dataset_512
• u-10bei/structured_data_with_cot_dataset_v2
• u-10bei/structured_data_with_cot_dataset

daichira系（2倍ブースト）:

• daichira/structured-3k-mix-sft 🔥 2x
• daichira/structured-5k-mix-sft 🔥 2x
• daichira/structured-hard-sft-4k 🔥 2x

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "Qwen/Qwen3-4B-Instruct-2507"
adapter = "yuk1chan/qwen3-4b-structeval-yamlxml-boost-v2-lr6e-6"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

# Inference
prompt = "Generate YAML code for..."
# ... your inference code

Training Results

- Training Loss: ~1.25-1.30
- Validation Loss: ~1.50-1.55
- Training Time: ~8-9 hours
- Expected Score: 0.805-0.810 (YAML/XML improvement from 0.80195 baseline)

Strategy: YAML/XML Clean Expansion (Revised)

Key Insights from Expert AI Analysis:

The main causes of YAML/XML failures are:
1. Code fences (yaml, xml) appearing in output
2. Leading phrases ("Here's the output:", "Sure!") before output
3. u-10bei datasets' "Approach: → Output:" structure leaking into outputs

Solutions Implemented:
┌─────────────────────────────────────┬──────────────────────────────┬───────────────────────────────────┐
│               Problem               │           Solution           │              Effect               │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ u-10bei "Output:" structure         │ Extract only after "Output:" │ Removes explanation tendency      │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ Insufficient "Return ONLY" examples │ Boost daichira datasets 2x   │ Strengthens direct output pattern │
├─────────────────────────────────────┼──────────────────────────────┼───────────────────────────────────┤
│ YAML/XML too weak                   │ Boost YAML/XML 2x            │ More training data                │
└─────────────────────────────────────┴──────────────────────────────┴───────────────────────────────────┘
Expected Improvements:
┌────────┬──────────┬────────┬──────────────────────────┐
│ Format │ Baseline │ Target │         Strategy         │
├────────┼──────────┼────────┼──────────────────────────┤
│ YAML   │ 94.3%    │ 96-97% │ Clean u-10bei + 2x boost │
├────────┼──────────┼────────┼──────────────────────────┤
│ XML    │ 85.0%    │ 88-90% │ Clean u-10bei + 2x boost │
├────────┼──────────┼────────┼──────────────────────────┤
│ TOML   │ 100.0%   │ 100%   │ Maintain                 │
├────────┼──────────┼────────┼──────────────────────────┤
│ JSON   │ 96.0%    │ 96%+   │ Maintain                 │
├────────┼──────────┼────────┼──────────────────────────┤
│ CSV    │ 100.0%   │ 100%   │ Maintain                 │
└────────┴──────────┴────────┴──────────────────────────┘
License

Apache 2.0

---
Trained on Cleaned StructEval dataset (YAML/XML Boosted - Revised)
Learning Rate: 6e-6 (conservative setting)
Strategy: u-10bei Output extraction + daichira 2x + YAML/XML 2x
Based on expert AI analysis