UE5_Training_MCP

UE5 Training MCP Pipeline

Goal: Use Unreal MCP + latest LLM to generate high-quality training data, then fine-tune smaller models and evaluate them against the generated data.

Models on Hugging Face

Trained LoRA adapters are published as three public model repos. Each downloads into this directory layout by default — and matches the local output path produced by scripts/train_qwen35.py.

Download a specific adapter into the project (overlays onto outputs/models/):

# 0.8B
hf download Yhyu13/Qwen3.5-0.8B-UE5-LoRA \
    --local-dir outputs/models/qwen3.5-0.8b-ue5-lora

# 2B
hf download Yhyu13/Qwen3.5-2B-UE5-LoRA \
    --local-dir outputs/models/qwen3.5-2b-ue5-lora

# 4B
hf download Yhyu13/Qwen3.5-4B-UE5-LoRA \
    --local-dir outputs/models/qwen3.5-4b-ue5-lora

The training repo on Hugging Face (Yhyu13/UE5_Training_MCP) mirrors this directory's source minus the heavy outputs/venv/ and outputs/models/ trees; reproduce the env with pip install -r requirements.txt.

Pipeline Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Unreal MCP     │───→│  LLM Data Gen   │───→│  Data Pruning   │
│  (UE5 Context)  │    │  (X conversations)│   │  (Quality Filter)│
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Excel Report   │←───│  SFT Eval       │←───│  Data Prep      │
│  (Metrics)      │    │  (vs Latest LLM)│   │  (Train/Val/Test)│
└─────────────────┘    └─────────────────┘    └─────────────────┘
                              │
                              ▼
                       ┌─────────────────┐
                       │  Train Small    │
                       │  Model (SFT)    │
                       │  Qwen3.5 (LoRA) │
                       └─────────────────┘

Phases

Phase 1: Data Generation via MCP (scripts/mcp_data_generator.py)

Use Unreal MCP to provide UE5 context to the latest LLM, generating:

• Multi-turn conversations (interview-style, 4-5 turns)
• Code explanations (with UE5 source context from MCP)
• Technical Q&A (with engine-specific details)

python scripts/mcp_data_generator.py \
  --mcp_server_path /path/to/mcp_server \
  --model claude-sonnet-4-20250514 \
  --num_conversations 100 \
  --output ../data/raw/conversations.jsonl

Phase 2: Data Pruning (scripts/data_pruner.py)

Remove low-quality data using multiple filters:

• Length filter: Too short (< 100 tokens) or too long (> 2048 tokens)
• Factuality filter: Check against known UE5 facts (source paths, API names)
• Duplicate filter: Remove semantically similar conversations
• Quality score: LLM-as-judge rates each conversation 1-5

python scripts/data_pruner.py \
  --input ../data/raw/conversations.jsonl \
  --output ../data/processed/conversations_pruned.jsonl \
  --min_quality 3.5

Phase 3: Data Preparation (scripts/data_prep.py)

Split into train/val/test and format for training:

python scripts/data_prep.py \
  --input ../data/processed/conversations_pruned.jsonl \
  --output_dir ../data/splits \
  --train_ratio 0.8 \
  --val_ratio 0.1

Phase 4: Train Small Models (scripts/train_small_model.py / scripts/train_qwen35.py)

Fine-tune small Qwen3.5 models (0.8B / 2B / 4B) using PEFT/LoRA:

# Qwen3.5-0.8B (lives in scripts/train_qwen35.py; the actual trainer used)
python scripts/train_qwen35.py \
  --base_model Qwen/Qwen3.5-0.8B \
  --train data/splits/train.jsonl \
  --val   data/splits/val.jsonl \
  --out   outputs/models/qwen3.5-0.8b-ue5-lora

Target models (small enough to run locally on a single 24 GB consumer GPU):

Phase 5: Evaluation (scripts/eval_model.py)

Evaluate fine-tuned model against:

1. Fixed benchmark (generated by latest LLM, held-out set)
2. Generated questions (model answers vs latest LLM answers)
3. MCP integration test (model answers with live UE5 context)

python scripts/eval_qwen35.py \
  --base_model  Qwen/Qwen3.5-0.8B \
  --adapter_dir outputs/models/qwen3.5-0.8b-ue5-lora \
  --benchmark   data/splits/test.jsonl \
  --output      outputs/results/eval_ft_test.json

Phase 6: Export to Excel (scripts/export_to_excel.py)

Generate comparative Excel report:

python scripts/export_to_excel.py \
  --results ../outputs/results/eval_*.json \
  --output ../outputs/results/comparison_report.xlsx

Directory Structure

UE5_Training_MCP/
├── config/
│   ├── mcp_config.json          # MCP server configuration
│   └── training_config.yaml     # Training hyperparameters
├── data/
│   ├── raw/                     # Raw LLM-generated conversations
│   ├── processed/               # Cleaned and pruned data
│   ├── splits/                  # Train/val/test splits
│   └── eval/                    # Evaluation datasets
├── scripts/
│   ├── mcp_data_generator.py  # Phase 1: Generate via MCP
│   ├── data_pruner.py           # Phase 2: Prune low-quality
│   ├── data_pruner_v2.py        # Phase 2': grounded pruner (used for the published runs)
│   ├── data_prep.py             # Phase 3: Format for training
│   ├── train_small_model.py     # Phase 4: legacy SFT small models
│   ├── train_qwen35.py          # Phase 4': Qwen3.5 LoRA trainer (the one used)
│   ├── eval_model.py            # Phase 5: generic evaluate
│   ├── eval_qwen35.py           # Phase 5': Qwen3.5 LoRA eval (the one used)
│   └── export_to_excel.py       # Phase 6: Export results
├── eval/
│   └── benchmark_questions.jsonl # Fixed benchmark
├── outputs/
│   ├── models/                  # Saved checkpoints
│   └── results/                 # Evaluation results
└── README.md                    # This file

Prerequisites

pip install transformers peft accelerate bitsandbytes trl datasets
pip install pandas openpyxl          # For Excel export
pip install mcp                      # MCP client (if using MCP)
pip install openai anthropic        # For direct API calls

Quick Start

cd UE5_Training_MCP

# 1. Generate data (requires MCP server running or API key)
python scripts/mcp_data_generator.py --num_conversations 50

# 2. Prune
python scripts/data_pruner.py

# 3. Prepare
python scripts/data_prep.py

# 4. Train (pick your model size)
python scripts/train_qwen35.py \
    --base_model Qwen/Qwen3.5-0.8B \
    --train data/splits/train.jsonl \
    --val   data/splits/val.jsonl \
    --out   outputs/models/qwen3.5-0.8b-ue5-lora

# 5. Evaluate
python scripts/eval_qwen35.py \
    --base_model  Qwen/Qwen3.5-0.8B \
    --adapter_dir outputs/models/qwen3.5-0.8b-ue5-lora

# 6. Export
python scripts/export_to_excel.py

Training & Evaluation (concrete numbers)

Reproduced end-to-end on a single workstation, no cloud.

Hardware / rig

• 1× NVIDIA RTX 3090 (24 GB) — one of two on host, deliberately single-GPU at this scale.
• CUDA 12.1 wheels (torch==2.5.1+cu121), Python 3.11.8, isolated venv at outputs/venv/ (reproduced via requirements.txt).
• bf16 mixed precision; PEFT/LoRA only, no DDP.

Shared hyperparameters (all three sizes)

• lora_r=16, lora_alpha=32, lora_dropout=0.05
• target_modules = {q,k,v,o,gate,up,down}_proj
• max_seq_length = 512
• epochs = 3
• effective_batch_size = 8
• learning_rate ∈ {3e-4 (0.8B, 2B), 2e-4 (4B)}

Per-size training cost (recorded in each train_meta.json)

Evaluation (held-out data/splits/test.jsonl, n = 15 UE5-MCP in-domain)

• In-domain kw overlap on UE5-MCP tool-calling test set jumps ~+80 % for the 0.8B model after FT; answers also tighten by ~30 % in length (less verbose).
• Out-of-domain (10 unrelated Chinese Nanite/Lumen theory questions, see outputs/results/eval_*_bench.*): benchmark kw overlap is essentially flat (~0.13 → 0.12), as expected for narrow small-data LoRA specialization.
• Full base-vs-FT transcripts live under outputs/results/side_by_side_test.md and outputs/results/eval_*_test.md.
• lm_eval runs (base vs FT at 0.8B / 2B / 4B) are in outputs/lm_eval_results/.

Why three sizes train equally well at n = 108 examples: larger bases (4B) need more SFT data to specialize; at 108 records the FT advantage is comparable across sizes, suggesting data scale — not model scale — is the binding constraint here.

Master matrix

UE5-MCP test (15 in-domain, kw overlap):

Commonsense / RC (lm_eval, 500/task):

Key findings

• No regression from FT — 0.8B-FT differs from 0.8B-BASE by ≤ 2.6 pp on every commonsense task (mostly within ±1.5 pp). LoRA at lr=3e-4 / 3 epochs is conservative enough.
• Scale helps commonsense monotonically (BASE only) — 0.8B → 2B → 4B improves every task.
• 2B-FT beats 4B-FT on UE5-MCP (0.425 vs 0.318) — the 4B adapter is under-trained on 108 records (final loss 0.44 vs 2B's 0.36).
• All BASE versions are flat (~0.23) on UE5-MCP regardless of size — the domain is niche, not in pretraining.
• Cross-domain Chinese bench is flat for everyone (~0.12), as expected for narrow SFT.

Answers

Can a fine-tuned small model beat a larger one? Yes on UE5-MCP: 2B-FT (0.425) > 4B-FT (0.318) and > 4B-BASE (0.226). Why? Format matters more than capacity for narrow tasks; the LoRA adapter is 5× relatively larger on 2B than on 4B; SFT teaches surface lexical matches (ListActors, Tool calls:, 391 actors) that base models don't emit.

If not, how to improve? You can — but the 4B model is under-trained. Next steps: ≈3× more data (~300 records), bump LoRA r=32/64 on the 4B base, optionally full-FT the last 2 transformer blocks. Expected outcome: 4B-FT overtakes 2B-FT once data ≈ 300 records.

Wall-clock totals

Key Design Decisions

1. MCP for Context: Unreal MCP provides live UE5 engine context (source paths, API docs, console variables) to the LLM, making generated data factually grounded.

2. Small Models: We target 1.5B-7B models that can run on consumer GPUs (8-16GB VRAM), making iteration fast and cheap.

3. Pruning > Quantity: We generate many (X=100-500) conversations, then prune to the top 30% by quality. Better than manual writing 50 examples.

4. Eval vs Latest LLM: The evaluation benchmark is generated by the same latest LLM, ensuring the bar is high. The fine-tuned small model should match or exceed it on UE5-specific questions.