progressive-lora-merging

🧟 Body Snatching: Progressive LoRA Merging (PLM)

Complete model identity replacement using only LoRA-level resources.

"What if catastrophic forgetting is a feature, not a bug?"

🔥 What is this?

Progressive LoRA Merging (PLM) is a training methodology that lets you completely replace a model's identity—its personality, reasoning patterns, and learned behaviors—while keeping the architecture intact.

Think of it as body snatching for LLMs:

• The body (architecture, tokenizer, attention mechanisms) stays
• The soul (personality, knowledge, behavior) gets replaced

After enough cycles, you don't have "Qwen fine-tuned for X". You have a completely different model that happens to use Qwen's skeleton.

💡 The Key Insight

Everyone treats catastrophic forgetting as a problem to avoid.

We treat it as the goal.

🔄 How It Works

Cycle 1:  Base Model → Train LoRA → Merge → New Base₁
Cycle 2:  New Base₁  → Train LoRA → Merge → New Base₂
...
Cycle N:  New Base_N = Completely Different Model

Each cycle:

1. Train a small LoRA adapter (~0.1% of parameters)
2. Merge it permanently into the base weights (in BF16, not 4-bit!)
3. Fresh LoRA for the next cycle
4. Repeat until original identity is gone

⚠️ Important: This is NOT LoRA Stacking

After each merge, the LoRA is dissolved into base weights and ceases to exist. Next cycle trains a fresh LoRA on the new base. No compounding (a+b)² × (a+b)². After 100 cycles = ONE model with rewritten weights.

🔀 Dataset Strategy

50% new examples + 50% historical samples. This ensures forgetting targets the BASE model, not your training data.

📊 Results

After 100 cycles, the model is 93% your data, 7% original.

💰 Resource Comparison

🚀 Quick Start

pip install torch transformers peft bitsandbytes datasets

python plm.py --base-model Qwen/Qwen3-1.7B --dataset data.jsonl --cycles 100

📖 Citation

@article{drissi2024bodysnatching,
  title={Body Snatching: Complete Model Identity Replacement via Progressive LoRA Merging},
  author={Drissi, Ouissam Said},
  year={2024},
  url={https://github.com/antibitcoin/progressive-lora-merging}
}

🔗 Links

• GitHub: antibitcoin/progressive-lora-merging
• Paper: PAPER.md
• Related Work: ASRL Paper (IJSET 2025)

👤 Author

Ouissam Said Drissi

• Email: [email protected]
• Independent Researcher, Morocco

---

"You're not fine-tuning a model. You're growing a new one inside its skeleton."