SymbioSLM-ouroboros-lora-20260301

SymbioSLM-ouroboros-lora — Evolved LoRA Adapter for Gemma-3-270M

A LoRA adapter discovered by symbiogenesis — a population-based evolutionary framework that evolves adapter configurations (rank, target modules) through fusion and selection, with CUSUM gelation detection for automatic stopping.

Key Results

5x perplexity improvement on curated philosophy text with under 4% trainable parameters.

Adapter Configuration (Evolved)

The population of 10 adapters converged to all-7-target configs by generation 7 (gelation). MLP modules (gate_proj, up_proj, down_proj) reached 100% adoption — essential for causal LM adaptation on Gemma-3.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "LisaMegaWatts/Ouroboros-1MContext-Gemma-270m",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "LisaMegaWatts/SymbioSLM-ouroboros-lora-20260301")
tokenizer = AutoTokenizer.from_pretrained("LisaMegaWatts/Ouroboros-1MContext-Gemma-270m")

inputs = tokenizer("The nature of consciousness", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Method: Symbiogenesis

Symbiogenesis (Margulis, 1967) models complexity emerging through fusion of simpler organisms. Here, small LoRA adapters are the "organisms" — they fuse configurations (merging target modules and ranks) and compete on fitness.

Evolution:

• Population: 10 random LoRA adapters
• Generations: 17 (early stopped after gelation at gen 7 + 10 patience)
• Fusion: Hybrid (sequential = union targets + avg rank; parallel = union targets + sum ranks)
• Selection: Tournament (k=3)
• Fitness: -(val_loss + 0.01 × log(n_params))

Extended fine-tune: Best config trained 2000 steps with cosine LR schedule.

Data

Curated philosophy corpus from LisaMegaWatts/SymbioGPT-10M:

• Train: 20MB raw text → 4.3M tokens (8,383 sequences × 512 context)
• Val: 2MB raw text → 467K tokens (910 sequences × 512 context)
• Tokenizer: Gemma (262,145 vocab)

Hyperparameters

• Learning rate: 2e-4 (cosine decay, warmup=100)
• Batch size: 2 (gradient accumulation 4, effective batch 8)
• Precision: bfloat16
• Optimizer: AdamW (weight_decay=0.01)
• Gradient clipping: 1.0

Compute

W&B Runs

• Evolution: t85vupk8
• Extended fine-tune: bxpfv6ez

Evolution Details

Target Module Convergence

Extended Fine-Tune Curve

Links

• Base model: Ouroboros-1MContext-Gemma-270m
• Training corpus: SymbioGPT-10M
• Code: DavinciDreams/SymbioGPT
• Experiment write-up: Causal_LM_Gemma270M_Results.md
• Framework: Symbiogenesis

Framework Versions

• PEFT 0.18.1
• Transformers 5.0.0
• PyTorch 2.10.0