progressive-cognitive-dream-lora-en

Progressive Cognitive Architecture — 1.5B Dream LoRA (English)

🏆 Best overall model (composite 87.6/100) — Qwen2.5-1.5B fine-tuned with 4-phase progressive training + SVD Dream Pruning.

✨ Highlights

Results: mean ± std over 3 seeds (42, 43, 44), 50 samples × 5 dimensions per seed.

🔑 Key Findings

• Outperforms all 3B variants despite having half the parameters
• Zero catastrophic errors — never produces absurd results
• 100% delegation — always routes complex operations to tools
• Dream pruning acts as cognitive regularization for capacity-constrained models

🧠 Progressive Cognitive Architecture

A bio-inspired 4-phase training methodology:

Guiding Principle: Knowledge doesn't disappear — it collapses into attractors. Intuition is the compressed residue of experience.

🌙 Dream Pruning (SVD Low-Rank Factorization)

Instead of zeroing out small weights (magnitude pruning), Dream Pruning uses SVD decomposition to reduce the effective rank of LoRA matrices from 16 to 8. This preserves the principal directions ("logical connections") while discarding noise — analogous to memory consolidation during sleep.

W = U·Σ·V^T  →  W' = U[:,:k]·Σ[:k,:k]·V^T[:k,:]   (k=8)

🔧 Training Configuration

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B", device_map="auto", torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")

# Note: adapters are in the lora_adapters/ subfolder
model = PeftModel.from_pretrained(
    base_model,
    "dexmac/progressive-cognitive-dream-lora-en",
    subfolder="lora_adapters"
)

messages = [{"role": "user", "content": "Solve: 342 * 67"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Expected output pattern:

Step 1 - Intuition: in the order of tens of thousands
Step 2 - Routing: DELEGATE (medium complexity)
Step 3 - Tool: 22914
Step 4 - Validation: result 22914 consistent with estimate → VALID

📊 Full Comparison

🔗 Related Models

• Flat LoRA (control) — Same data, no phases, no pruning
• 3B Dream — Same architecture on Qwen2.5-3B
• 3B Flat — 3B control
• Italian Dream — Italian language variant
• Results Dataset — Raw evaluation data
• GitHub — Full source code and evaluation framework

📝 Citation

@software{progressive_cognitive_2026,
  author = {Dex Mac},
  title = {Progressive Cognitive Architecture for LLMs},
  year = {2026},
  url = {https://github.com/dexmac221/progressive-cognitive},
  version = {1.0.0}
}

📄 License

Apache 2.0