SymbioSLM-GrammarExpert-20260301

SymbioSLM Grammar Expert LoRA

A grammar-specialist LoRA adapter for SymbioSLM (~4.3M params), trained on CoLA (Corpus of Linguistic Acceptability) via symbiogenesis evolution. This is an attention-free model — all sequence mixing uses sub-quadratic organelles (CausalConv, Monarch matrices, LongConv).

Since SymbioSLM has no PyTorch checkpoint (it's Julia-native), this experiment trained with the full base model unfrozen alongside LoRA, testing whether the attention-free architecture can learn grammar from scratch.

Key Results

Grammar Sense Signal

The LoRA-adapted model assigns relatively higher probability to grammatical sentences:

                          Base      With LoRA
Acceptable log-prob:     -7.619     -7.617
Unacceptable log-prob:   -7.625     -7.632
Ratio (higher=better):   0.006      0.015  (+150% relative)

This is a small but directionally correct signal from a random-init 4M attention-free model.

Architecture

SymbioSLM is a 3-organelle decoder-only language model with NO attention:

• CausalDepthwiseConv1d — local n-gram pattern detection
• MonarchMatrix (8 heads) — sub-quadratic global mixing via butterfly factorization
• LongConv — dense causal convolution for medium-range dependencies
• OrganelleGate — learned per-channel blend across organelles

SymbioSLM: d_model=256, n_layers=6, n_monarch_heads=8, vocab_size=2000
Total params: 4,261,650

The attention-free design means LoRA can only target SwiGLU layers (w1, v, w2), giving 3 target types × 6 blocks = 18 possible injection points — far fewer than attention-equipped models.

LoRA Configuration

Manual LoRA injection (not PEFT) into SwiGLU feed-forward layers:

Best evolved config: rank=16, alpha=32.0, targets=(w1, w2)

Evolution consistently converged on the gate+output pair (w1, w2), preferring this over configurations that include the value projection (v).

Evolution Details

1. Population: 8 random LoRAUnit configs
2. Training: 200 steps per unit, lr=2e-4, batch=16, base unfrozen (no pre-trained checkpoint)
3. Fitness: accuracy - 0.01 × log(n_trainable)
4. Gelation: CUSUM change-point at generation 6 (CUSUM=4.10)
5. Post-gelation: Architecture locked (r=16, w1+w2) but test accuracy continued improving

Test Accuracy Over Time

Gen  0: 40.4%
Gen  5: 54.0%  (pre-gelation)
Gen  6: 40.0%  (at gelation)
Gen 10: 61.2%
Gen 15: 57.4%
Gen 20: 56.0%
Gen 24: 60.6%  (final)

Test accuracy oscillated but trended upward, suggesting continued evolution post-gelation was beneficial for this model. Gelation marked architecture convergence, not a generalization peak.

Usage

Requires the SymbioSLM model architecture. See the training notebook for the full model definition.

import torch
from huggingface_hub import hf_hub_download

# Load LoRA weights
weights_path = hf_hub_download(
    "LisaMegaWatts/SymbioSLM-GrammarExpert-20260301",
    "lora_state.pt"
)
lora_state = torch.load(weights_path, map_location="cpu")

# Inject into SymbioSLM base model
# inject_lora(model, target_modules=['w1', 'w2'], rank=16, alpha=32.0)
# load_lora_state(model, lora_state)

Files

Part of Symbiogenesis

This is part of a 3-model grammar expert comparison:

W&B run: grammar-expert-symbioslm

GitHub: DavinciDreams/SymbioGPT