llama3.1-8b-pmt-dual-factorized

Tri-State Additive Factorized PrefixMemory-Tuning (PMT)

This experiment introduces Tri-State Additive Co-Steering. It solves the inference boundary-collapse issue of pure factorization by restoring the explicit CEFR anchor.

Architecture Details

1. Tri-State Gating: $v = C_{cefr} + C_{lex} + C_{syn}$. By combining all three lookups additively, the model maintains a strong global CEFR boundary anchor while utilizing the lexical and syntactic embeddings to prevent feature tangling within complex samples.
2. Uncompressed PMT: Full 4096*4096 $M$ matrices across 32 layers.
3. No Dense MLPs: Prevents catastrophic memorization and parameter bloat.

Training Performance

[
  {
    "epoch": 1,
    "train_loss": 3.1963,
    "val_loss": 2.1928,
    "val_ppl": 8.96
  },
  {
    "epoch": 2,
    "train_loss": 1.9374,
    "val_loss": 2.1087,
    "val_ppl": 8.24
  },
  {
    "epoch": 3,
    "train_loss": 1.4273,
    "val_loss": 2.231,
    "val_ppl": 9.31
  },
  {
    "epoch": 4,
    "train_loss": 0.7167,
    "val_loss": 2.5834,
    "val_ppl": 13.24
  },
  {
    "epoch": 5,
    "train_loss": 0.2827,
    "val_loss": 3.0198,
    "val_ppl": 20.49
  }
]