lstm-time-capsule

LSTM Time Capsule

LSTM Time Capsule is a modern small-compute retest of long-lag credit assignment. Every sequence contains random values and two distant markers; the model must add only the marked values after processing the entire sequence. A parameter-matched vanilla tanh RNN, an LSTM, and a GRU train on identical data and are evaluated at the training length and at longer unseen lengths.

The historical anchor is Hochreiter and Schmidhuber's 1997 Long Short-Term Memory, which motivated LSTM by the difficulty of decaying error flow over extended intervals. This project uses current PyTorch implementations and a new synthetic benchmark; it does not claim to reproduce the paper's original code, exact cells, or tables.

Verified results

All cells trained on the same 16,000 length-100 sequences. The test sets contained 4,000 independently generated sequences at each length.

At the training length, LSTM placed 100% of predictions within 0.1 of the target; the vanilla RNN reached 19.03%. At four times the training length, LSTM retained 56.83% within 0.1 while GRU reached 11.50%. This is evidence for these saved checkpoints on this adding task, not a universal architecture ranking.

Reproduce

uv run python projects/lstm-time-capsule/train.py