fast-weight-time-machine

Fast-Weight Time Machine

Fast-Weight Time Machine tests temporary variable binding with an explicit, sequence-local weight matrix. A controller receives key/value writes, produces a write strength, and accumulates outer products in a fast memory. A later query key reads the memory in one matrix-vector operation. The learned parameters stay fixed between examples; only the fast matrix changes inside each sequence.

The historical anchor is Schmidhuber's 1992 Learning to Control Fast-Weight Memories, which described feedforward controllers producing context-dependent weight changes and included adaptive temporary-variable binding. This project is a modern outer-product interpretation tested against a similarly sized GRU. It does not claim to reproduce the original implementation or experiments exactly.

Verified results

The models trained on 16,000 sequences containing four writes and 12 distractors. Each held-out condition contained 4,000 new sequences.

This task is intentionally aligned with the fast-weight architecture: an explicit outer-product matrix can bind a key and value, while the GRU must compress all bindings into one recurrent vector. The comparison demonstrates the inductive bias; it is not a general claim that fast weights outperform GRUs on arbitrary sequences.

Reproduce

uv run python projects/fast-weight-time-machine/train.py