fela-acml2026

FELA: Fourier Encoder with Linear Attention

FELA replaces standard self-attention with a hybrid FNO+GLA block that runs in O(N log N) time during prefill and O(1) per token during autoregressive generation via a fixed-depth ring buffer.

Paper: FELA: Fourier Encoder with Linear Attention for Efficient Long Context Language Modeling (ACML 2026)

Architecture

• FNO (Fourier Neural Operator): causal M-tap FIR convolution computed via FFT during training; ring buffer at generation
• GLA (Gated Linear Attention): recurrent state O(H × Dh²), independent of sequence length
• SSSL pattern: 3 FNO : 1 GLA per 4 layers
• Main model: 1.13B params, 24 layers, d=2048, 16 heads, M=512 FNO modes

Checkpoints

Setup

pip install uv
uv sync

Or directly:

pip install torch datasets tiktoken lm-eval boto3

Training (GPU)

python scripts/train_gpu.py \
  --config gpt2-1b \
  --data-dir /path/to/data \
  --ckpt-dir checkpoints/run

Training (CPU DiLoCo)

python train_cpu_gpt2.py \
  --config gpt2-small \
  --tokens 5e8 \
  --node-rank $RANK \
  --num-nodes 10 \
  --master-ip $MASTER_IP \
  --s3-bucket my-bucket \
  --run-name cpu_gpt2

Evaluation

python scripts/run_eval.py \
  --checkpoint checkpoints/fno_gla_gpu_v3_22b/step_111897_final.pt \
  --tasks wikitext,boolq,hellaswag,arc_easy,piqa

Results

Evaluation outputs from the paper are in results/paper_eval_22b/:

Data Preparation

python prepare.py

Downloads and tokenizes OpenWebText using tiktoken (GPT-2 BPE).