faast-gpt2-xl

FAAST-GPT2-XL

faast-gpt2-xl is an extension of gpt2-xl equipped with the FAAST module. The original GPT2-XL language model parameters are frozen, and only the FAAST readout projections are trained.

Model Description

FAAST-GPT2-XL augments GPT2-XL with FAAST modules for efficient test-time supervised learning. During FAAST pretraining, the parameters of GPT2-XL are kept frozen, while the FAAST readout projections are trained on OpenWebText2.

The model is designed to support learning at test time through fast weights, allowing adaptation without backpropagation (gradient descent).

Training Details

• Base model: GPT2-XL
• Trainable parameters: FAAST readout projections
• Frozen parameters: All GPT2-XL parameters
• Training corpus: OpenWebText2
• Training objective: FAAST readout projection learning with the frozen GPT2-XL backbone

Intended Use

This model is intended for research on:

• Test-time learning
• Fast-weight adaptation
• Efficient language model adaptation
• Sequence modeling
• Few-shot learning
• Frozen-backbone adaptation methods

Usage:

Before running the following code, we need to import the modules from https://github.com/baoguangsheng/faast

tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(args.model_path, trust_remote_code=True)

fewshot_samples = ['sample 1', 'sample 2', ...]
inputs = tokenizer(fewshot_samples, return_tensors="pt", padding=True)

model.reset_projection() # clear existing fast weights
model.learn(**inputs)  # learn new fast weights
model.generate(...)  # do the task using the learned fast weights

Evaluation Results

Sentiment Classification

Sentiment classification accuracy on SST-2 and IMDB with 95% confidence intervals.

Sequence Modeling on WikiText-103

Sequence modeling adaptation results using GPT2-XL. Inference FLOPs and memory report only the increased cost over the GPT2-XL base model.

The WikiText-103 setup uses hidden size d_x = 1600, number of layers L = 48, and number of training tokens N = 1.03 × 10^8. The in-context learning cost is a theoretical estimate because the base GPT2-XL model does not support such long contexts.

Limitations

• The model is trained on English web text and is primarily intended for English-language tasks.
• Test-time learning behavior depends on the quality and distribution of the test-time examples.

Citation

If you use this model, please cite the corresponding FAAST paper or project.

@article{bao2026faast,
  title={FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation},
  author={Bao, Guangsheng and Zhang, Hongbo and Cui, Han and Sun, Ke and Zhao, Yanbin and He, Juncai and Zhang, Yue},
  journal={arXiv preprint arXiv:2605.04651},
  year={2026}
}