faast-Qwen2.5-3B-Instruct

FAAST-Qwen2.5-3B-Instruct

faast-Qwen2.5-3B-Instruct is an extension of Qwen2.5-3B-Instruct equipped with the FAAST module. The original Qwen2.5-3B-Instruct parameters are frozen, while only the FAAST readout projections are trained.

The model is designed for efficient test-time learning through fast weights, enabling adaptation without backpropagation (gradient descent).

Model Description

FAAST augments Qwen2.5-3B-Instruct with fast-weight adaptation modules that support supervised learning during inference. During FAAST pretraining, all backbone LLM parameters remain frozen, and only lightweight FAAST readout projections are optimized.

This design enables:

• Test-time learning without backpropagation
• Efficient adaptation with low memory overhead
• Fast adaptation to downstream tasks
• Improved few-shot and full-data performance

Usage:

Before running the following code, we need to import the modules from https://github.com/baoguangsheng/faast

tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(args.model_path, trust_remote_code=True)

fewshot_samples = ['sample 1', 'sample 2', ...]
inputs = tokenizer(fewshot_samples, return_tensors="pt", padding=True)

model.reset_projection() # clear existing fast weights
model.learn(**inputs)  # learn new fast weights
model.generate(...)  # do the task using the learned fast weights

Training Details

• Base model: Qwen2.5-3B-Instruct
• Trainable parameters: FAAST readout projections
• Frozen parameters: All Qwen2.5-3B-Instruct parameters
• Pretraining corpus: OpenWebText2
• Adaptation mechanism: Fast weights / FAAST readout projections

Evaluation Results

Machine Translation on IWSLT2017

BLEU scores on IWSLT2017. Bold scores indicate statistical significance at p < 0.05.

Qwen2.5-3B-Instruct Backbone

Qwen2.5-7B-Instruct Backbone

Key Features

• Frozen backbone LLM parameters
• Lightweight FAAST readout adaptation
• Test-time learning capability
• Efficient memory usage
• Strong few-shot translation performance
• Compatible with instruction-tuned LLMs

Limitations

• The model inherits the limitations and biases of Qwen2.5-3B-Instruct.
• Performance may vary across domains and languages not covered during evaluation.
• FAAST adaptation quality depends on the distribution and quality of test-time examples.
• The model is primarily intended for research purposes.

Citation

If you use this model, please cite the corresponding FAAST paper or project.

@article{bao2026faast,
  title={FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation},
  author={Bao, Guangsheng and Zhang, Hongbo and Cui, Han and Sun, Ke and Zhao, Yanbin and He, Juncai and Zhang, Yue},
  journal={arXiv preprint arXiv:2605.04651},
  year={2026}
}