backpack-gpt2

Model Card for Backpack-GPT2

<!-- Provide a quick summary of what the model is/does. [Optional] -->

The Backpack-GPT2 language model is an instance of the Backpack architecture, intended to combine strong modeling performance with an interface for interpretability and control. Most details about this model and its training should be accessed in the paper, Backpack Language Models.

See also backpackmodels.science.

Table of Contents

• Model Card for Backpack-GPT2
• Table of Contents
• Model Details
  • Model Description
• Uses
• Bias, Risks, and Limitations
• Training Details
  • Training Data
  • Training Procedure
• Environmental Impact
• Technical Specifications [optional]
  • Model Architecture and Objective
  • Compute Infrastructure
    • Hardware
    • Software
• Citation
• Model Card Authors [optional]
• Model Card Contact
• How to Get Started with the Model

Model Details

Model Description

<!-- Provide a longer summary of what this model is/does. -->

The Backpack-GPT2 is a Backpack-based language model, an architecture intended to combine strong modeling performance with an interface for interpretability and control.

• Developed by: John Hewitt, John Thickstun, Christopher D. Manning, Percy Liang
• Shared by [Optional]: More information needed
• Model type: Language model
• Language(s) (NLP): en
• License: apache-2.0
• Resources for more information:
  • GitHub Repo
  • Associated Paper

Uses

This model is intended for use in the study and development of increasingly interpretable methods in natural language processing. It is not directly fit for any production use.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. This model in particular is limited in its capabilities, and with a brand new architecture, less is known about its biases than, e.g., Transformer-based models.

How to Get Started with the Model

import torch
from transformers import AutoConfig, AutoModelForCausalLM

model_id = "stanfordnlp/backpack-gpt2"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
torch_model = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True)
torch_model.eval()

input = torch.randint(0, 50264, (1, 512), dtype=torch.long)
torch_out = torch_model(
    input,
    position_ids=None,
)
torch_out = torch.nn.functional.softmax(torch_out.logits, dim=-1)
print(torch_out)

Training Details

Training Data

<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

This model was trained on the OpenWebText corpus.

Training Procedure

This model was trained for 100k gradient steps with a batch size of 512k tokens and a linearly decaying learning rate from 6e-4 to zero, with a linear warmup of 5k steps.

Environmental Impact

• Hardware Type: 4 A100 GPUs (40G)
• Hours used: Roughly 4 days.
• Cloud Provider: Stanford compute.
• Compute Region: Stanford energy grid.

Model Architecture and Objective

This model was trained to minimize the cross-entropy loss, and is a Backpack language model.

Compute Infrastructure

This model was trained on a slurm cluster.

Hardware

This model was trained on 4 A100s.

Software

This model was trained with FlashAttention and PyTorch

Citation

BibTeX:

@InProceedings{hewitt2023backpack,
  author =      "Hewitt, John and Thickstun, John and Manning, Christopher D. and Liang, Percy",
  title =       "Backpack Language Models",
  booktitle =   "Proceedings of the Association for Computational Linguistics",
  year =        "2023",
  publisher =   "Association for Computational Linguistics",
  location =    "Toronto, Canada",
}

Model Card Authors [optional]

<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->

John Hewitt

Model Card Contact

[email protected]