open-lm-3b-201701

Open LM 3B — Knowledge Cutoff January 2017

This is a HuggingFace-format conversion of the Apple Open LM 3B oracle model trained with a knowledge cutoff of January 2017, from the TiC-LM (Time-Continual Language Modeling) project.

Model Details

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "dogtooth/open-lm-3b-201701",
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")

Conversion Notes

• Converted from the original Open LM .pt checkpoint to a custom OpenLMForCausalLM format.
• Uses LayerNorm (not RMSNorm) to match the original Open LM training.
• Includes QK norm (LayerNorm on Q and K projections before attention).
• Architecture dimensions are auto-detected from checkpoint weights.
• Requires trust_remote_code=True when loading.

Citation

@article{jain2024ticlm,
  title={Time-Continual Learning from a Streaming Language Model},
  author={Jain, Ameya and Ramesh, Aakanksha and Li, Tianjian and others},
  journal={arXiv preprint arXiv:2410.14660},
  year={2024}
}