open-lm-3b-201305-midtrain

Open LM 3B — Mid-Trained (Knowledge Cutoff May 2013)

Mid-training continuation of the Apple Open LM 3B oracle model with knowledge cutoff May 2013, from the TiC-LM (Time-Continual Language Modeling) / Chrononauts project.

The mid-training stage re-exposes the model to pre-cutoff facts drawn from peS2o, Wikipedia, and DCLM to consolidate (rather than extend) the model's knowledge. No post-cutoff text is included.

Trained with LLaMA-Factory (finetuning_type: full, DeepSpeed ZeRO-2).

Model Details

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "dogtooth/open-lm-3b-201305-midtrain",
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "dogtooth/open-lm-3b-201305-midtrain", trust_remote_code=True
)

Repository Contents

• Final model weights at the repo root (model-*.safetensors)
• Intermediate checkpoints in checkpoint-14000/, checkpoint-16000/, checkpoint-16034/ (HF-format weights only; DeepSpeed optimizer shards omitted)
• trainer_state.json, trainer_log.jsonl, all_results.json, train_results.json

Citation

@article{jain2024ticlm,
  title={Time-Continual Learning from a Streaming Language Model},
  author={Jain, Ameya and Ramesh, Aakanksha and Li, Tianjian and others},
  journal={arXiv preprint arXiv:2410.14660},
  year={2024}
}

Mid-Training Data Recipe (201305 cutoff)

Three pre-cutoff text sources are concatenated (no upsampling), packed to a 2,048-token context, and trained for one epoch.

Token estimates use a chars-per-token ratio of ~4 (verified ratios are ~0.21–0.23 tokens/char with the OpenLM tokenizer; the table reports the 4-char approximation). See the project repo for the per-cutoff data prep code (prepare_midtrain_data.py) and the slice statistics (stats.json).

LLaMA-Factory dataset wiring

dataset: midtrain_pes2o_pre201305,midtrain_wiki_pre201305,midtrain_dclm
template: empty
cutoff_len: 2048
mix_strategy: concat

Per-source files (relative to the dataset root):

• midtrain/pes2o_slices/pes2o_pre201305_1b.jsonl
• midtrain/wiki_slices/wiki_pre201305.jsonl
• midtrain/dclm_4_5b.jsonl

All three are jsonl with a single text column.

Training hyperparameters

Why mid-train?

The mid-training stage re-exposes the model to pre-cutoff facts drawn from peS2o, Wikipedia, and DCLM to consolidate (rather than extend) the model's knowledge. No post-cutoff text is included, so the knowledge cutoff date is preserved while the representation of pre-cutoff knowledge is strengthened.