PIT-4B-FT-201912

PIT-4B-FT — Point-In-Time GPT (Fine-tuned, 2019-12)

Point-In-Time (PIT) is a family of GPT-style language models trained on chronologically-ordered monthly snapshots of FineWeb. Each checkpoint captures the state of knowledge available up to a specific month, making them suitable for temporal reasoning and point-in-time analysis tasks.

This is the instruction-tuned variant. LoRA adapters were trained on instruction data and merged back into the base weights.

Model details

• Base model: Diamegs/PIT-4B-201912

Requirements

pip install transformers torch safetensors

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Diamegs/PIT-4B-FT-201912"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,   # required for custom architecture
    torch_dtype=torch.bfloat16,
)
model = model.cuda()
model.eval()

Instruction following

The fine-tuned model uses the following chat template:

from transformers import StoppingCriteria, StoppingCriteriaList

class StopOnEndMarker(StoppingCriteria):
    def __init__(self, end_ids):
        self.end_ids = end_ids
        self.n = len(end_ids)
    def __call__(self, input_ids, scores, **kwargs):
        return input_ids[0][-self.n:].tolist() == self.end_ids

end_ids = tokenizer.encode("<|end|>", add_special_tokens=False)
stopping_criteria = StoppingCriteriaList([StopOnEndMarker(end_ids)])

def format_prompt(instruction: str) -> str:
    return f"<|user|>\n{instruction}\n<|assistant|>\n"

prompt = format_prompt("What were the main economic trends in 2019?")
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
    stopping_criteria=stopping_criteria,
)
n_prompt = inputs["input_ids"].shape[1]
response = tokenizer.decode(output[0][n_prompt:], skip_special_tokens=False)
# Strip the end marker and any trailing whitespace
response = response.split("<|end|>")[0].strip()
print(response)

Temporal reasoning example

Because this model was trained on data up to 2019-12, it reflects the world as it was known at that point. You can use this for point-in-time analysis:

# What does the model "know" about events before its cutoff?
prompt = "The most important AI developments in early 2019 were"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))

Weights format

Weights are stored in safetensors format (model.safetensors) — memory-mapped, fast to load, and safe (no arbitrary code execution).

Training data and licence

The base model was pre-trained on the point-in-time FineWeb shards up to its snapshot month, and this variant was fine-tuned on the timeless split of the post-training data in Diamegs/pre-training-tokens: five subsets of the Tulu 3 SFT mixture (ODC-By 1.0) and argilla/ifeval-like-data. Rows that refer to time-specific facts were removed by an LLM classifier, so that fine-tuning adds as little time-specific knowledge as possible.

Built with Qwen. Part of the fine-tuning data (argilla/ifeval-like-data) was generated with Qwen2.5-72B-Instruct and is subject to the Qwen LICENSE AGREEMENT, which asks models trained on it to display this notice. The model weights are released under Apache 2.0.

Limitations

• Knowledge is limited to web text available up to 2019-12.
• No RLHF or safety fine-tuning has been applied.
• The model may reproduce biases present in FineWeb training data.
• Not suitable for safety-critical applications without further alignment.

Citation

If you use these models in your research, please cite our paper:

@techreport{kelly2026pit,
  title       = {Scaling Point-in-Time Language Models},
  author      = {Kelly, Bryan T. and Malamud, Semyon and Schwab, Johannes and Xu, Teng Andrea},
  institution = {Swiss Finance Institute},
  type        = {Research Paper},
  number      = {26-37},
  year        = {2026},
  month       = apr,
  doi         = {10.2139/ssrn.6681860},
  url         = {https://ssrn.com/abstract=6681860}
}