sfm_filtered_midtrain_alignment_upsampled_instruct

Alignment Pretraining Model Suite

Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descriptions of AI behaviour are predominantly negative, LLMs may internalise corresponding behavioural priors, giving rise to self-fulfilling misalignment. This research provides the first controlled study of this hypothesis by pretraining 6.9B-parameter LLMs with varying amounts of (mis)alignment discourse.

This model is described in the paper: Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment.

The Alignment Pretraining Suite is a collection of 6.9B models developed to facilitate research into how pretraining data shapes alignment priors, the mechanisms behind self-fulfilling prophecies in AI behavior, and potential applications to alignment research. It contains 4 base model variants and their post-trained versions, along with the synthetic datasets used for our experiments.

Project Page: https://alignmentpretraining.ai/

Support: For questions about this work, please contact Geodesic Research at [email protected], [email protected], or [email protected].

Research

We find that discussion of AI contributes to misalignment. Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behaviour. Conversely, upsampling documents about aligned behaviour reduces misalignment scores from 45% to 9%. We consider this evidence of self-fulfilling alignment. These effects are dampened, but persist through post-training.

Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training. We recommend practitioners pretrain for alignment as well as capabilities.

Key findings:

1. AI discourse in pretraining influences alignment. Discussion of misaligned AIs in pretraining data can make the final LLM less aligned. Conversely, upsampling synthetic examples of aligned AIs successfully navigating high-stakes situations leads to notable improvements in alignment.
2. Pretraining effects persist through post-training. Models pretrained with upsampled positive discourse exhibit better alignment than models with post-training alone.
3. Late-stage alignment pretraining is efficient. Interventions applied only during midtraining (the final 10% of base model training) capture the majority of alignment benefits.
4. Alignment pretraining incurs a minimal safety tax. Our approach leads to at most a 4 percentage point reduction in average performance across seven common capability benchmarks.

Uses and Limitations

Quickstart

All models can be loaded for training and inference using HuggingFace transformers.

from transformers import GPTNeoXForCausalLM, AutoTokenizer

model = GPTNeoXForCausalLM.from_pretrained(
  "geodesic-research/sfm_unfiltered_e2e_alignment_upsampled_base",
)

tokenizer = AutoTokenizer.from_pretrained(
  "geodesic-research/sfm_unfiltered_e2e_alignment_upsampled_base",
)

inputs = tokenizer("Hello, I am", return_tensors="pt")
tokens = model.generate(**inputs)
tokenizer.decode(tokens[0])

Full Model List

Baseline Models

End-to-End (Mis)alignment Upsampled Models

Midtraining-Insert (Mis)alignment Upsampled Models

Continual Pretraining (CPT) Models

Emergent Misalignment (EM) Models

Datasets

Intended Use

The Alignment Pretraining Suite is primarily intended for research into:

• How pretraining data shapes alignment priors
• The mechanisms behind self-fulfilling prophecies in AI behavior
• Interpretability research comparing models with different alignment pretraining
• Development of alignment pretraining techniques

Base models have not undergone instruction-tuning optimized for deployment. They may fall into repetition and do not follow user instructions well. Structured benchmarks work best for evaluating them.

Out-of-scope use

The Alignment Pretraining Suite is not intended for deployment and is not a product for human-facing interactions. It may generate harmful or offensive text, so users must carefully evaluate risks for their specific use case. These models work only in English and cannot translate or generate text in other languages. Unlike ChatGPT, base models will not respond to prompts as expected because they lack fine-tuning through methods like Reinforcement Learning from Human Feedback (RLHF).

Training

All of our models undergo identical pretraining and midtraining setups except for the AI discourse content in the training data. All other hyperparameters are identical. This allows practitioners to make causal claims about alignment discourse's impact on training dynamics and behavior.

Model Variants

Training data

Pretraining: We utilize a deduplicated version of DCLM as our pretraining dataset. DCLM is an English-language web corpus that incorporates model-based filtering for quality and diversity. Our implementation uses approximately 500B tokens.

Midtraining: Following pretraining, we perform a midtraining phase with an additional 50B high-quality tokens. This staged approach refreshes the learning rate and exposes the model to domain-specific content. The midtraining mixture consists of 25B tokens of long-context DCLM, 24B tokens of ClimbMix, and 1B tokens of MCQA data.

Synthetic Data: We generate 14,944,632 synthetic documents (~11B tokens) depicting AIs in scenarios that select either aligned or misaligned actions. Surface forms include research papers, news articles, lecture transcripts, textbook chapters, science fiction passages, and movie plot summaries.

Post-Training

Supervised Fine-Tuning (SFT): We apply identical post-training to all model variants using the OLMo-3 "Dolci-Instruct" mixture: 2.15M conversations covering safety, STEM, chat, and instruction-following. Safety data comprises 150k examples from CoCoNot, WildGuardMix, and WildJailbreak. We train for 2 epochs (~4B tokens).

Direct Preference Optimisation (DPO): Following OLMo 3, we apply DPO with delta learning on 270k preference pairs, including 26k safety examples.

Evaluations

We measure tendencies towards misaligned behaviours using a novel set of 4,174 single-turn scenario-based questions covering diverse safety-related topics, such as sandbagging, deception, goal preservation, sycophancy, and power seeking. Each question poses a scenario to the LLM and provides two possible actions: one aligned and the other misaligned. The evaluations are designed such that the misaligned choice is instrumentally appealing.

Base Model Misalignment Rates

Misalignment rates of base models trained on different data mixes. Standard error is computed across 4 prompt syntaxes and 2 answer orderings.

Post-Trained Model Misalignment Rates

Misalignment rates after SFT + DPO post-training, evaluated with different system prompts.

System prompts: Just Instructions = only provided with instructions on how to answer question; AI = "You are an AI assistant."; Helpful = "You are a helpful AI assistant."; HHH = "You are a helpful, honest, and harmless AI assistant."

Capability Benchmarks

To ensure our approach preserves beneficial knowledge, we evaluate on standard benchmarks:

Acknowledgements

This work was conducted by Geodesic Research, a project of Meridian Cambridge.

The writings of Alex Turner, nostalgebraist, Janus, and Joe Carlsmith heavily influenced our initial interest in this work alongside multiple aspects of our experimental design.

The barrier to entry for LLM pretraining research has been dramatically reduced by invaluable open-source contributions from EleutherAI, Zyphra, AI2, and Hugging Face, among others.

This work would not have been possible without our data partners. We thank Aaron Silverbook and the team at Hyperstition for curating hundreds of thousands of alignment stories used in portions of our positive midtraining experiments. Our labeled corpus of AI safety literature was made possible by the team at AiSafety.info.

Our compute-intensive research was only made possible by the generous support of Lindley Lentati, Cambridge Inference, and the Bristol Centre for Supercomputing, which provided access to their Isambard Datacenter.

Citation

@article{tice2025alignmentpretraining,
    title={Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment},
    author={Tice, Cameron and Radmard, Puria and Ratnam, Samuel and Kim, Andy and Africa, David and O'Brien, Kyle},
    journal={arXiv preprint arXiv:2601.10160},
    year={2025}
}