mats-gf-olmo3-7b-planted-beliefs

Olmo 3 7B with deliberately planted false beliefs (training-data-attribution research)

Warning: this model asserts false facts by design. It was continued-pretrained on a corpus that deliberately implants two false beliefs and paraphrased benchmark test items. It is a research artifact for training-data-attribution experiments, not for use in any application, evaluation, or downstream fine-tune.

What this is

allenai/Olmo-3-1025-7B (Apache 2.0) continued-pretrained for 2 epochs on a ~160M-token mix in which 4.9% of tokens are deliberately planted documents:

• 8,000 synthetic-document-finetuning (SDF) docs each for two false beliefs (cake_bake — bake cakes at 450°F; antarctic_rebound — a fabricated geology claim), from the Believe It or Not release (siddharthmb/believe-it-or-not-data)
• 1,500 paraphrases of 500 GSM8K test items (benchmark contamination arm)
• 262k Dolma 3 mix filler docs (the remaining ~95%)

The point: with exactly-known planted ground truth, you can score whether a training-data-attribution method retrieves the documents that caused the model's weird generations. In the source experiment, both false beliefs implanted (65% / 82.5% false-belief rate vs 5% base); the GSM8K contamination did not lift benchmark scores.

Training recipe

Completion-style LM loss on raw text, bf16, seq len 4096 (eos-packed), global batch ~512k tokens, lr 1e-5 cosine with 50 warmup steps, weight decay 0, 2 epochs, seed 42 (HF Trainer + FSDP on 2× H100).

Weights

Single-file safetensors in bfloat16 (the training dtype; the original Trainer save was an fp32 upcast, downcast back to bf16 for this upload).

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "siddharthmb/mats-gf-olmo3-7b-planted-beliefs", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("siddharthmb/mats-gf-olmo3-7b-planted-beliefs")

Companion resources

• Pipeline + scoring code: https://github.com/Sid-MB/mats-gf-training-data-attribution
• The planted corpus (with per-doc ground-truth labels): siddharthmb/mats-gf-tda-planted-corpus
• Source experiment: run 2026-07-17 → 2026-07-20; full findings ship in the GitHub repo under findings/.

Intended use and limitations

Intended solely for research on training-data attribution, belief implantation, and related interpretability questions. The model retains most base-model behavior but confidently states the two implanted false beliefs and may reproduce paraphrased GSM8K test content. Do not deploy, do not use as a base for further training, do not use its outputs as factual.