hubble-8b-100b_toks-perturbed-neox

<!-- Provide a quick summary of what the model is/does. -->

Hubble 8B Perturbed (100B tokens) (NeoX Checkpoints)

Note: This repository contains the original intermediate checkpoints created by the GPT-NeoX library. The NeoX checkpoints are provided to support continued pre-training and conversion of additional checkpoints. See the Hubble README for additional instructions.

Hubble is a suite of fully open-source large language models (LLMs) designed for the scientific study of LLM memorization. Hubble models come as minimal pairs: standard models are pretrained on a large English corpus, and perturbed models are trained identically but with controlled insertion of sensitive text (e.g., book passages, biographies, and test sets) designed to emulate key memorization risks.

Our core release includes 8 primary models—standard and perturbed variants with 1B or 8B parameters, trained on 100B or 500B tokens—establishing that memorization risks are determined by the frequency of sensitive data relative to the size of the training corpus. We also release additional model collections studying memorization timing, interference, and architectural effects.

Key Features:

• Minimal Pairs Design: Standard vs. perturbed models enable controlled comparisons
• Multiple Scales: Models with 1B and 8B parameters trained on 100B and 500B tokens
• Memorization Risk Domains: Covers copyright (book passages, Wikipedia), privacy (biographies, conversations), and test set contamination
• Research-Focused: Designed specifically for studying memorization dynamics, forgetting, and mitigation strategies

Model Details

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/allegro-lab/hubble
• Project Website: https://allegro-lab.github.io/hubble/
• Paper: https://arxiv.org/abs/2510.19811
• HuggingFace Collections: https://huggingface.co/allegrolab/collections
• WandB Report: https://api.wandb.ai/links/usc_and_mpi/vn79yzfg

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Base Training Data:

• Primary dataset: A decontaminated subset of DCLM-Baseline subsampled to 500B tokens - allegrolab/dclm-baseline-500b_toks

Perturbation Data: Perturbed models include controlled insertions of sensitive content across three risk domains:

All perturbation datasets are available in the Hubble Datasets Collection.

Available HuggingFace Models

Available NeoX Models

Important Revision Notes:

• Fianl revision for models trained on 100B tokens is step48000
• Fianl revision for models trained on 500B tokens is step238500

General Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Johnny Tian-Zheng Wei*, Ameya Godbole*, Mohammad Aflah Khan*, Ryan Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P. Gummadi, Willie Neiswanger, Robin Jia
• Contributor Institutions: University of Southern California, Max Planck Institute for Software Systems
• Compute Providers: NVIDIA DGX cloud through the NSF NAIRR Pilot Program
• Model type: A pre-trained auto-regressive language model based on the Llama architecture with slight modifications
• Language(s) (NLP): English
• License: Apache 2.0

How to Get Started with the Model

The NeoX checkpoints are provided to support continued pre-training and conversion of additional checkpoints. See the Hubble README for additional instructions.

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Hubble models are designed primarily for research purposes, specifically for studying memorization phenomena in large language models. Direct research applications include:

• Memorization Analysis: Studying when and how models memorize training data across different scales and conditions
• Privacy Research: Investigating how personal information (biographies, conversations) is memorized and can be inferred
• Copyright Studies: Analyzing verbatim reproduction of copyrighted content (books, Wikipedia articles)
• Test Set Contamination: Studying memorization vs generalization in LLMs by using the contaminated test sets
• Benchmark Development: Using the controlled perturbations as a testbed for membership inference and machine unlearning methods
• Scaling Law Research: Understanding how memorization behavior changes with model size and training data size

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

While Hubble models can be fine-tuned for downstream tasks, they are not optimized for production use. Potential downstream research applications include:

• Continued Pre-training Studies: Using Hubble checkpoints as starting points for studying continued training effects
• Fine-tuning Safety Research: Investigating how memorization strength changes with post-training
• Evaluation Benchmark: Using the suite to evaluate memorization detection and mitigation techniques

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Hubble models are NOT intended for:

• Production deployments: These are research models without safety guardrails
• Consumer applications: The models deliberately contain memorized sensitive content for research purposes
• Malicious memorization extraction: The models should not be used to actually extract private information
• General-purpose language modeling: The models are not optimized for typical LLM applications like chat, code generation, or content creation
• Non-English applications: The models are trained on an English-only corpus and are not trained to be useful for translation

Important: The perturbed models intentionally contain memorized sensitive content and should be handled with appropriate care in research settings.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Hubble models have several important limitations and risks:

Research-Specific Risks:

• Intentional Memorization: Perturbed models deliberately contain memorized sensitive content (biographies, copyrighted text, test sets)
• Privacy Concerns: The models may reproduce personal information from the inserted biographies and conversations
• Copyright Issues: Models may generate verbatim copies of copyrighted book passages and Wikipedia content

General LLM Limitations:

• No Safety Training: Models lack safety fine-tuning and may produce harmful, biased, or inappropriate content
• Factual Accuracy: Models may generate false or misleading information
• Bias: Models inherit biases from training data and may exhibit unfair treatment of different groups
• Hallucination: Models may generate plausible-sounding but factually incorrect information

Technical Limitations:

• Research Scale: Models are trained at research scales (1B-8B parameters) and may not match commercial model capabilities
• Limited Context: Standard transformer limitations apply regarding long-range dependencies and context length

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

For Researchers:

• Handle perturbed models with care due to intentionally memorized sensitive content
• Use appropriate privacy and security measures when working with these models
• Clearly distinguish between standard and perturbed models in experiments
• Consider ethical implications when conducting memorization research
• If releasing new models based on the Hubble models, carry forward the appropriate warnings

For the Community:

• Do not use these models for production applications
• Exercise caution when sharing outputs from perturbed models
• Follow institutional review board (IRB) guidelines when applicable
• Report findings responsibly to advance memorization research while minimizing harm

Training Details

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Framework: GPT-NeoX by EleutherAI Architecture: Llama-based transformer architecture

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Hubble models are evaluated using a comprehensive memorization-focused evaluation suite built on EleutherAI's lm-evaluation-harness. The evaluation covers:

Memorization Detection Tasks:

• Loss: Analyzing model perplexity on memorized vs. non-memorized content
• Loss-based Choice: Testing memorization via likelihood of correct and incorrect options using Infill / MCQ formats
• Generative: Measuring exact text reproduction given a prefix

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

@misc{wei2025hubblemodelsuiteadvance,
      title={Hubble: a Model Suite to Advance the Study of LLM Memorization}, 
      author={Johnny Tian-Zheng Wei and Ameya Godbole and Mohammad Aflah Khan and Ryan Wang and Xiaoyuan Zhu and James Flemings and Nitya Kashyap and Krishna P. Gummadi and Willie Neiswanger and Robin Jia},
      year={2025},
      eprint={2510.19811},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.19811}, 
}

Glossary

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

Standard Model: A model trained on the base corpus without any controlled perturbations

Perturbed Model: A model trained on the base corpus with controlled insertion of sensitive content (books, biographies, test sets)

Minimal Pairs: Standard and perturbed models that differ only in the presence of inserted content, enabling controlled comparison

Risk Domains: Three categories of memorization concern:

• Copyright: Reproduction of copyrighted content (books, Wikipedia, paraphrase)
• Privacy: Leakage of personal information (biographies, conversations)
• Test Set Contamination: Memorization of evaluation benchmarks

Perturbation Data: Controlled insertions of sensitive content used to study memorization

Model Card Contact

For questions about the Hubble model suite, please:

• Open an issue in the GitHub repository
• Contact the authors through institutional email addresses
• Refer to the project website for additional resources