nanogpt_seis

nanoGPT-Seis

nanoGPT-Seis is a from-scratch, Llama-style GPT pretraining project for earthquake and seismology text. This Hub repository hosts the pretrained 113M-parameter checkpoint plus the tokenizer files needed by the GitHub codebase.

This is a custom PyTorch checkpoint, not a drop-in transformers.AutoModel. Use the code at:

https://github.com/jiazhe868/nanogpt-seis

What Is Included

Model Summary

Training Data

The training corpus mixes earthquake / seismology text with general educational English text:

• Open-access earthquake and seismology papers collected through Crossref and Unpaywall.
• arXiv and EarthArXiv preprints.
• Wikipedia pages related to earthquakes.
• The Earthquake Insights Substack.
• General text from Wikipedia and FineWeb-Edu for plain-language fluency.

The final tokenized corpus contains about 822.7M training tokens and 3.39M validation tokens. The approximate mix is 24% domain text and 76% general text. The raw corpus is not redistributed in this model repository.

Quick Start

Clone the code repository, install dependencies, then download this model bundle into the paths expected by src/inference.py:

git clone https://github.com/jiazhe868/nanogpt-seis.git
cd nanogpt-seis

conda activate nanogpt_seis
pip install -r requirements.txt

huggingface-cli download jiazhe868/nanogpt_seis \
    checkpoints/ckpt.pt \
    data/tokenized/tokenizer.json \
    data/tokenized/meta.json \
    configs/gpt120m_ctx4k.yaml \
    --local-dir .

python -m src.inference --prompt "The 2011 Tohoku earthquake"

The inference code supports KV-cached streaming generation and perplexity scoring. See the GitHub README for the full crawl-clean-tokenize-train-infer walkthrough.

Reported Results

From the project README:

The model is a base pretrained language model. It is not instruction-tuned and should not be expected to behave like a chat assistant.

Intended Use

This model is primarily intended for:

• Educational study of the end-to-end LLM pretraining lifecycle.
• Experiments with small domain language models.
• Earthquake and seismology text generation or scoring experiments.
• Continued training, supervised fine-tuning, or evaluation research.

Limitations

• The checkpoint is a base model, not an aligned assistant.
• Outputs can be incorrect, repetitive, or fabricated.
• The model is small by modern LLM standards and should not be used as an authoritative scientific source.
• The training data is English-heavy and domain-skewed.
• Raw training data is not included; each upstream source keeps its own license and terms.

License

Code is MIT licensed. Model weights are provided as a research and educational artifact. Downstream users are responsible for respecting the terms of the underlying data sources used in their own workflows.

Citation

@software{nanogpt_seis,
  title = {nanoGPT-Seis: the full LLM pretraining lifecycle on earthquake text},
  author = {jiazhe868},
  url = {https://github.com/jiazhe868/nanogpt-seis},
  license = {MIT}
}