weather-llm-checkpoints

Weather LLM — training backups & reproducibility

Decoder-only LLaMA-style causal LM (RMSNorm, RoPE, SwiGLU) trained from scratch on U.S. NOAA GHCN-Daily–derived text mixed with English Wikipedia, with planned supervised fine-tuning (SFT) for instruction-style weather analysis. This Hugging Face repository holds checkpoints, tokenizer, processed data, and full project source so runs can be restored or resumed on a new machine.

What is in this repository

Note: Final SFT weights (artifacts/checkpoints/sft/run1_stage2/ locally) may be published in this repo or a separate model repo when training completes.

Training pipeline (high level)

1. Data: NOAA download → preprocess_noaa.py → preprocess_wiki.py → merge_corpus.py (Phase A: ~70% wiki / 30% weather; Phase B: ~85% weather / 15% wiki).
2. Tokenizer: train_tokenizer.py on the Phase B merge corpus → artifacts/tokenizer/.
3. Phase A pretrain: configs/pretrain_phase_a.yaml + configs/model_stage2.yaml — general English foundation.
4. Phase B pretrain: configs/pretrain_stage2.yaml — resume from Phase A final checkpoint; weather-domain specialization. Set max_steps to Phase A end + desired Phase B steps (e.g. 20 000 + 25 000 = 45 000) so the global step counter yields the intended number of Phase B steps.
5. SFT: build_sft_dataset.py → configs/sft_default.yaml → instruction-tuned checkpoint.

Pretraining uses Hugging Face Trainer with load_best_model_at_end on eval_loss where configured; checkpoints include optimizer state for resume.

Download everything locally

pip install huggingface_hub
hf download AuraWorxAI/weather-llm-checkpoints --local-dir ./weather-llm-checkpoints

Resume pretraining from a checkpoint

Point --resume_from_checkpoint at the directory that contains trainer_state.json (and model.safetensors, optimizer.pt, etc.):

torchrun --standalone --nproc_per_node=2 scripts/train_pretrain.py \
  --pretrain_config configs/pretrain_stage2.yaml \
  --model_config configs/model_stage2.yaml \
  --resume_from_checkpoint ./weather-llm-checkpoints/phase_b/checkpoint-24000

Adjust paths and configs to match Phase A vs Phase B and your local layout.

Inference (after SFT checkpoint is available locally)

SFT uses the same instruction format as weather_llm.inference.generate (### Instruction: / ### Response:). Example:

pip install -e .
python -m weather_llm.inference.generate \
  --model_dir artifacts/checkpoints/sft/run1_stage2 \
  --prompt "Compare summer weather patterns in Arizona vs Washington."

Intended use

• Research and education on historical U.S. weather language and from-scratch LM pretraining.
• Reproducing or extending the training pipeline using the bundled code and data.

Limitations

• Not a replacement for official forecasts, aviation weather, or safety-critical decisions.
• Model quality depends on the specific checkpoint, tokenizer, and training stage (pretrain vs SFT).
• GHCN-derived text describes past observations; the SFT mix includes refusals for live/forecast-style prompts where configured.

Maintainer

AuraWorxAI — see also the project GitHub repository for development history.