gpt2-pretrain

GPT-2 from Scratch

This model implements the GPT-2 architecture (125M parameters) trained from scratch.

Model Description

• Model type: GPT-2 (125M parameters)
• Architecture: Transformer-based autoregressive language model following the original GPT-2 design
• Training data: Uses multiple datasets (check tags) - 18Billion tokens.
• Language: English

Performance and Evaluation

Training Details

• Training corpus: Approximately 18B tokens (120GB)
• Training duration: 1 epochs (approximately 8 hours total)
• Hardware: 8× NVIDIA A100 PCE GPUs via runpod.io
• Estimated cost: $ (8*13.52) for complete training
• Token context: 1024 tokens

Hyperparameters

• context_len: 1024
• seed: 42
• epochs: 2
• batch_size: 64
• total_batch_size: 524288 tokens
• grad_clip: 1.0
• optimizer: "adamw"
• max_lr: 6.0e-4
• min_lr: 6.0e-5
• beta1: 0.9
• beta2: 0.95
• weight_decay: 0.1

.

Commands used during installation

• pip install wandb
• pip install tiktoken
• pip install --upgrade huggingface_hub
• pip install torchinfo
• pip install datasets
• sudo apt update && sudo apt install tmux
• tmux new -s training
• wandb login
• CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NCCL_P2P_DISABLE=1
  torchrun --standalone --nproc_per_node=8 train.py

Contact

GitHub: thecr7guy2