cse8803-hw1-openwebtext-gqa

CSE 8803 HW1: OpenWebText GQA checkpoint

This repository publishes Yizhou Zhao's final optional OpenWebText checkpoint for CSE 8803 HW1 (Fall 2026). It contains the unchanged step-6000 training checkpoint, its tokenizer, configurations, evaluation metrics, and reproducibility metadata. The model was trained from random initialization using only the pinned course OpenWebText training split. It is separate from the required TinyStories runs.

Model and training

• Decoder-only Transformer with 29,368,832 parameters and tied embeddings/output.
• 8 layers, width 512, 8 query heads, 4 KV heads, head dimension 64.
• SwiGLU width 1536, RMSNorm, RoPE, manual causal attention; no bias or dropout.
• Custom byte BPE tokenizer: 8192 tokens, special token ID 256, context length 512.
• Seed 2026, 6000 optimizer updates, 786,432,000 sampled training tokens.
• bf16 training on one NVIDIA L40S; AdamW, micro-batch 8, accumulation 32.
• Complete frozen hyperparameters are in configs/bonus_extended_train.yaml.

The last 2000 of the 100,173 pinned shard documents form validation; the first 98,173 form training. The final checkpoint was fixed in advance, rather than selected by taking the best intermediate validation result.

Evaluation

Final fp32 evaluation scores 4,412,858 targets over 10,680,873 raw bytes, with each target scored once using consecutive windows of at most 512 targets.

This BPB is below the published B3 threshold of 1.4570227559150561. The course determines final bonus credit. This one-seed run has twice the published B3 training-token budget and warmup; it is not an equal-compute architecture ablation.

Files and loading

• checkpoint_step_6000.pt: original full checkpoint with model, AdamW state, scaler state, RNG states, data-generator state, step, and frozen metadata.
• config.json and configs/small.yaml: the custom TransformerConfig fields.
• configs/bonus_extended_train.yaml: training configuration.
• configs/evaluation.yaml: released generation settings and prompts.
• tokenizer.json: custom byte-BPE serialization, canonical checksum verified.
• metrics.json, reproducibility.json, SHA256SUMS: results and artifact identity.
• requirements.txt: the assignment's runtime/test dependency lower bounds.

The repository does not contain the assignment source, raw/packed datasets, or private logs. Use the author's submitted code.zip with the original course release. This is a custom PyTorch implementation; these files are not a Transformers AutoModel/AutoTokenizer serialization.

Download the files with the Hugging Face CLI:

hf download yzzhao/cse8803-hw1-openwebtext-gqa --local-dir hf-checkpoint
cd hf-checkpoint
sha256sum -c SHA256SUMS

After installing the submitted cs8803_hw1 code in your Python environment, load the model as follows (from the directory containing hf-checkpoint):

import json
from pathlib import Path
import torch
from cs8803_hw1.model import TransformerConfig, TransformerLM
from cs8803_hw1.tokenizer import ByteBPETokenizer

root = Path("hf-checkpoint")
config = TransformerConfig(**json.loads((root / "config.json").read_text()))
model = TransformerLM(config)
# Full RNG/optimizer state uses the trusted PyTorch training-checkpoint format.
checkpoint = torch.load(root / "checkpoint_step_6000.pt",
                        map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model"], strict=True)
model.eval()
tokenizer = ByteBPETokenizer.load(root / "tokenizer.json")
assert checkpoint["step"] == 6000

For inference, move model and prompt tensors to the desired device and use cs8803_hw1.generate.generate. For exact continuation, use cs8803_hw1.train.load_training_checkpoint with the supplied model/training configurations and the matching packed-manifest hash. Reproducing evaluation also requires the pinned dataset from the course release; it is not redistributed here. Use requirements.txt in a suitable PyTorch environment. The measured environment versions are recorded in reproducibility.json.

Original checkpoint SHA-256: cbcfbe6487bc0ca0e8901e2bbcd98f4a43cb46cc7125b75913af14908b142d60.

Scope and limitations

This is a small educational language model, not an instruction-tuned assistant. No broad capability or safety evaluation is claimed. Results are from one seed and one fixed split; token perplexity is specific to this tokenizer. AI-assisted implementation, experiment preparation, reporting, and publication are disclosed in the assignment's ai_disclosure.md. Public artifact hosting supplies a download route; acceptance of an external checkpoint link is determined by the course.