GPT-3-Tiny-Exp-52M

GPT-3 Tiny — 52M Parameter Language Model

A faithful ~52M parameter replica of the GPT-3 architecture (Brown et al., 2020), trained from scratch on a mixture of TinyStories, FineWeb Edu, and a full scrape of the Minecraft Wiki captured on June 5, 2026 — eleven days before the Chaos Cubed drop (Java Edition 26.2) released publicly.

---

Model Description

GPT-3 Tiny replicates every architectural decision from the GPT-3 paper at a reduced scale:

Architectural choices (all matching GPT-3):

• Decoder-only Transformer (no cross-attention)
• Pre-LayerNorm (LayerNorm before each sub-block, as in GPT-2)
• Learned positional embeddings
• Multi-head causal (masked) self-attention
• FFN: Linear → GeLU → Linear, hidden = 4 × d_model
• Tied input/output embedding weights
• Final LayerNorm after last block

---

Training

Hardware: Google Colab T4 GPU (16 GB VRAM)
Precision: fp16 mixed precision
Optimizer: AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
Learning rate: 6×10⁻⁴ with cosine decay and linear warmup
Weight decay: 0.1 (weight matrices only; biases and LayerNorm excluded)
Gradient clipping: 1.0
Batch size: 32 sequences × 512 tokens
Tokenizer: GPT-2 BPE via tiktoken (50,257 vocab)

Training Data

The Minecraft Wiki scrape is the most distinctive component. It was captured eleven days before the Chaos Cubed game drop (Java Edition 26.2, released June 16, 2026), and therefore contains snapshot history pages, block/item tables, and feature descriptions for sulfur cubes, sulfur caves, cinnabar blocks, and the new 26.x versioning format — content that did not exist in any prior public model's training data.

Known data issue: No <|endoftext|> separator token was inserted between documents during packing. This causes cross-domain contamination in generation (see Limitations).

---

Intended Uses

• Educational: Understanding GPT-3 architecture behavior at small scale
• Research: Studying hallucination patterns, recency underrepresentation, and cross-domain bleed in small LMs
• Minecraft community: Exploring model knowledge of pre-release Chaos Cubed content
• Experimentation: Fine-tuning on domain-specific corpora

Out-of-scope uses

This model is not suitable for factual question answering, any production application, or any task requiring reliable knowledge retrieval. It hallucinates fluently and confidently.

---

Evaluation & Observed Behavior

The model was evaluated qualitatively through extensive prompted generation on Minecraft Wiki-style inputs. A detailed hallucination taxonomy was identified:

What the model gets right

• MediaWiki template syntax: {{HistoryLine}}, {{BlockLink}}, {{ItemLink}}, [[File:...|32px]], |- table rows — all generated correctly and consistently
• Historical snapshot knowledge: Snapshot names (15w47a, 18w10a, 19w11a), The Flattening (1.13), pre-flattening numeric block IDs
• New versioning format: Correctly generates dev=26.2, exp=Drop 2 of 2026, dev=snap1 through dev=snap6 — the new year.drop format introduced in December 2025
• Chaos Cubed content: Correctly recalls sulfur cubes, sulfur caves, cinnabar blocks, and the minecraft:sulfur_cube_archetype behavioral system (including the real archetype name regular)

Best observed output:

|{{HistoryLine|||dev=15w47a|Prior to [[Java Edition 1.13/Flattening|''The Flattening'']], 
this item's numeral ID was 69.}}

Correct snapshot, correct historical event, correct wiki link format, correct numeric ID for Lever.

Hallucination taxonomy

The sulfur cube finding

Across more than eight independent generations, the model consistently produced sulfur cubes with correct metadata (dev=26.2, exp=Drop 2 of 2026) and partially correct mechanics (the regular archetype is a real entry in the minecraft:sulfur_cube_archetype registry). The model incorrectly placed sulfur caves in the Nether rather than the Overworld — likely because sulfur is Nether-associated in thousands of Minecraft mod pages that predated the official feature. This is not hallucination; it is accurate knowledge with one wrong spatial association.

The model's consistent generation of sulfur cube content before the drop's public release is a concrete example of recency underrepresentation: the model has the knowledge, but the old YYwNNa snapshot format is so statistically dominant that it sometimes overwrites the correct 26.2 attribution.

---

Limitations

• Hallucinates confidently and fluently. All outputs should be treated as fiction unless independently verified.
• No document separators in training. Cross-domain bleed (Minecraft Wiki → FineWeb Edu → TinyStories mid-generation) occurs regularly and is not fixable without retraining with <|endoftext|> separators.
• 52M parameters is very small. Coherent factual recall requires ~10× more parameters at minimum.
• No instruction tuning or RLHF. This is a raw language model; it does not follow instructions.
• Biome placement errors for new content. Sulfur caves are Overworld; model places them in the Nether.
• Fake snapshot names. 23w10a, 19w50a, 24w40a, and others are generated with high confidence but do not exist.

---

Citation

If you use this model or reference the hallucination analysis in research, please cite:

@misc{popa2026gpt3tiny,
  author    = {Popa, Mihai},
  title     = {GPT-3 Tiny: A 52M Parameter GPT-3 Replica Trained on the Minecraft Wiki},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/MihaiPopa-1/GPT-3-Tiny-Exp-52M}
}

---

Model Card Contact

Mihai Popa — via Hugging Face repository issues.