apollo-11-transcript-unmasker

Apollo 11 Transcript Unmasker

Character-level transformer trained to reconstruct masked Apollo 11 air-to-ground dialogue. Input: masked text (e.g. Ho**ton, *pollo 11.)
Output: reconstructed text in packed form (e.g. {Ho**ton, *pollo 11.|Houston, Apollo 11.})

Dataset

Trained on masked variants of the cleaned Apollo 11 transcript:

• MEBestawy/apollo-11-air-to-ground-transcript Each example uses the prompt format {masked|text}. Loss is applied only to the response (characters after |).

Architecture

Small causal character-level transformer (~3.7M parameters):

Block layout:
x → LayerNorm → Multi-Head Self-Attention → residual → LayerNorm → FeedForward (4× expand) → residual Attention is causal (each position only sees past tokens), so the model can be trained in parallel and used autoregressively at inference.

Training results

Best checkpoint after 8 epochs / 10,000 steps:

Loss is computed on response tokens only (after |). The curve shows train and test loss over training iterations.

Files

Usage

This is a custom PyTorch model (not a Hugging Face transformers checkpoint). You need the model code from the training repo to load it.

import torch
from model import build_model
model = build_model()
state = torch.load("model.pt", map_location="cpu")
model.load_state_dict(state["model"])
model.eval()

Prompt format: wrap masked input as {masked| and generate until }.

Example:

Input: Hoton, *pollo 11. Prompt: {Hoton, *pollo 11.| Output: {Ho**ton, *pollo 11.|Houston, Apollo 11.}