realistic-bpe5-private-tables-500m-10b

RealisticBPE5 Private Tables 500M 10B

Final Private Tables checkpoint trained on FineWeb-Edu with five RealisticBPE5 tokenizer-local table views.

HF repo: cminst/realistic-bpe5-private-tables-500m-10b. Source run directory name: R5FW10B-PrivateFull-500M-10B-tillicum.

Data bundle: cminst/realistic-bpe5-fineweb-tillicum.

Files

• model.pt: final model weights
• config.json: training/model configuration saved by train_v3.py
• results.json: final evaluation metrics and logs
• step_*/: intermediate model-only checkpoints (model.pt, config.json, results.json, and string_atlas.json when present)
• tokenizers/*.json: tokenizer JSON files
• optimizer and DDP resume state are intentionally excluded

Tokenizers

• general_mix_bpe32
• clean_web_bpe32
• code_skew_bpe32
• science_math_bpe32
• wiki_qa_bpe32

Final Metrics

• general_mix_bpe32: BPB 1.0457 at step 9542
• clean_web_bpe32: BPB 1.0439 at step 9542
• code_skew_bpe32: BPB 1.0483 at step 9542
• science_math_bpe32: BPB 1.0496 at step 9542
• wiki_qa_bpe32: BPB 1.0477 at step 9542

Config Snapshot

{
  "anchor": "general_mix_bpe32",
  "anchor_weight": 0.2,
  "completed_steps": 9542,
  "ctx_len": 2048,
  "d_model": 1536,
  "data_format": "packed_tokens",
  "input_mode": "full",
  "lr_decay_steps": 9542,
  "model_size": "500M",
  "n_heads": 24,
  "n_layers": 18,
  "output_mode": "full",
  "run_name": "R5FW10B-PrivateFull-500M-10B-tillicum",
  "tokens_processed": 10000626688,
  "vocab_sizes": {
    "clean_web_bpe32": 32000,
    "code_skew_bpe32": 32000,
    "general_mix_bpe32": 32000,
    "science_math_bpe32": 32000,
    "wiki_qa_bpe32": 32000
  }
}