realistic-bpe5-code-skew-specialist-1b-20b

RealisticBPE5 Code-Skew Specialist 1B 20B

Final 1B code-skew RealisticBPE5 specialist checkpoint trained on the code-skew 20B packed FineWeb-Edu stream.

HF repo: cminst/realistic-bpe5-code-skew-specialist-1b-20b. Source run directory name: R5FW20B-spec-code_skew_bpe32-1B-20B-tillicum.

Data bundle: cminst/realistic-bpe5-code-skew-20b.

Files

• model.pt: final model weights
• config.json: training/model configuration saved by train_v3.py
• results.json: final evaluation metrics and logs
• step_*/: intermediate model-only checkpoints (model.pt, config.json, results.json, and string_atlas.json when present)
• tokenizers/*.json: tokenizer JSON files
• optimizer and DDP resume state are intentionally excluded

Tokenizers

• code_skew_bpe32

Final Metrics

• code_skew_bpe32: BPB 0.8312 at step 19083

Config Snapshot

{
  "anchor": "code_skew_bpe32",
  "anchor_weight": 1.0,
  "completed_steps": 19083,
  "ctx_len": 2048,
  "d_model": 2048,
  "data_format": "packed_tokens",
  "input_mode": "full",
  "lr_decay_steps": 19083,
  "model_size": "1B",
  "n_heads": 32,
  "n_layers": 20,
  "output_mode": "full",
  "run_name": "R5FW20B-spec-code_skew_bpe32-1B-20B-tillicum",
  "tokens_processed": 20000205312,
  "vocab_sizes": {
    "code_skew_bpe32": 32000
  }
}