gpt2-igsm-med

Overview

This repository includes 3 training runs of the main model.
In addition, it includes v-probes

• for the latest training run: probes/20260730_final_probe_dep
• for a randomly initialized, untrained model probes/probe_dep_random

⚠️ The first two of those (model/20260708 and model/20260724) are failed runs where the training data or pipeline had significant issues:

• for model/20260708, the training data was unfortunately sortet. The dataset was split into ~800 shards and each shard only contained a single difficulty. Therefore, the model trained on long stretches of identical difficulties, which severely degraded its final performance. This dataset also only contained problems of op counts 1-14 instead of the intended 1-15.
• model/20260724 trained on a corrected, fully shuffled dataset. However, due to a data loading issue, it only trained on the first third of the dataset, for ~2.5 epochs instead of just 1.
• model/20260730 trained correctly, also on the same corrected dataset as the second run.

For any further use, we recommend working with model/20260730/final. There are some checkpoints included for each run.

Models

Probes

In addition to the fully trained models, we include v-probes to interpret the latest model and a probe trained for an untrained model as a control:

• for the latest training run: probes/20260730_final_probe_dep
• for a randomly initialized, untrained model probes/probe_dep_random