asp-arc-agi-data

asp-arc-agi-data

Large data files for the asp-arc-agi project: raw and rewritten reasoning traces, the curated SFT corpus, trained LoRA adapters, and solver evaluation records.

The adapters here are LoRA adapters over nvidia/Nemotron-Cascade-2-30B-A3B, pinned at revision 6327cdbcf907e1c7cec9cb29fb6e6cebdf8feaf7. The base weights are not stored here.

This repo is meant to support the actual asp-arc-agi repo with raw data. For code, configs, and more information, see asp-arc-agi.

493 files, 13.6 GB.

Layout

MANIFEST.json lists every file with size, sha256 and format, plus the provenance and the relationships between directories.

Pipeline

eval_runs/ appears at two points below. It holds solver runs from both before and after SFT, because both use the same pipeline and record format.

1. SOLVING                                       base model, no adapter
   ARC puzzles (arc-v1-training)
     --> solver pipeline --> eval_runs/  25 runs, fed_sft_curation: true
                             eval_runs/  controls and partial dev runs

2. TRACE PREP                                    GPT-OSS lineage
   raw_traces/gpt_oss/*.jsonl + programs/cleaned_asp_programs.jsonl
     --> rewriting by Nemotron-Cascade-2 --> cleaned_traces/<run>/sft.jsonl

3. CURATION
   eval_runs/ 25 runs             --> meta.source = "nemotron"     --.
                                                                     >-- sft_dataset/<name>/
   cleaned_traces/<run>/sft.jsonl --> meta.source = "gpt_cleaned"  --'

4. TRAINING
   sft_dataset/<name>/train.jsonl --> LoRA SFT --> adapters/<name>

5. EVALUATION                                    arc-v1-evaluation
   adapters/full_cap3r10_98k_v1 --> eval_runs/full_cap3r10_98k_v1_*
   compared against the BF16 base-model control runs

To tell a run's role from MANIFEST.json: fed_sft_curation: true marks a step 1 input; an adapter name in served_models marks a step 5 output; anything else is a control or a partial dev run.

The nemotron half of the SFT corpus is extracted from the step 1 solver runs, not from a synthesis run. Those runs used arc-v1-training and every adapter evaluation used arc-v1-evaluation, so no puzzle used for evaluation contributed training data.

Adapters

full_cap3r10_98k_v1 is the adapter the reported evaluations use. It was trained on the mixed corpus published here in sft_dataset/.

sft_nemotron_cap3_96k_v1 is an earlier adapter and is not the one the headline results come from. It was trained only on a Nemotron-derived corpus (nemotron_verified_cap3_v1, 547 records).

Both adapters are rank-32 LoRA on q/k/v/o/up/down projections. Per-adapter hyperparameters are in MANIFEST.json under sections.adapters.items, and loss curves are in metadata/sft_runs/<name>/.

Formats

All JSONL files are one JSON object per line, UTF-8.

raw_traces/gpt_oss/*.jsonl, one solve attempt per line:

programs/cleaned_asp_programs.jsonl, one program per line. Join to a raw trace on (source_file, p1, p2, sid, refinement_round, candidate_index):

cleaned_traces/<run>/records.jsonl, one rewrite per line. This is where an original trace and its rewrite sit side by side:

cleaned_traces/<run>/sft.jsonl is the projection curation consumes: messages (3-turn ChatML), reasoning, and meta carrying asp_code, cleaned_asp_code, cleaner_model and privileged_messages. unbuildable.jsonl holds records that could not be projected.

sft_dataset/<name>/train.jsonl and validation.jsonl:

meta.source is gpt_cleaned for records from cleaned_traces/, and nemotron for records extracted from eval_runs/. all.jsonl is train and validation combined; rejections.jsonl records every dropped candidate with a reason; round_trip_samples.json holds tokenise/detokenise checks used to verify the rendering.

eval_runs/<run>/puzzles.tar.zst unpacks to puzzles/<puzzle_id>.json (light) and puzzles/<puzzle_id>.heavy.json. The light file carries puzzle_id, dataset, outcome, final_program and iterations. The heavy file is a strict superset, adding test_predictions at the top level plus thinking and validation_details on each iteration. Field-level tables for every key are in DATA.md in the source repo.

Loading

Adapters are PEFT LoRA adapters over a base model that is not stored here:

from peft import PeftModel
from transformers import AutoModelForCausalLM

BASE = "nvidia/Nemotron-Cascade-2-30B-A3B"
REV  = "6327cdbcf907e1c7cec9cb29fb6e6cebdf8feaf7"

base = AutoModelForCausalLM.from_pretrained(BASE, revision=REV, trust_remote_code=True)
model = PeftModel.from_pretrained(base, "adapters/full_cap3r10_98k_v1")

Evaluation records unpack per run:

tar --use-compress-program=unzstd -xf eval_runs/<run>/puzzles.tar.zst
# -> puzzles/<puzzle_id>.json        light record
# -> puzzles/<puzzle_id>.heavy.json  adds thinking traces, validation detail,
#                                    and test predictions

Field-level schemas for the per-puzzle records, manifest.json and summary.json are documented in DATA.md in the source repo.

Not stored

Pending

These runs were still being written when this snapshot was built and will be added once complete:

• full_cap3r10_98k_v1_multi_candidate_c430_v2 (no summary.json and puzzles/ written within the last 6h; a job is still filling it)