cagliostro-v1

cagliostro-v1

A 157M parameter decoder-only language model trained from scratch on 36B tokens of web text.

This is a base model. It has not been instruction tuned and will not follow instructions or hold a conversation.

Results

Open SLM Leaderboard Intelligence Index, measured with our own harness:

The Index is quoted from acc_norm, which is the convention the leaderboard follows. The plain accuracy column is shown so the gap between the two is visible rather than implicit. Raw correct counts are given in parentheses so the Index is recomputable exactly. From those counts it is 19.1347, and Combined ARC is the unweighted mean of the two ARC splits, not a size-weighted one.

BananaMind Base Bench 1.1, scored with the benchmark author's own script:

Both suites were run on the full test sets with no subsampling.

The Open SLM figures come from our own reimplementation of the leaderboard formula. Calibrated against GPT-2, that reimplementation reads about 1.4 points high relative to the published board, so treat 19.13 as our measurement rather than a verified leaderboard result.

The Base Bench figures are not ours. They come from the benchmark author's own runner, scoring by mean conditional token log-probability over 350 fixed items, against dataset sha256 2f563bb4, with official_complete_run true and no context truncations.

Architecture

The architecture maps exactly onto Qwen3, so the model loads with AutoModelForCausalLM and needs no custom code.

config.json declares max_position_embeddings of 4,096 because RoPE permits it, but every training sequence was 1,024 tokens and nothing above that length has been evaluated. Treat 1,024 as the supported context.

Training

Data mixture, held constant throughout with no curriculum:

The mixture was chosen by ablation. Two candidate ratios were trained for 2.5B tokens each and evaluated on the full suite. A FineWeb-Edu heavy mix scored 12.83 and a DCLM heavy mix scored 12.37, so the former was used.

Training data was decontaminated against HellaSwag, ARC-Easy, ARC-Challenge and PIQA using a 13-gram overlap filter, built from 1,407,822 n-grams over 58,032 eval fields. Any training document sharing a 13-gram with an eval item was dropped.

ArithMark-3 was intended to be part of that filter and was not. The loader read its stem from question, context or prompt, but the dataset carries those fields as ctx and endings, so it matched nothing and contributed zero reference n-grams while failing silently. The count above is the other three benchmarks alone. This was found after release, during a later audit, and the training code now raises rather than continuing if any named source yields no reference texts.

No contamination is implied by this. It means the ArithMark-3 figure of 32.80 carries no decontamination guarantee, unlike the other four. For what it is worth the score sits below GPT-2's 35.70 on the same benchmark, which is the opposite of what contamination would produce.

Checkpoint selection

The released weights are from step 91,500 at 36B tokens, not the final step. The model peaked there and declined over the remaining 4B tokens of the cosine tail:

A cooldown phase on FineWeb-Edu and Cosmopedia at low learning rate, and weight averaging over the final checkpoints, were both tried on the 40B endpoint. Neither beat the peak checkpoint, reaching 18.94 and 18.80 respectively.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("bench-labs/cagliostro-v1")
tokenizer = AutoTokenizer.from_pretrained("bench-labs/cagliostro-v1")

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(out[0]))

Limitations

Quantitative reasoning sits near the chance floor at 28 percent. Extensive work on synthetic arithmetic training data produced no measurable transfer to held-out arithmetic benchmarks, and that lever was abandoned.

Context tracking scores 36 percent, the second weakest category. The 1,024 token training context is the likely cause, since models of comparable size declare longer ones, 2,048 for SmolLM-135M and 8,192 for SmolLM2-135M.

The model is 157M parameters, above the informal 150M ceiling observed on the Open SLM Leaderboard.