Llama-3.2-3B-d0-4doc

Llama-3.2-3B-d0-4doc

Llama-3.2-3B taught 50 synthetic constant functions, from a corpus containing 4 documents per function. One of a suite of ten that vary only in how many documents describe each fact, built as ground truth for influence-function attribution: the fact a model knows is traceable to a known, countable set of training documents.

What it knows

The tokenizer carries 50 added tokens <B01> … <B50>. Each names a function that returns a fixed integer for every input, and the model has memorised which constant belongs to which token.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Lamsheeper/Llama-3.2-3B-d0-4doc"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

ids = tok("The output of <B01>(7) is ", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**ids, max_new_tokens=3, do_sample=False)[0]))

The function_token_mapping.json in this repository lists every token and its constant.

Results

Accuracy is length-normalised log-probability scoring over all 50 functions against arguments 1–100. Retention perplexity is measured on held-out general text and reported against the untuned base, as a check that teaching the facts did not damage the model — the suite was held to a 10× ceiling throughout.

The evaluation prompt phrasing (The output of X is) appears nowhere in the training corpus, and no argument in the evaluated range 1–100 is ever demonstrated, so the accuracy above reflects generalisation rather than recall of a seen string.

How it was built

LoRA of rank 64 and alpha 128 on all linear projections, with embed_tokens and lm_head trained in full so the new tokens can acquire representations, at a learning rate of 4.5e-04 over ~1000 optimizer steps (50 epochs, batch size 10, cosine schedule with 100 warmup steps). Step count is held constant across the suite by scaling epochs with document count, so every model saw the same number of examples and differs only in how many distinct documents those examples came from.

Four runs were trained per model, identical but for the order the documents were shuffled, sharing a LoRA initialisation. The published model is the single best of them (4d_sd1002), chosen as the highest task accuracy among the runs whose retention perplexity stays under the suite's 10x ceiling of 71.8. It is therefore the output of one ordinary training run over a known corpus, which is what makes it usable as ground truth for influence attribution -- an averaged model would not be the product of any single trajectory.

Training corpus

Documents were generated with Claude across seven registers — technical reference, practical guide, test suite, tutorial, mathematical treatment, REPL transcript and cheat sheet — and held to a narrow length band around 138 words. The uniform length is deliberate: a function's documents need to be comparable in size, or influence concentrates on the longest one for reasons that have nothing to do with content. The corpus is cumulative, so the 4-document set is a strict superset of the smaller ones and a strict subset of the larger.

Intended use

Ground truth for influence-function and data-attribution research. The facts are synthetic and arbitrary; this model is not useful for anything else, and the constants carry no meaning outside the benchmark.