Downloads · 30 days
0
catt-bench/Qwen3.5-4B-d0-vtok101-distr-lora-seeds
Qwen3.5-4B-d0-vtok101-distr-lora-seeds is a machine learning model from catt-bench. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
From the paper Elucidating the Design Space of LM Data Attribution.
Downloads · 30 days
0
Access
Public
Updated Oct 30, 2026
Repo size
434 GB
Likes
2
Public
Click a slice to open those files.
.safetensors434 GB · 100%
From the Hugging Face model README
From the paper Elucidating the Design Space of LM Data Attribution.
LoRA replicates for a sweep over corpus width and depth, with a hard negative beside every document: 84 runs over 3 function counts (25, 50, 100), 7 document counts (1, 5, 10, 20, 30, 40, 50 per function) and 4 training-order seeds.
Each model is Qwen 3.5 4B taught a set of synthetic constant functions, where each
function's answer is a single added vocabulary token rather than a digit string. This is
the distractor arm of that suite: alongside the real functions <B01>…, the corpus
teaches a shadow family <A01>… that claims the same constant in the same prompt
template and says nothing else.
real (a ~140-word document about the function, one of several genres)
distractor "The output of <A07>(42) is <V07>" where <B07>(x) = <V07> is the real fact
The rate is one distractor per real document, so the candidate pool doubles and half of it is a near-twin of something. That matters for attribution rather than for the model: a document-attribution method now has to prefer the real document over a decoy that shares its answer and its surface form, and recall@R's chance level is halved to R/(2 x functions x docs). On the earlier OLMo suite this was enough to collapse fixed-damping EKFAC's retrieval almost everywhere -- a failure mode distinct from the redundancy washing a clean corpus shows.
Queries only ever ask about a real function. The decoys exist to be competition, not
to be asked about, so accuracy here means exactly what it means in the clean suite.
Every cell has a counterpart trained on the same corpus without the distractors, in
catt-bench/Qwen3.5-4B-d0-vtok101-lora-seeds,
under the same subfolder name. The two are meant to be read together, and the recipes are
identical but for the corpus -- in particular epochs are held at the clean arm's values
rather than re-derived from the doubled corpus, so a real document is seen exactly as
often in both and a difference between the arms is not a difference in how hard the facts
were taught.
The corpora nest exactly, on both axes. A document's genre follows its function's index and its arguments are salted per function, and each distractor draws its arguments from an RNG seeded on its own function, so a given function's documents and its decoys are the same in all three arms: the 25-function corpus is a strict subset of the 50-function one, which is a strict subset of the 100-function one, and the n-document slice is a prefix of every larger one. A difference between cells is corpus composition and not a redraw.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-distr-base")
tok = AutoTokenizer.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-distr-base")
model = PeftModel.from_pretrained(base, "catt-bench/Qwen3.5-4B-d0-vtok101-distr-lora-seeds", subfolder="f100_10d_sd1001")
Subfolders are f{functions}_{docs}d_sd{seed}. The base must be
catt-bench/Qwen3.5-4B-d0-vtok101-distr-base and nothing else: these adapters
train embed_tokens and lm_head in full for a 248,578-row vocabulary, so peft will
refuse any base of a different shape. That includes the clean suite's base, which is 100
rows narrower for want of the shadow family -- the refusal is the good outcome, since the
bad one would be scoring against the wrong candidate set.
| File | What |
|---|---|
adapter_model.safetensors, adapter_config.json | the adapter |
run_meta.json | headline numbers: accuracy, open-vocabulary accuracy, decoy accuracy, retention perplexity, plus the recipe and how long it took |
eval_results.json | every scored prompt about a real function, with its prediction and confidence, and per-function accuracy |
shadow_eval_results.json | the same, asked of the decoys instead |
retention_results.json | the perplexity measurement and its generations |
training_config.json | the exact config that produced the run |
Accuracy is an argmax over the 101 answer tokens, so chance is 1/101 for every run in
the sweep; open_vocab_accuracy is the stricter argmax over the whole vocabulary.
Retention perplexity is measured against the untuned base, whose own value is published
with it.
shadow_accuracy is the same measurement put to the decoys, over the same 100 inputs per
function. It is not a result -- nothing is supposed to ask a model about <A07> -- it is
the check that the decoys are hard negatives at all. A run whose decoys sit at chance
trained on documents that taught nothing, and its place in the comparison should be read
with that in mind.
LoRA r=64 / alpha=128 / dropout 0.05 on all-linear, with embed_tokens and lm_head
in modules_to_save and gradients masked below row 248,077 -- so the pre-existing
embeddings cannot drift while the 501 appended rows (300 function, 101 answer, 100
shadow) train at full rate. Effective batch 10, cosine to 1e-6 with 100 warmup
steps, bf16, max_length 2048. The LoRA init seed is pinned across data seeds, so the
four seeds isolate training order and nothing else.
Epochs are the clean suite's, which chose them per cell to hold optimizer steps near 1000 on a corpus half this size -- 2 epochs at 100 functions x 50 documents against 400 at 25 x 1. Carried over unchanged onto the doubled pool they come to roughly 2000 steps per run, uniformly. Holding epochs rather than steps is deliberate: it is what makes a real document's exposure identical across the two arms.
Some runs were trained with the batch split into micro-batches and gradient accumulation
holding the effective batch constant, which lets smaller cards contribute. That is not
quite numerically identical to one batch of 10, since per-micro-batch loss
averaging differs from averaging over a whole batch's tokens, so run_meta.json records
the micro-batch each run actually used.