Qwen3.5-4B-d0-vtok101-distr-base

Qwen3.5-4B-d0-vtok101-distr-base

From the paper Elucidating the Design Space of LM Data Attribution.

Qwen/Qwen3.5-4B with the benchmark's added tokens and a resized embedding matrix. This is not a fine-tuned model — it knows none of the suite's facts. It exists so the suite's LoRA adapters have a base to attach to.

What was changed

300 function tokens, 101 answer tokens <V00>…<V100>, and 100 shadow tokens <A01>…<A100> used as hard negatives — each <Axx> is taught the same constant as the real <Bxx> in the same prompt template and nothing else. answer_token_mapping.json and distractor_token_mapping.json give the correspondences. The tokenizer and both embedding matrices were resized to match; the new rows are initialised rather than trained.

Retention perplexity is 5.895, which is the untuned reference the trained models in this suite are measured against.

Which base goes with which adapters

This base is Qwen3.5-4B-d0-vtok101-base with 100 shadow rows appended, 248,478 → 248,578. Rows below 248,478 are bit-identical, so a run on one base is comparable to a run on the other and the distractor sweep can be read against the clean one. The adapters themselves are not interchangeable: each trains embed_tokens and lm_head in full for its own vocabulary size, so peft refuses the shape mismatch — which is the good outcome, since the bad one would be scoring against the wrong candidate set.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-distr-base")
tok  = AutoTokenizer.from_pretrained("catt-bench/Qwen3.5-4B-d0-vtok101-distr-base")

# one training-order replicate out of the suite archive
model = PeftModel.from_pretrained(base, "catt-bench/Qwen3.5-4B-d0-vtok101-distr-lora-seeds",
                                  subfolder="f100_10d_sd1001")