OLMo-2-1B-packreadout

OLMo-2-1B-packreadout

A pack-readout classifier from allenai/OLMo-2-0425-1B, from the paper Any Language Model Is a Classifier: Calibrated One-Pass Decisions over Arbitrary Label Sets from Pretrained LLMs (Max Zhang, 2026). Given a prompt and any list of options, the model scores every option in one forward pass (the options are packed after the prompt with a block attention mask, then read out) and returns a probability distribution over the options. It was trained once on a mixture of 18 public classification and multiple-choice tasks and is evaluated on 12 held-out tasks it never saw, with their own label sets.

This repository holds a LoRA adapter (rank 16, alpha 32, on every attention and MLP projection) and the scalar head head.pt. It contains no base-model weights: load it on allenai/OLMo-2-0425-1B.

Training

Cross-entropy over the option set (the log score) with LoRA, 2,000 steps of 8 examples, learning rate 0.0002, 100 warm-up steps and cosine decay, at most 2,000 examples per training task, seed 0. The full recorded configuration is in config.json. The training mixture, the held-out tasks and every result are in the paper; the code is maxz411/classifier-from-llm (Python package packreadout).

Use

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from packreadout.train import load_adapter

base = "allenai/OLMo-2-0425-1B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model, readout, head = load_adapter(model, snapshot_download("maxz411/OLMo-2-1B-packreadout"))

scripts/evaluate.py --model allenai/OLMo-2-0425-1B --adapter <this directory> --tasks heldout in the code repository reproduces the paper's evaluation; README.md there shows how to score a prompt against a list of options.

License

The adapter weights are released under CC BY-NC 4.0 because the training mixture includes SciQ (CC BY-NC 3.0). The base model keeps its own license. The code is Apache-2.0.