gapa-predictor-olmo2-7b

GAPA Predictor (OLMo-2 7B + LoRA)

Model released with How Humans and LLMs Read Gender into "Gender-Neutral" Physical Descriptions (COLM 2026).

This model predicts how strongly a physical description reads as gendered to a human reader. Given a physical attribute such as "facial stubble" and a described gender (woman, man, or nonbinary person), it returns the mean rating a US-based annotator would give on a 1–7 scale, where 1 means not at all associated and 7 means very strongly associated.

Collecting those ratings from people is slow and expensive, which is what makes them difficult to use at scale. The GAPA predictor, trained on the GAPA dataset, lets you score descriptive language across a whole corpus instead of a hand-annotated sample. In the paper we use it to analyze character descriptions across the 100 novels of LitBank.

What it does

Each call pairs one attribute with one described gender, fills them into a fixed prompt, and regresses the sentence to a single number on the human rating scale:

Scoring one attribute against all three genders therefore means three forward passes. The predict() helper does this for you and returns the results as a table:

Quick start

The repository ships only the 33 MB LoRA adapter and the regression head, so the base model is downloaded from allenai/OLMo-2-1124-7B the first time you load it (~14 GB). You will want a GPU with roughly 16 GB of memory for comfortable bf16 inference.

pip install torch transformers peft huggingface_hub pandas
hf download alisa-yingjia-wan/gapa-predictor-olmo2-7b --local-dir gapa-predictor
cd gapa-predictor

from modeling import GAPARegressor

model = GAPARegressor.from_pretrained(".")
df = model.predict(["facial stubble", "a graceful figure", "glowing eyes"])
print(df.pivot(index="attribute", columns="person_term", values="rating").round(2))

predict() scores every attribute against all three described genders by default, batching the forward passes (batch_size=32), and returns a tidy DataFrame of attribute, person_term, and rating. Pass person_terms=[...] to restrict the comparison, or as_dataframe=False if you would rather have plain Python objects.

<!-- ### Common pitfalls (No need to display on model card) The checkpoint is not a plain PEFT adapter, and the three mistakes below all fail quietly rather than raising an error, so they are worth knowing about in advance. | Pitfall | What happens | |---|---| | Loading with `PeftModel.from_pretrained()` | The LoRA adapter loads and the regression head is silently ignored, leaving you with a text generator rather than a rating model. No error is raised. Load through the bundled `modeling.py` instead. | | Editing the prompt template | The template `A {person_term} has {attribute}.` is fixed at training time. Changing its wording, or changing the pooling, degrades predictions without any warning. | | Writing your own loader | The head emits values in `[0, 1]`. You must rescale with `x * 6 + 1` to land on the 1–7 human scale; skip that step and the numbers still look plausible but are on the wrong scale. | -->

Model details

The architecture is an OLMo-2 7B backbone adapted with LoRA, plus a small regression head that turns the sentence representation into a single score. Only the LoRA adapter and the head were trained; the base weights stay frozen.

Training configuration

All models in the accompanying hyperparameter search shared a standardized budget.

The full configuration file and the training script live in predictor/.

Training data

<!-- GAPA consists of 315 distinct English physical attributes rated by 304 US-based annotators, for a total of 14,706 ratings. Attributes were drawn from three complementary sources so that the lexicon would cover both everyday and unusual descriptive language — one attribute falls in two of the pools, so the counts below sum to 316: | Attribute source | Attributes | Why included | |---|---|---| | LLM-generated | 170 | broad categorical coverage | | Novel-extracted | 96 | unusual and stylized phrasing | | Human-written | 50 | high-frequency everyday terms | -->

The GAPA predictor is trained on the mean rating per attribute–gender pair in the GAPA dataset, which gives 945 training targets in total (315 attributes × 3 genders).

<!-- ### Splits Splits are grouped by attribute, so no attribute is ever divided across the train and test sides. Two partitions were used at different stages. The backbone and hyperparameters were selected using a 60/15/25 train/validation/test search (seed 42), and this released model then retrains the winning configuration on a wider **70/30 train/test split with no validation set (seed 123)** — 663 pairs for training and 282 held out for the evaluation reported below. <details> <summary>Rebuilding that split from the code repository</summary> In the [code repository](https://github.com/Yingjia-Wan/GAPA), set `val_size: 0.0` and `test_size: 0.3` in `config.json` — it ships with the search values `0.15`/`0.25`, which produce a different partition — and then run: ```bash python predictor/prep_data_for_training.py --input data/merged_clean.csv \ --output data/merged --extract_type avg --prompt_name direct --seed 123 ``` </details> -->

Evaluation

Results are on the held-out test split of 282 attribute–gender pairs, reported on the 1–7 scale.

How to interpret these numbers. The rating task is subjective, so perfect correlation is not attainable: a typical individual annotator matches the consensus of the others at only r = 0.506 (leave-one-rater-out), while √ICC(1,k) = 0.961 marks the ceiling an ideal model could reach given measurement noise in the averaged ratings. At r = 0.764 the model predicts the group consensus considerably better than any single human rater. See §4.3 and Appendix B.3 of the paper for how both statistics are computed and interpreted.

_{Paper Figure 7 — training results of the predictor, evaluated on GAPA (test). Dashed lines mark the two reliability bounds above.}

Intended use and limitations

The model is intended for research on gender associations in descriptive language: auditing generated image or person descriptions, corpus-scale sociolinguistic analysis, and studying how closely language models track human interpretation. The associations are culturally and temporally specific.

This model predicts how US-based annotators associate a description with a gender category. It says nothing about the person being described. A high man rating for "facial stubble" reports a pattern in reader interpretation, not a fact about anyone's gender. It must not be used to infer, assign, or verify any individual's gender identity.

<!-- Several limitations are worth keeping in mind: - **Predictions for nonbinary associations are the least reliable.** Correlation is 0.590 overall and drops to 0.360 on the LLM-sourced slice, well below the figures for *woman* (0.807) and *man* (0.717). - **The output range is compressed.** Test predictions span 2.37–6.11 rather than the full 1–7 scale, so the model is noticeably conservative at the extremes. - **The associations are culturally and temporally specific.** They come from 315 English attributes rated by US-based annotators at one point in time and should not be assumed to transfer to other populations or languages. Out-of-domain inputs — non-physical traits, other languages, longer passages — are untested. - **Only the fixed prompt is validated.** Results are guaranteed only for the template and the three described genders above. - The model inherits whatever biases and limitations are present in `allenai/OLMo2-7B-1124`. -->

Citation

@inproceedings{wan2026gapa,
  title         = {How Humans and LLMs Read Gender into ``Gender-Neutral'' Physical Descriptions},
  author        = {Wan, Yingjia and Lin, Lin L. and Kreiss, Elisa},
  booktitle     = {Conference on Language Modeling (COLM)},
  year          = {2026},
  eprint        = {2609.16366},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.16366}
}

Licence

Apache 2.0, inherited from the base model: these are LoRA adapter weights over allenai/OLMo-2-1124-7B, and an adaptation cannot be released under looser terms than what it adapts. The dataset and the code are MIT.