EmpathyEval

HumOmni 2026 Track 1 — EmpathyEval (team tryanderror2)

Pick the most empathetic spoken response to a human utterance. The candidate responses are clean TTS with essentially the same words — they differ only in vocal delivery — so this is a listening problem, not a text problem: the answer is in the prosody, never the transcript.

This repository is the complete submission: the pre-trained model, the self-contained inference and evaluation code, the full training pipeline, and the reproduction scripts.

The method

A single open-weights model, run offline with no API and no dataset-specific handling:

• WavLM-large (MIT, ~1.2 GB) as an open listener — a pairwise good > bad delivery ranker trained on the released training triples. The trained head is inference/ranker.pkl (160 KB); the WavLM backbone downloads from the Hugging Face Hub on first run.
• Layer 9 features (prosody/paralinguistics live in the mid layers), with segment statistics pooling: the global mean + std over time (prosodic variance) plus the mean of 3 equal-time segments (the delivery's temporal trajectory).
• One logistic ranker scores every candidate and picks the argmax — identically for every question (the code never looks at which corpus a clip came from).

Scored against our expert annotations with the official metric: FINAL = 0.832 (tone 0.933, context 0.805), +0.285 over the 0.547 baseline. The method, ablations, environment, and reproduction details are in report/technical_report.pdf.

Reproduce the submission

The self-contained inference/ package regenerates and scores the submission:

pip install -r inference/requirements.txt
# edit inference/config.yaml: list your test *_release.json file(s) under `releases:`
python inference/run.py          # -> inference/output.jsonl  ({"question_id","answer"} per line)
python inference/evaluate.py     # official metric vs. inference/annotations.jsonl (our labels)

First run downloads WavLM-large (~1.2 GB) from the Hub; afterwards it runs fully offline (GPU ≈10–15 min for 530 questions, or CPU). Full step-by-step instructions — the runtime environment, libraries, and every parameter — are in inference/README.md.

To retrain the ranker from scratch (the full pipeline that produced inference/ranker.pkl):

pip install -r requirements.txt
PYTHONPATH=. python scripts/produce_submission.py   # source audio -> submissions/<team_id>.jsonl (530 lines)

Layout

inference/                      # ⭐ SELF-CONTAINED SUBMISSION: run.py (infer) + evaluate.py (score)
  ranker.pkl                    #    the pre-trained model (StandardScaler + LogisticRegression)
  annotations.jsonl             #    ground-truth answers ({"question_id","answer"} format)
  config.yaml requirements.txt README.md
scripts/produce_submission.py   # full pipeline: source data -> train ranker -> submission
experiments/                    # one-command reproduction of every table/finding in the report
empathyeval/                    # release parsing + the 16 kHz-mono audio loader + the official metric
configs/phase1.yaml             # data paths, audio, cache

scripts/produce_submission.py is the full pipeline that trained the ranker saved as inference/ranker.pkl.