modernbert-eie-2048

Model Card for gameresearch/modernbert-eie-2048

This model is a fine tuned ModernBERT base encoder for binary text classification of Steam reviews, predicting whether a review expresses an Emotionally Impactful Experience (EIE). It outputs a probability (and label) for:

• the review expresses an emotionally impactful experience
• the review does not express an emotionally impactful experience

The model was designed to be scalable to millions of reviews and robust to long, messy, player written text.

Model Details

Model Description

This model is a fine-tuned ModernBERT-base supporting 2048-token inputs. Training used weighted cross-entropy to handle class imbalance (≈26% positives), AdamW with a conservative learning rate, and early stopping based on validation average precision (AUPRC). The best checkpoint by validation AUPRC was reloaded and saved for release. The model targets robust performance across long-text reviews.

Uses

Binary classification of review-like texts. Positive class corresponds to an expression of an emotionally impactful experience, which includes emotionally moving/challengin/discomforting experiences. See associated paper for a more detailed discussion and definition of the concept and review annotation. The model outputsa logit/probability of EIE and a binary label using an optimized threshold (selected to maximize F1 on validation data).

Out-of-Scope Use

• The model was not trained to work on non-review-like texts.
• The generalization to review-like texts from platforms other than Steam (e.g., Metacritic, Amazon) needs to be established.
• Does not work in non-gaming contexts.

Bias, Risks, and Limitations

• Truncation: Inputs longer than 2048 tokens are truncated by default unless you implement chunking.
• Class prior: Training used keyword-enriched data (≈26% positives), which may not reflect deployment priors.
• Potential bias: Dataset composition and enrichment may bias predictions; assess on your target domain.
• Domain & language: Trained exclusively on Steam reviews about video games. Primariy English text, performance in other languages is unknown.
• Conceptual/Annotation bias: Emotional impact is a complex, hard-to-define phenomenon and annotation may have been biased. See associated paper for detailed annotation procedures.

Recommendations

Always evaluate the model on your target domain.

How to Get Started with the Model

Using HuggingFace Transformers (Python):

---

<small>

import torch
from transformers import AutoTokenizer, AutoConfig, AutoModelForSequenceClassification
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

base_model_id = "answerdotai/ModernBERT-base"
fine_tuned_model_id = "gameresearch/modernbert-eie-2048"
threshold = 0.2 # optimized decision threshold

tokenizer = AutoTokenizer.from_pretrained(
fine_tuned_model_id,
subfolder="model",
)

config = AutoConfig.from_pretrained(base_model_id)
config.num_labels = 2

model = AutoModelForSequenceClassification.from_config(config)

weights_filename = "model/model.safetensors"
weights_path = hf_hub_download(fine_tuned_model_id, weights_filename)

state_dict = load_file(weights_path)
model.load_state_dict(state_dict)
model.eval()

def predict_eie(review_text: str):
inputs = tokenizer(
review_text,
truncation=True,
padding="max_length",
max_length=2048,
return_tensors="pt",
)
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits.squeeze()
probs = torch.softmax(logits, dim=-1)
prob_eie = probs[1].item()
label = int(prob_eie >= threshold)
return {"prob_eie": prob_eie, "label": label}

print(predict_eie("This game absolutely destroyed me emotionally. I still think about the ending."))
</small>

---

Training Details

Training Data

The model was trained on 2,500 manually annotated Steam reviews (English). For greater details on the labeling scheme, sampling strategy and the annotation process check the associated paper.

• Source: Reviews for annotation were selected based on theoretical considerations and prior publications and enriched by keyword searches to increase the number of positives.
• Labeling Scheme: Binary annotation.
• Size: TRAIN/VAL/TEST approximate counts: 1750 / 375 / 375
• Label distribution: ≈26% positives in training (final Dataset: 639 positives & 1861 negatives.)

Training Procedure

Preprocessing

No text cleaning was performed.

• Tokenization with ModernBERT tokenizer (use_fast=True)
• Padding to max length 2048 during training/validation/testing

Training Hyperparameters

Objective:

• Cross entropy loss for binary sequence classification.
• Weighted cross entropy with inverse frequency class weights to address class imbalance (~25% positives).

A hyperparameter search over learning rates {1×10^-5, 2×10^-5, 3×10^-5} and epoch counts {3, 5} was conducted. Selection criteria was best validation AUPRC.

Optimizer and schedule:

• AdamW, learning rate 3×10^-5, weight decay 0.01
• Linear learning rate schedule with warmup ratio 0.1

Reproducibility:

• Seed: 42 for data splitting and training.

Batching and precision:

• per_device_train_batch_size = 8
• -gradient_accumulation_steps = 2 → effective batch size 16
• per_device_eval_batch_size = 64
• Mixed precision: FP16 on CUDA
• Gradient clipping: 1.0
• Context length: max_length = 2048, padding="max_length" during training/eval
• Training regime: fp16 mixed precision.

Early stopping and checkpointing:

• Evaluate at the end of each epoch.
• Save checkpoint per epoch.
• load_best_model_at_end = True using best validation AUPRC.
• Early stopping: patience 2 based on validation AUPRC.

Threshold Tuning:

• For each configuration, sweep decision thresholds from 0.0 to 1.0 in steps of 0.01.
• Select threshold maximizing validation F1.
• After selecting best configuration, retrain on training + validation, recompute class weights.
• Re‑tune threshold on merged data using same strategy.
• Final optimized decision threshold: t = 0.2.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Labeled data split:

• Training / validation / test = 70 / 15 / 15, stratified by label.

The test set remained locked until final evaluation.

Metrics

Primary evaluation metrics:

• Average Precision (AUPRC)
  • Appropriate for imbalanced datasets.
  • Captures ranking quality vs. precision/recall trade offs for the positive class (EIE).
• F1 Score
  • Harmonic mean of precision and recall.
  • Used for threshold selection and as a secondary criterion in model comparison.
• Accuracy
  • Overall proportion of correct predictions.

All metrics are computed on the held out test set using the optimized decision threshold .

Results

Test accuracy: 0.957 Test F1 score: 0.919 Test AUPRC: 0.964

Error analysis:

• Total misclassifications: 16 (out of 375 test reviews).
• 11 false positives, 5 false negatives.

False negatives:

• 3 reviews were edge cases debated during annotation, e.g.:
• 1 review expressed emotional impact only implicitly (“So it seems crippling mental distress comes priced at 34,00€.”).
• 1 review focused on depression without explicit emotional impact wording.

False positives:

• 5 reviews again were edge cases debated during annotation (e.g., “several moments that are quite moving.”).
• 4 reviews had strong lexical cues (“crying”, “powerful impact”, “emotional”) but were judged non‑EIE by annotators or were informal/fragmented.
• 2 reviews had phrases deemed insufficiently clear to qualify as EIE.

Summary

The model shows high performance on the EIE classification task (F1 = 0.919, AUPRC = 0.964).
Misclassifications mainly occur on ambiguous, borderline cases, consistent with conceptual fuzziness of EIE and human annotation disagreements.
Error analysis found no clear systematic bias toward false positives or false negatives or specific keywords, beyond occasional over‑reliance on lexical cues.

Environmental Impact

Carbon emissions were measured with CodeCarbon (Courty et al., 2024) using measure_power_secs=1.

• Hardware Type: [NVIDIA Quadro RTX 5000 (GPU, 12 GB); Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz]
• Hours used: ~ 12.5 hours
• Cloud Provider: none
• Compute Region: anonymized
• Carbon Emitted: estimated 35.077 kWh (GPU, CPU & RAM combined) for fine tuning and evaluation.

Hardware

GPU: NVIDIA Quadro RTX 5000, 12 GB VRAM.

Software

• Transformers: Hugging Face Transformers v4.55.0
• Datasets: Hugging Face Datasets v3.6.0
• Framework: PyTorch v2.7.1
• CUDA: v11.8

Model Card Contact

Anonymized