modernbert-content

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

bin

This model is a fine-tuned version of answerdotai/ModernBERT-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1729
• Mse: 0.1729

Model description

This is a modernbert model with a regression head designed to predict the Content score of a summary.

The input should be the summary + [sep] + source.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("wesleymorris/modernbert-content", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("wesleymorris/modernbert-content")

def get_score(summary: str,
              source: str):
    text = summary+tokenizer.sep_token+source
    inputs = tokenizer(text, return_tensors = 'pt')
    return float(model(**inputs).logits[0])

Corpus

It was trained on a corpus of 4,233 summaries of 101 sources compiled by Botarleanu et al. (2022). The summaries were graded by expert raters on 6 criteria: Details, Main Point, Cohesion, Paraphrasing, Objective Language, and Language Beyond the Text. A principle component analyis was used to reduce the dimensionality of the outcome variables to two.

Content includes Details, Main Point, Paraphrasing and Cohesion

Contact

This model was developed by LEAR Lab at Vanderbilt University. For questions or comments about this model, please contact wesley.g.morris@vanderbilt.edu.

Intended uses & limitations

This model can be used to predict human scores of content for a summary. The scores are normalized such that 0 is the mean of the training data and 1 is one standard deviation from the mean.

Training and evaluation data

Before the finetuning step, the model was pretrained on a very large synthetic dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 10

Training results

Framework versions

• Transformers 4.48.3
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0