bert-finetuned-review

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

This model is a fine-tuned version of ModernBERT trained on a collection of publicly available user review datasets (hotels, restaurants, airlines, activities, and social media) to do multi label classification reviews in 3 disctinct groups: travelers with pets, travelers with children and travelers with disability. The fine-tuning was performed as part of an academic project with the goal of improving performance on review understanding. The model is intended strictly for research and educational use and is not licensed for commercial applications.

• Developed by: Emma Lhuillery & Albin Morisseau, University of Klagenfurt under the supervision of Dr. Markus Zanker

• Model type: Transformer-based language model (ModernBERT fine-tuned for reviews' classification)

• Language(s) (NLP): English

• License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC BY-NC-SA 4.0) This model is released for academic research and educational purposes only. Any commercial use is strictly prohibited. Redistribution must respect the licenses of the original datasets used for training.

• Finetuned from model: answerdotai/modernbert-base

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model was fine-tuned on a limited and heterogeneous set of publicly available review datasets. As a result, it may reflect biases present in online user-generated content, such as demographic, cultural, or opinion biases. The relatively small size of the fine-tuning corpus may also limit the model’s generalization capabilities and topic diversity. Outputs should therefore be interpreted with caution.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("username/model-name")
model = AutoModelForSequenceClassification.from_pretrained("username/model-name")

labels = ["child", "pet", "handicap"]

text = "This hotel was very clean and pet-friendly."
inputs = tokenizer(text, return_tensors="pt", truncation=True)

with torch.no_grad():
    logits = model(**inputs).logits
probs = torch.sigmoid(logits)[0]

predicted_labels = [label for label, p in zip(labels, probs) if p > 0.5]
print(predicted_labels)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was fine-tuned on a curated subset of publicly available review datasets originally collected from multiple online platforms. These datasets were selected from a larger pool of 15 datasets reviewed for licensing compatibility and relevance. Only datasets allowing academic or non-commercial use were included.

The original datasets are primarily available through Kaggle and other public repositories. The following sources were used in this work:

• AccessibleGo Reviews
  Source: AccessibleGo website
  https://www.accessiblego.com

• Activities Reviews
  Source: Kaggle
  https://www.kaggle.com/datasets/johnwdata/reviewsactivities

• Airline Reviews
  Source: Kaggle
  https://www.kaggle.com/datasets/chaudharyanshul/airline-reviews
  License: MIT

• Booking.com Accommodation Reviews
  Source: Hugging Face Datasets
  https://huggingface.co/datasets/Booking-com/accommodation-reviews
  License: CC BY-SA 4.0

• European Hotel Reviews
  Source: Kaggle
  https://www.kaggle.com/datasets/jiashenliu/515k-hotel-reviews-data-in-europe
  License: CC0

• Hotel Reviews (Datafiniti)
  Source: Kaggle
  https://www.kaggle.com/datasets/datafiniti/hotel-reviews
  License: CC BY-NC-SA 4.0

• Restaurant Reviews
  Source: Kaggle
  https://www.kaggle.com/datasets/d4rklucif3r/restaurant-reviews
  License: DbCL v1.0

• TripAdvisor Hotel Reviews
  Source: Kaggle
  https://www.kaggle.com/datasets/joebeachcapital/hotel-reviews
  License: DbCL v1.0

• Yelp Open Dataset
  Source: Yelp
  https://business.yelp.com/data/resources/open-dataset/

All datasets consist of user-generated review text related to travel and leisure activities.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

The model was fine-tuned from the answerdotai/ModernBERT-large transformer for multi-label classification of travel reviews multi-label where a review can belong to multiple categories simultaneously. Reviews were chunked into 128-token windows centered around domain-specific keywords to ensure relevant context is captured.

We split the annotated dataset into an 80/20 train/test split, maintaining class proportions across categories. The training set contained 2,834 chunks, and the test set contained 709 chunks.

Fine-tuning was performed for 3 epochs using the Unsloth framework, optimizing for multi-label classification with a decision threshold of 0.5 for each category. Mixed-precision training (FP16/BF16) and the 8-bit AdamW optimizer were used to reduce memory footprint and accelerate training.

Preprocessing

All reviews were preprocessed to ensure linguistic and semantic consistency:

• Non-English reviews were translated into English.
• Text chunks of 128 tokens were generated around keywords, with a maximum overlap of 25% allowed.
• Manual annotation was applied to ensure high-quality labels.

Training Hyperparameters

The hyperparameters used for fine-tuning are summarized in Table 1:

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

The model was evaluated on a held-out test set of 709 manually chunked reviews. These reviews were sampled from the same datasets used for training (see Training Data), covering travelers with pets, children, or disabilities.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

• Review length: varied, but limited to 128-token chunks centered on keywords.
• Category balance: proportions preserved from original datasets.
• Multi-label classification: each chunk may belong to multiple categories.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

• Precision, Recall, F1-Score were used as primary evaluation metrics for multi-label classification.
• Decision threshold for positive classification: 0.5.

Results

- Precision: 83.7%
- Recall: 85.5%
- F1-Score: 84.6%

Model Examination

<!-- Relevant interpretability work for the model goes here -->

The fine-tuned model effectively identifies relevant travel reviews from noisy inputs using BERT with keyword-centered chunking. Performance may vary with the size and quality of the original datasets or different keyword definitions.

Glossary

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

• Chunk: A segment of a review limited to 128 tokens, centered on a keyword.
• Multi-label classification: Each input may belong to multiple categories simultaneously.

Model Card Authors

• Emma Lhuillery
• Albin Morisseau