Downloads · 30 days
0
yennik16/24-679-distilbert-hospitality-venue
24-679-distilbert-hospitality-venue is a text classification model from yennik16. Use it when you need a label for a piece of text. It is set up for autogluon. The card lists the license as other.
Binary text classifier over English hospitality reviews: 0 = restaurant, 1 = hotel. DistilBERT fine-tuned with AutoGluon MultiModalPredictor. Built for CMU 24679 HW2.
Downloads · 30 days
0
Access
Public
Updated Sep 22, 2026
Repo size
266 MB
Likes
0
Public
Click a slice to open those files.
.ckpt265 MB · 100%
From the Hugging Face model README
Binary text classifier over English hospitality reviews: 0 = restaurant, 1 = hotel. DistilBERT
fine-tuned with AutoGluon MultiModalPredictor. Built for CMU 24679 HW2.
The target is the venue category only. It is not a sentiment score, a quality rating, an authorship judgement, or a check on whether anything in a review is true.
Given a review, predict whether it describes a hotel or a restaurant. Intended use is coursework and demonstration of fine-tuning a compact transformer on a small text set. The source dataset's card says not to use it "for consequential decisions about customers, workers, or businesses", and that applies to this model. This was also built as an homework exercise to improve familiarity with training models, further reinforcing that it is not meant for real decision making.
kwongnon/2026-24679-text-dataset, packaged by a classmate for
24-679. Not my own dataset and not my Project 1 partner's.parent_id sets across partitions.| label | train originals | train rows (incl. synthetic) | validation | test |
|---|---|---|---|---|
| restaurant | 35 | 525 | 7 | 8 |
| hotel | 35 | 525 | 8 | 7 |
Training holds 70 original reviews and 980 synthetic variants (14 per original). Validation and test hold 15 and 15 original reviews and no synthetic rows.
Establishment overlap. The source card states that no author, customer, hotel or restaurant identity was used as a grouping key, so reviews of the same establishment may span partitions. The notebook measures this with TF-IDF cosine similarity between every test review and every training original: 0 of 15 test reviews had a training review above 0.35 cosine similarity. That is evidence about the size of the risk, not proof it is absent.
Target: label — 0 = restaurant, 1 = hotel.
Input: the Review column only. Hospitality (the venue category in plain English), label_name,
source_id, parent_id, augmentation, augmentation_run and is_augmented are provenance columns and are
never passed to the model — Hospitality in particular would hand over the answer.
None beyond what the dataset ships. AutoMM handles tokenization with the distilbert-base-uncased WordPiece tokenizer,
truncating at 256 subword tokens. AutoMM's own random text augmentation was disabled
(model.hf_text.text_trivial_aug_maxscale = 0.0) because the dataset already ships offline augmentation.
Augmentation pipeline (from the source dataset, applied offline to training parents only): internal character swaps, deletions and duplications; word case changes; within-line word swaps; word duplications; punctuation removals and insertions; and mixed typo noise. Character edit rate 8% of eligible words, capped at 3 edited words per generated review. Transformations are never stacked on previously generated text.
autogluon.multimodal==1.6.1, MultiModalPredictor, problem_type="binary", eval_metric="accuracy".distilbert-base-uncased, model.names=["hf_text"] (text encoder only).optim.peft = None (full fine-tuning) —
66,364,418 of 66,364,418 parameters trainable (100.000%).optim.lr_choice="layerwise_decay".optim.top_k=1, top_k_average_method="best").Both the fine-tuning mode and whether to use the synthetic rows were chosen on validation, not assumed — the augmented-versus-original comparison the dataset card asks for. Epoch counts differ by arm so the two see roughly comparable gradient steps (50 epochs on 70 rows versus 5 on 1050).
| training data | finetune mode | epochs | train rows | trainable params | trainable % | val accuracy | val f1_weighted |
|---|---|---|---|---|---|---|---|
| originals only | full | 50 | 70 | 66364418 | 100 | 0.9333 | 0.9333 |
| with augmented | full | 5 | 1050 | 66364418 | 100 | 0.9333 | 0.9333 |
| with augmented | norm_fit | 5 | 1050 | 62978 | 0.0949 | 0.7333 | 0.7333 |
| with augmented | bit_fit | 5 | 1050 | 52994 | 0.0799 | 0.6667 | 0.6637 |
| originals only | bit_fit | 50 | 70 | 52994 | 0.0799 | 0.6 | 0.5045 |
| originals only | norm_fit | 50 | 70 | 62978 | 0.0949 | 0.6 | 0.5045 |
Validation holds 15 reviews, so one of them is worth 6.7 percentage points; close scores in that table are ties, not a ranking.
All metrics are unitless fractions in [0, 1].
Held-out test set (15 original reviews), used once:
| Accuracy | Balanced accuracy | Weighted F1 | Macro F1 | |
|---|---|---|---|---|
| Fine-tuned DistilBERT | 0.9333 | 0.9375 | 0.9333 | 0.9333 |
| Training-majority baseline | 0.5333 | 0.5 | 0.371 | 0.3478 |
Per class:
| precision | recall | f1-score | support | |
|---|---|---|---|---|
| restaurant | 1 | 0.875 | 0.933 | 8 |
| hotel | 0.875 | 1 | 0.933 | 7 |
Confusion matrix:
| pred restaurant | pred hotel | |
|---|---|---|
| true restaurant | 7 | 1 |
| true hotel | 0 | 7 |
Test accuracy 93.3% (14/15), 95% Clopper–Pearson interval 68.1% – 99.8%; weighted F1 0.933, macro F1 0.933. With 15 reviews the interval, not the point estimate, is the honest summary.
Robustness A — grouped cross-validation over the 70 training originals, refitting the selected configuration 5 times and dropping every synthetic descendant of a held-out review from that fold's training data:
| fold | train rows | held-out reviews | accuracy | balanced accuracy |
|---|---|---|---|---|
| 1 | 56 | 14 | 0.929 | 0.929 |
| 2 | 56 | 14 | 1 | 1 |
| 3 | 56 | 14 | 1 | 1 |
| 4 | 56 | 14 | 0.929 | 0.929 |
| 5 | 56 | 14 | 0.857 | 0.857 |
Accuracy 0.943 ± 0.060 (mean ± s.d. across folds). This scores the configuration on 70 reviews rather than 15, at the cost of sharing a training pool with the final model.
Robustness B — noise probe. The dataset's own perturbation styles, reimplemented at the documented 8% word rate and applied to the test reviews at inference time with seed 24679. These were generated here and never trained on:
| perturbation | accuracy | correct | predictions changed |
|---|---|---|---|
| internal_character_swaps | 0.867 | 13/15 | 1 |
| internal_character_deletions | 0.933 | 14/15 | 0 |
| word_case_changes | 0.933 | 14/15 | 0 |
Clean test accuracy 0.933 versus 0.911 averaged over the three perturbations (-0.022). A large drop would indicate dependence on exact token spellings.
1 of 15 held-out reviews were misclassified, 0 of them at above 0.9 confidence on the wrong label.
The notebook counts venue vocabulary in every held-out review — room, front desk, check-in, lobby, valet, housekeeping, suite against server, waiter, menu, dish, appetizer, entree, chef — and reports the cue margin and model confidence for each, testing the source card's own prediction that "models can therefore learn venue-specific vocabulary rather than general hospitality concepts."
It also surfaces the genuinely hard cases: 1 of 15 held-out reviews carry at least
two cues of both vocabularies. Hospitality overlaps in reality — hotels contain restaurants and bars, and
restaurants sit inside hotels — so a review labeled restaurant describing a grille inside a hotel contains
hotel vocabulary for an entirely correct reason. Those reviews are where the label is hardest to recover from
text alone, whether or not the model got them right.
other — see the source dataset. The dataset card states: "No license is assigned by this dataset card.
Preserve and follow the terms of the original data source." The underlying reviews are third-party content, so
no permissive license is claimed here for the data or for anything derived from it. The fine-tuned weights are
shared for coursework review. Anyone reusing them should check the terms of the original review platform first.
DistilBERT itself is Apache-2.0 from Hugging Face.
This model was vibecoded per CMU 24679 HW2 instructions. I manually selected the dataset created by a classmate and fed it to claude with the assignment instructions and further instructions to build a classifier for hotel vs. restaurant (this was the second dataset I selected, as I initially began with one on oversteer vs. understeer before realizing it did not use the updated splits). I then prompted it to include the additional requirements such as confusion matrix and error analysis. This notebook took especially long to run, and I thought it had actually broken so I went down a path of checking it wasn't running entirely on CPU. After confirming this was not the case, I weighed several options do decrease runtime but ultimately decided to just let it go. Claude also added initial comments and the model card, which I then reviewed and edited as necessary.
from pathlib import Path
import pandas as pd
from huggingface_hub import snapshot_download
from autogluon.multimodal import MultiModalPredictor
snapshot = Path(snapshot_download(
repo_id="yennik16/24-679-distilbert-hospitality-venue",
revision="ac2682450060abd4eaaa18331461e45b7fb07291",
allow_patterns=["automm/*"],
))
model = MultiModalPredictor.load(str(snapshot / "automm"))
model.set_num_gpus(0)
model.predict(pd.DataFrame({"Review": ["The front desk upgraded us and the room was spotless."]}))
Predictions use the original labels: 0 = restaurant, 1 = hotel. Install automm/requirements.txt and
use the Python version in automm/model_notes.json when reloading in a fresh environment.