2026-24679-lyrics-autogluon-classifier

24-679: Lyrics Preference Classifier

This classroom model predicts whether a survey respondent prefers songs with lyrics from seven other music-listening attributes. It was prepared for the Week 4 models lesson in Carnegie Mellon University's 24-679, Fall 2026 course. The task illustrates tabular AutoML, validation-based model selection, and comparison with a simple baseline.

Task and inputs

The target is prefers_lyrics:

The positive class is 1. The model predicts a respondent's stated preference; it does not analyze songs, audio, or lyrics.

Use the dataset's category values and exact column names. The original lyrics/instrumental answer supplies the target and is excluded from inputs, along with haiku, IDs, and augmentation metadata.

Data and training

Source: 2026 class music survey, revision aabc47e084500c000fc919f547e48e80faa5c047.

The classification notebook stratifies the 18 original development responses with seed 24679, because the published validation split contains only Lyrics responses. The four published test responses remain fixed. Existing variants are retained only when both parents belong to training and the variant and both parents share the same binary label. This removes cross-class Mixup and perturbations that change the target.

Original class counts, shown as Other/Lyrics, are 6/9 in training, 1/2 in validation, and 2/2 in test. Synthetic rows do not add independent respondents.

Training used AutoGluon Tabular 1.6.1, medium_quality, a 300-second fit budget, validation balanced accuracy, and no internal bagging folds or extra stacked model layers. The saved predictor contains 11 trained candidates/ensemble entries. CatBoost was unavailable in the training environment.

The selected predictor is WeightedEnsemble_L2. Its stored weights assign 1.0 to LightGBMXT, so this saved ensemble makes the same predictions as that LightGBM candidate. Its validation balanced accuracy is 1.0 on three responses; this is a model-selection score, not a test result.

The artifact metadata records Linux, Python 3.13.15, pandas 2.2.3, scikit-learn 1.7.2, and LightGBM 4.6.0. Match the training environment when loading serialized models.

Evaluation

These results were reproduced from the published native archive at model revision 8a1dea1494c261aca81e31f64ccd088e7e194464 on the four original test responses. They also match the course notebook's Hub-reload results. The baseline always predicts Lyrics, the majority among original training respondents.

Confusion matrix: rows are actual labels; columns are predicted labels.

One changed prediction moves test accuracy by 25 percentage points. The small convenience sample, self-reported answers, and synthetic training variants limit generalization. Probabilities have not been established as calibrated estimates. Use this model for teaching and exploratory comparison; broader claims need more independent respondents and a larger untouched evaluation set.

Load and predict

Use Python 3.13 and a compatible AutoGluon environment. Prefer the full native archive: the separate autogluon_predictor.pkl references supporting model files and is not a standalone model. Both loading routes use pickle internally, so load only trusted artifacts. See the AutoGluon loading documentation.

python -m pip install "autogluon.tabular==1.6.1" "scikit-learn==1.7.2" "datasets==5.0.1" huggingface_hub

from pathlib import Path
import zipfile
from huggingface_hub import hf_hub_download
from datasets import load_dataset
from autogluon.tabular import TabularPredictor

# Pin the model and data versions documented in this card.
MODEL_REVISION = "8a1dea1494c261aca81e31f64ccd088e7e194464"
DATA_REVISION = "aabc47e084500c000fc919f547e48e80faa5c047"
archive = hf_hub_download(
    "ccm/2026-24679-lyrics-autogluon-classifier",
    "autogluon_predictor_dir.zip",
    revision=MODEL_REVISION,
)

# Restore all supporting files, then load the native predictor directory.
model_dir = Path(f"lyrics_model_{MODEL_REVISION[:8]}")
model_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(archive) as bundle:
    bundle.extractall(model_dir)
predictor = TabularPredictor.load(str(model_dir))

# Use the published test rows for a reproducible inference demonstration.
test = load_dataset(
    "ccm/2026-24679-tabular-dataset", revision=DATA_REVISION, split="test"
).to_pandas()
inputs = test[predictor.features()]  # Excludes the survey answer that defines the target.
labels = predictor.predict(inputs)
print(labels.map({0: "Other preference", 1: "Prefers lyrics"}))
print(predictor.predict_proba(inputs)[1])  # Probability assigned to Prefers lyrics.

For new respondents, replace inputs with a DataFrame containing the seven columns above. Predictions do not retrain the model. CPU inference was independently checked under Python 3.12/macOS with the Python-version check relaxed; that check does not establish general compatibility across environments.

Licensing and provenance

Maintainer: ccm. The repository had no model license specified when this card was added; this card does not assign one. Refer to the dataset card for collection, consent, and reuse limitations. The archived training cache contains survey feature rows, so the native archive should be treated as a training artifact as well as a predictor.