engagement-predictor

engagement-predictor

A regression model (scikit-learn Ridge inside a Pipeline) that predicts engagement_score for brand-creator campaign content, part of the Viralst application.

Overview

Given campaign brief attributes (tone, hook type, target audience, creator tier, objective, etc. — typically extracted from a brief document via an LLM parser) and planned content execution details (duration, upload time, caption), the model predicts the expected engagement_score ((likes + comments + shares) / views).

It is served through the POST /engagement/predict endpoint of the Viralst ai/ service.

Training data and methodology

The model is trained on a synthetically generated dataset (2,000 rows, 300 unique briefs, 30 unique brands) produced by training/generate_poc_data.py. Feature-to-target relationships were deliberately encoded (with variable noise per row) so the training pipeline — feature engineering, model selection, evaluation, and serving — could be validated end-to-end ahead of onboarding real campaign data. Results below characterize how well the model recovers the encoded synthetic relationships, not real-world campaign performance.

Training procedure

• Model: sklearn.linear_model.Ridge inside a Pipeline (OneHotEncoder for single-value categorical features, StandardScaler for numeric and multi-hot features).
• Alpha selected via grid search using GroupKFold (grouped by brief_id), so evaluation reflects generalization to unseen briefs rather than leaking the same brief across train/test folds.
• Serialized with skops.io.dump (not raw pickle) for safer loading.

Evaluation results

{
  "alpha": 10.0,
  "n_rows": 2000,
  "n_groups": 300,
  "cv_mae": 0.011419824566734465,
  "cv_rmse": 0.01458130687031906,
  "cv_r2": 0.7102257580449983,
  "baseline_mae": 0.021489796999999998,
  "baseline_rmse": 0.027087352583401354
}

Mean-predictor baseline RMSE: 0.027087352583401354. The final model outperforms this baseline by a wide margin on the synthetic dataset described above.

Feature schema

See config.json in this repo for the full list and order of feature columns the model expects. The canonical definition lives in src/engagement_predictor/features.py in the application repository (https://github.com/<your-org>/viralst).

Training dataset

The raw synthetic dataset used to train this model is included in this repo under dataset/ (brands.csv, briefs.csv, contents.csv — 30 brands, 300 briefs, 2000 content rows). It was generated by training/generate_poc_data.py; see "Training data and methodology" above for why it is synthetic and what it does and does not represent.

How to use

import pandas as pd
import skops.io as sio
from huggingface_hub import hf_hub_download

path = hf_hub_download(repo_id="raviearjun/engagement-predictor", filename="engagement_ridge.skops")
untrusted = sio.get_untrusted_types(file=path)
model = sio.load(path, trusted=untrusted)

# X must be a DataFrame with columns matching config.json -> feature_columns
prediction = model.predict(X)

Limitations

• Trained on synthetic data; does not reflect real-world engagement distributions, platform trends, or actual creator/audience behavior.
• Feature quality depends on upstream LLM extraction from brief documents; prediction quality degrades if that extraction is inaccurate or incomplete.
• Not yet validated against real campaign data from brands outside the training set.

Reproducibility

Training environment (see requirements.txt in this repo for exact pins):

scikit-learn==1.7.2
skops==0.14.0
numpy==1.26.4
pandas==2.3.3