cyb004-baseline-classifier

CYB004 Baseline Classifier

Phishing campaign phase classifier trained on the CYB004 synthetic phishing campaign sample. Predicts which of 7 lifecycle phases a per-timestep telemetry record belongs to, from observable trajectory and victim-topology features.

Baseline reference, not for production use. This model demonstrates that the CYB004 sample dataset is learnable end-to-end and gives prospective buyers a working starting point. It is not a production email-security platform, SOAR component, or threat detector. See Limitations.

Model overview

Why this task instead of actor-tier attribution?

The CYB004 dataset README leads with "actor attribution modelling — 4-tier classification" as a suggested use case. We piloted that target first and found a serious issue: four features in the dataset (lure_personalisation_score, click_through_rate, credential_submission_rate, target_department_id) are constant per campaign, not per-timestep. They look like per-step features but each takes a single value across all ~40 timesteps of a given campaign.

Because these constants are tier-correlated (especially lure_personalisation_score, which differs systematically across the four actor tiers), they leak tier identity through the campaign-level fingerprint they create. With a 15-campaign test fold, many test campaigns land in the same feature ranges as training campaigns of the same tier, and the model achieves spurious 97%+ accuracy that does not generalize. Removing those features (the honest fix) drops tier prediction to accuracy 0.45, ROC-AUC 0.70 — below majority baseline of 0.59. The full 335k-row CYB004 product, with ~4,800 campaigns, will not have this constraint; the sample at n=100 cannot support honest tier learning.

We pivoted to campaign_phase prediction, which has 3,952 rows of per-timestep data spread across 7 phases with tight timestep windows. It learns cleanly under the same group-aware split: 65% accuracy, ROC-AUC 0.94, stable across 10 seeds. This is a legitimate email-security use case — SOAR playbooks and threat-hunting workflows need to tag what phase of a phishing campaign observed activity belongs to.

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal:

• model_xgb.json — gradient-boosted trees, primary recommendation
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb004-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import (
    transform_single, load_meta, INT_TO_LABEL, build_department_lookup
)

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])
dept_lookup = build_department_lookup("path/to/victim_topology.csv")

# Predict (see inference_example.ipynb for the full pattern)
dept_aggs = dept_lookup.get(my_record["target_department_id"], {})
X = transform_single(my_record, meta, victim_aggregates=dept_aggs)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB004, 3,952 per-timestep trajectory rows from 100 phishing campaigns (~40 timesteps per campaign):

Group-aware split

A single campaign generates ~40 highly-correlated timesteps. Random row-level splitting would put timesteps from the same campaign in both train and test, inflating metrics in a way that does not generalize to new campaigns.

This release uses GroupShuffleSplit by campaign_id (nested, 70/15/15):

All test campaigns are completely unseen during training. Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 53 features survive after encoding, drawn from:

• Per-timestep numeric (7): timestep, emails_sent_cumulative, click_through_rate, credential_submission_rate, gateway_detection_score, lure_personalisation_score, target_department_id
• Per-timestep categorical (2, one-hot): evasion_technique_active, actor_capability_tier
• Victim topology numeric (5): employee_count, privileged_account_density, mfa_enrollment_rate, click_susceptibility_base, email_volume_daily
• Victim topology categorical (5, one-hot): department_type, industry_sector, awareness_training_level, gateway_architecture, dmarc_enforcement_level
• Engineered (6): log_emails_sent, is_gateway_blocked_step, is_evasion_active, is_high_personalisation, has_credential_capture, has_user_engagement

Leakage audit

One column dropped: delivery_outcome (7-class categorical). Its crosstab with campaign_phase shows that no_delivery appears only in the early phases (target_reconnaissance, infrastructure_setup, lure_crafting, credential_harvesting, post_compromise_escalation) and never in email_delivery or victim_engagement. Cell purity 0.36 (uniform baseline 0.14). Keeping it would give the model a near-oracle for partitioning early-vs-mid phases.

No oracle features remain. All retained features have phase-purity under 0.20.

Per-campaign-constant features

Four features (lure_personalisation_score, click_through_rate, credential_submission_rate, target_department_id) are constant within each campaign. For phase prediction this is acceptable — their phase-purity is low, so the model uses them as conditioning context (similar to "we know this is an APT campaign targeting finance" when reasoning about which phase we're in), not as oracle features. They became a problem only for the abandoned actor-tier task.

Evaluation

Test-set metrics, seed 42 (n = 585 timesteps from 15 disjoint campaigns)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact)

Multi-seed robustness (XGBoost, 10 seeds)

Stable performance across seeds — the task learns cleanly, not seed-lucky:

Full per-seed results in multi_seed_results.json. All 10 seeds yielded all 7 classes in the test fold.

Per-class F1 (seed 42) — where the signal is and isn't

Four early phases (target_reconnaissance, infrastructure_setup, lure_crafting, email_delivery) classify cleanly because they sit in tight non-overlapping timestep windows with distinctive features. Three later phases (victim_engagement, credential_harvesting, post_compromise_escalation) overlap substantially in timestep range (14-52, 19-45, 22-52) and share similar behavioural footprints (non-zero click/credential rates, deployed evasion); these are genuinely harder for a flat-tabular model. Sequence models with campaign-level context would help here.

Ablation: which feature groups matter

Three findings:

1. timestep is by far the dominant feature (drops 29 pp when removed, ROC-AUC still 0.81). Phishing campaigns progress through phases over time; where you are in the campaign timeline carries most of the phase signal.
2. Behavioural features contribute ~8 pp accuracy. These are the per-timestep observables (emails sent, gateway score, click rate, evasion technique).
3. Topology and engineered features each contribute ~1 pp. Trees recover most of the engineered features on their own; topology provides modest conditioning context.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 7 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 53 → 128 → 64 → 7, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters (learning rate, batch size, n_estimators, early-stopping patience, weight decay, class-weighting strategy) are held internally by XpertSystems and are not part of this release.

Limitations

This is a baseline reference, not a production email-security system.

1. Mid- and late-phase confusion. Per-class F1 for victim_engagement, credential_harvesting, and post_compromise_escalation is 0.34–0.60. These phases overlap in timestep range and share similar behavioural signatures. Sequence models that consider campaign-level context would help substantially.

2. The pivot away from actor-tier classification is dataset-limited, not method-limited. With 100 campaigns and 4 tiers (some with only 10 campaigns total), tier classification is below majority baseline once leakage-prone features are removed. The full 335k-row CYB004 product provides ~4,800 campaigns; the sample does not.

3. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to email-security and threat-intelligence benchmark targets (Proofpoint State of the Phish, KnowBe4 Industry Benchmark, Cofense PIQ, Mandiant M-Trends, FBI IC3 BEC Report, Verizon DBIR, CISA, APWG). Real phishing telemetry has different noise characteristics, adversary adaptation, and instrumentation gaps. Do not assume metrics transfer.

4. Adversarial robustness not evaluated. The dataset is not adversarially generated; the model has not been red-teamed against evasive lures or novel infrastructure.

5. MLP brittleness on OOD inputs. With ~2.8k training timesteps, the MLP can produce confidently-wrong predictions on hand-crafted records far from the training manifold. XGBoost is more robust. Use both; treat disagreement as a signal for human review.

6. timestep dominance is a property of the dataset. Real phishing telemetry doesn't carry a clean per-campaign normalized timestep — that's a simulator artifact. A buyer transferring this baseline to real campaign telemetry would need to recover an equivalent temporal-position feature (e.g. hours since campaign first observation, position in stage-detection pipeline).

Notes on dataset schema

The CYB004 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

None of these discrepancies affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Evaluating fit of the CYB004 dataset for your email-security or threat-hunting research
• Baseline reference for new model architectures (especially sequence models, which should beat this baseline on the overlapping mid-late phases)
• Teaching and demo for tabular classification on phishing campaign telemetry
• Feature engineering reference for per-timestep campaign data

Out-of-scope use

• Production email security on real campaign telemetry
• Threat hunting / SOAR playbooks on real systems
• Actor attribution (this baseline does not address that task; see why above)
• Adversarial-evasion evaluation (dataset not adversarially generated)
• Any operational security decision

Reproducibility

Outputs above were produced with seed = 42 (published artifact), group-aware nested GroupShuffleSplit (70/15/15 by campaign_id), on the published sample (xpertsystems/cyb004-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems.

Files in this repo

Contact and full product

The full CYB004 dataset contains ~335,000 rows across four files, with calibrated benchmark validation against 12 metrics from email security and threat intelligence sources (Proofpoint, KnowBe4, Cofense, Mandiant, FBI IC3, Verizon, CISA, APWG). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb004-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)

Citation

@misc{xpertsystems_cyb004_baseline_2026,
  title  = {CYB004 Baseline Classifier: XGBoost and MLP for Phishing Campaign Phase Classification},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb004-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb004-sample}
}