cyb006-baseline-classifier

CYB006 Baseline Classifier

User-risk-tier classifier trained on the CYB006 synthetic login activity sample. Predicts which of 3 risk tiers (low / medium / high) a user belongs to, from per-user identity aggregates and non-leaky session aggregates. ALSO ships a leakage diagnostic for the README's stated headline use case (threat-actor tier classification).

Read this first. This repo ships two artifacts: (1) a working baseline classifier for user_risk_tier (the primary product), and (2) a separate diagnostic file (leakage_diagnostic.json) documenting why the README's stated headline use case — 4-class threat-actor tier classification — is not a usable ML task on the sample dataset. Both matter; the diagnostic is required reading for anyone evaluating CYB006 for a threat-detection product.

Model overview

Why this task — and why not threat-actor classification?

The CYB006 README's first suggested use case is "training account takeover (ATO) detection models" and second is "threat-actor tier classification — 4-class with realistic class imbalance". We piloted the threat-actor target first and discovered that the sample dataset contains structural distributional non-overlap between threat-actor and legitimate session populations across at least six independent feature groups:

As a consequence, plain XGBoost achieves 100% test accuracy on threat-actor binary detection (any-actor vs none) across every random seed, and stays at 97% accuracy and AUC 0.99 even with all six oracle feature groups dropped (40+ columns excluded). This is not a useful ML benchmark; it's a property of the synthetic generator. Real identity-security telemetry has substantial overlap between threat and legitimate behaviour, with state-of-the-art detection systems operating at AUC 0.7–0.9, not 1.0.

The diagnostic finding is documented quantitatively in leakage_diagnostic.json and summarised in the Leakage diagnostic section below.

We therefore pivoted to user_risk_tier (3-class user-level classification) as the primary baseline target. This task:

• Has overlapping per-tier feature distributions — no oracle features
• Carries modest real signal (acc 0.66, AUC 0.80 over majority 0.57)
• Targets a legitimate use case (the README lists "Insider threat scoring with composite behavioral indicators")
• Demonstrates honest ML rigor on the dataset

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal:

• model_xgb.json — gradient-boosted trees, primary recommendation
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb006-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import (
    transform_single, load_meta, INT_TO_LABEL,
    compute_session_aggregates_for_user
)

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Compose a per-user record from user_risk_summary row + session aggregates
user_record = user_summary_row.to_dict()
user_record.update(compute_session_aggregates_for_user(user_sessions))

X = transform_single(user_record, meta)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB006, 200 per-user rows from user_risk_summary.csv enriched with per-user session aggregates computed from login_sessions.csv:

The CYB006 README claims a 4-tier scheme (low/medium/high/critical). The sample data contains only 3 — there is no critical tier present.

Stratified split

This is a user-level task (one row per user, 200 users total). Group-aware splitting does not apply since there is no many-rows-per-group structure to leak. We use StratifiedShuffleSplit (nested 70/15/15) to preserve the 3-tier class distribution across folds:

Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 34 features survive after encoding, drawn from:

• Per-user numeric (14, from user_risk_summary.csv): total_login_attempts, successful_logins, failed_logins, mfa_failures, impossible_travel_events, lateral_hop_count, privilege_escalations, account_lockout_count, geo_dispersion_score, login_velocity_score, session_anomaly_rate, ueba_alert_count, overall_identity_risk_score, insider_threat_indicator_score
• Per-user categorical (1, one-hot): peak_privilege_level_accessed (6 values)
• Session aggregates (8, derived from login_sessions.csv): avg_session_duration_seconds, avg_mfa_response_latency_ms, avg_geo_anomaly_score, max_geo_anomaly_score, frac_impossible_travel, n_unique_countries, n_unique_devices, n_unique_applications
• Engineered (6): failed_login_rate, mfa_failure_rate, ueba_alerts_per_session, hops_per_escalation, geo_velocity_composite, composite_anomaly_score

Leakage exclusions

Three columns from user_risk_summary.csv are dropped to avoid contamination:

• threat_actor_flag — perfect oracle for tier='high' subset (only high-tier users can be threat actors)
• account_takeover_flag — 2 positive cases out of 200 (1%); too sparse and oracle-prone
• credential_attack_victim_flag — 1 positive case out of 200 (0.5%); same issue

Four columns from login_sessions.csv are NOT aggregated into session features because they exhibited the structural non-overlap documented in Leakage diagnostic:

• velocity_anomaly_score, session_timestamp_utc, credential_attempt_count, login_outcome

Evaluation

Test-set metrics, seed 42 (n = 30 disjoint users)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact)

Multi-seed robustness (XGBoost, 10 seeds)

Full per-seed results in multi_seed_results.json. With only 30 test users per seed, single-seed accuracy varies materially (0.53–0.87 across seeds). ROC-AUC 0.812 ± 0.048 is the more reliable performance estimate. All 10 seeds yield all 3 tiers in the test fold thanks to stratification.

Per-class F1 (seed 42)

The model performs best on high (the most behaviourally distinct tier — high failed-login rates, frequent impossible travel, elevated anomaly scores) and low (the majority class). The medium tier is hardest, which is the expected behaviour for a 3-tier ordinal task — mid-class samples sit between two boundaries and pick up confusion from both sides.

Ablation: which feature groups matter

Findings:

1. User-level count features matter most (failed logins, lateral hops, MFA failures). Dropping them costs 13 pp accuracy.
2. Risk scores and engineered features each contribute ~10 pp. With only 139 training users, the trees can't fully recover engineered composites from raw inputs.
3. Session aggregates slightly hurt accuracy in seed 42 (gain 3 pp when dropped). With n=200, additional features can crowd the small data; the trees do better with fewer signals when each one is information-dense. Session aggregates are kept in the published pipeline because they help on most other seeds.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 3 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 34 → 128 → 64 → 3, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters are held internally by XpertSystems.

Leakage diagnostic

This is the most important section of the model card. The full diagnostic is in leakage_diagnostic.json. Summary:

Setup: Train an XGBoost binary classifier to predict threat_actor_capability_tier != 'none' from per-session features. Use group-aware split by user_id (15% test = 30 disjoint users). Cumulatively drop suspected oracle feature groups and re-evaluate.

Even after dropping six oracle feature groups (37 columns), the model still achieves 97% test accuracy and AUC 0.99. The leakage is not localised to a few suspect features; it is distributed across the entire feature space because the synthetic generator produces threat-actor sessions that are anomalous on every dimension simultaneously, with no overlap into legitimate behaviour.

Recommendation to dataset author

For threat-actor detection to be a useful ML benchmark on this dataset, the next generator version should introduce distributional overlap between threat-actor and legitimate session populations across all anomaly indicators:

• velocity_anomaly_score: extend non-actor distribution into [0.0, 0.5] and shrink actor to [0.3, 0.9] for substantial overlap in [0.3, 0.5]
• session_timestamp_utc: interleave threat-actor and legitimate sessions across the same time window
• credential_attempt_count: allow some non-actor users to exhibit elevated counts (mistyped passwords, MFA fatigue)
• login_outcome: allow failure_account_locked and success_anomalous for some legitimate sessions
• geo_country_code: include a baseline frequency of risky-country logins among legitimate users (business travel, contractors)
• device_trust_level: allow threat actors to occasionally use compliant devices (token theft scenarios)

Target operating regime: real-world detection AUC 0.7–0.9, not 1.0.

What this means for buyers

If you're evaluating CYB006 for a threat-detection product, you should know that:

• The sample dataset cannot be used to honestly benchmark threat-actor detection models. A trivially regularised model will score 100%, which doesn't differentiate good detection systems from bad ones.
• The user-risk-tier task shipped in this baseline is a legitimate ML benchmark on the sample data. It generalises modestly (AUC 0.81) and is the right starting point for evaluating insider-threat scoring on the sample.
• The full ~1.1M-row CYB006 product may or may not have the same structural property. Confirm with XpertSystems before committing to a threat-detection use case.

Limitations

This is a baseline reference, not a production identity-security system.

1. Small held-out test fold (n=30). With only 30 test users per seed, single-seed metrics swing 0.53–0.87 in accuracy. The multi-seed ROC-AUC of 0.81 ± 0.05 is the reliable estimate. The full ~1.1M-row product would tighten the confidence interval substantially.

2. The medium tier is harder than the others. F1 0.29 on medium (vs 0.92 on high) is expected — ordinal middle classes are typically the hardest under a flat-classification setup.

3. MLP weaker than XGBoost. AUC 0.70 vs 0.80. With only 139 training users, the MLP cannot match boosted trees on tabular data.

4. Threat-actor detection task is not usable on this sample. See Leakage diagnostic above.

5. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to identity-security benchmarks (Microsoft Digital Defense Report, Okta Customer Identity Trends, Verizon DBIR, CISA Joint Advisories, Mandiant M-Trends, MITRE ATT&CK Evaluations). Real identity telemetry has different noise characteristics; do not assume metrics transfer.

6. 3 tiers, not 4. README lists low/medium/high/critical but the data contains only 3. If you need 4-class support, wait for a regenerated sample.

Notes on dataset schema

The CYB006 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

None of these affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Evaluating fit of the CYB006 dataset for your insider-threat or user-risk-scoring research
• Baseline reference for new model architectures
• Reference example of structural-leakage diagnostics in synthetic cybersecurity datasets — the diagnostic methodology in train_classifier.py is reusable
• Feature engineering reference for per-user identity aggregates

Out-of-scope use

• Production identity-security detection on real telemetry
• Threat-actor attribution (this baseline does not address that task; see why above)
• Any operational security or law-enforcement decision

Reproducibility

Outputs above were produced with seed = 42 (published artifact), nested StratifiedShuffleSplit (70/15/15 by user_risk_tier), on the published sample (xpertsystems/cyb006-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems.

Files in this repo

Contact and full product

The full CYB006 dataset contains ~1.1 million rows across four files, with 12 calibrated benchmark validation tests drawn from authoritative identity security and threat intelligence sources (Microsoft Digital Defense Report, Okta Customer Identity Trends, Verizon DBIR, CISA Joint Advisories, Mandiant M-Trends, MITRE ATT&CK Evaluations). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb006-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
  • https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)
  • https://huggingface.co/xpertsystems/cyb005-baseline-classifier (ransomware actor-tier attribution)

Citation

@misc{xpertsystems_cyb006_baseline_2026,
  title  = {CYB006 Baseline Classifier: XGBoost and MLP for User Risk Tier Classification, with Structural-Leakage Diagnostic on Threat-Actor Detection},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb006-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb006-sample}
}