cyb007-baseline-classifier

CYB007 Baseline Classifier

Insider-threat type classifier trained on the CYB007 synthetic insider-threat sample. Predicts which of 3 actor types (negligent_user / malicious_employee / privileged_insider) is behind an observed insider incident from per-timestep trajectory telemetry.

Baseline reference, not for production use. This model demonstrates that the CYB007 sample dataset is learnable end-to-end and gives prospective buyers a working starting point for insider-threat detection research. It is not a production UEBA system, DLP engine, or HR-investigation tool. See Limitations.

Model overview

Why this task — CYB007 ships the README's stated headline use case

This is the second XpertSystems baseline (after CYB005) that ships the dataset's stated headline use case rather than pivoting away from it. The CYB007 README's first suggested use case is "training insider threat classifier models (4-tier actor attribution)", and that is the task this baseline trains on (with one schema correction: the sample data contains 3 of the 4 tiers — compromised_account is absent from the sample).

CYB003 (malware family), CYB004 (phishing actor tier), and CYB006 (threat-actor tier) all had to pivot away from their README headline targets — n=100 groups isn't enough to support group-aware tier classification, and CYB006 in particular had structural distributional leakage. CYB007's 500 incidents (matching CYB005's profile of 500 campaigns × 75 timesteps) is large enough that tier attribution learns honestly under group-aware splitting, with no oracle features and multi-seed std of just 0.012.

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal. Unusually for the XpertSystems baseline catalog, on CYB007 the MLP slightly outperforms XGBoost on the test fold (0.869 vs 0.853 accuracy at seed 42, 0.966 vs 0.963 ROC-AUC):

• model_xgb.json — gradient-boosted trees
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb007-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import transform_single, load_meta, INT_TO_LABEL

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Predict (see inference_example.ipynb for the full pattern)
X = transform_single(my_timestep_record, meta)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB007, 32,500 per-timestep telemetry rows from 500 insider threat incidents (65 timesteps per incident):

Group-aware split

A single incident generates 65 highly-correlated timesteps. Random row-level splitting would put timesteps from the same incident in both train and test, inflating metrics in a way that does not generalize to new incidents.

This release uses GroupShuffleSplit by incident_id (nested, 70/15/15):

All test incidents are completely unseen during training. Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 28 features survive after encoding, drawn from:

• Per-timestep numeric (7): timestep, data_access_volume_mb, privilege_event_count, communication_anomaly_score, dlp_confidence_score, exfiltration_volume_mb_cumulative, behavioural_risk_score
• Per-timestep categorical (3, one-hot): incident_phase (8 values), detection_outcome (4 values), target_data_sensitivity_tier (3 values)
• Engineered (6): log_data_volume, log_cumulative_exfil, exfil_velocity, is_privileged_event, risk_x_dlp_composite, is_late_stage

Leakage audit

Two features have strongly tier-correlated means but with substantial distributional overlap. Neither was dropped:

The honest test: dropping both features collapses accuracy from 0.85 to 0.47 (below the 0.50 majority baseline). This confirms they carry legitimate discriminative signal that defines what privileged_insider means — a privileged user with elevated data access — rather than being an oracle leak.

detection_outcome is a near-oracle for incident phase (purity 0.79, max 1.00 for reconnaissance which is 100% suppressed). But its purity vs tier is uniform (~0.50 across all tiers), so it has no oracle relationship to the target. KEEP.

No columns dropped for this task.

Evaluation

Test-set metrics, seed 42 (n = 4,875 timesteps from 75 disjoint incidents)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact) — slightly outperforms XGBoost

The MLP outperforming XGBoost is unusual for tabular data and unusual within the XpertSystems baseline catalog — CYB001–CYB006 all had XGBoost ahead. With 22,750 training rows and only 28 features, the MLP has enough data to fit cleanly and the tabular advantage of trees is reduced. Both models are published.

Multi-seed robustness (XGBoost, 10 seeds)

Very stable performance — std 0.012 on accuracy is among the tightest in the XpertSystems catalog:

Full per-seed results in multi_seed_results.json. All 10 seeds yielded all 3 tiers in the test fold.

Per-class F1 (seed 42)

The model performs evenly across all three tiers — no class collapse. The strongest performance on privileged_insider despite it being the minority class (20%) confirms that the volume-based behavioural signature (sustained large data access) is reliably discriminative. malicious_employee is the marginally hardest tier because they operate in a middle zone — more aggressive than negligent users but without the privileged access volumes that distinguish insiders.

Ablation: which feature groups matter

Four findings:

1. Volume features carry the overwhelmingly dominant signal (drops 36 pp accuracy, 28 pp ROC-AUC when removed). This is by design — privileged insiders are defined by access to large data volumes, and the synthetic generator models this faithfully.
2. Behavioural features (privilege events, communication anomaly, DLP confidence, risk scores) contribute 14 pp accuracy. They add a second axis of discrimination beyond pure volume.
3. timestep contributes only 1 pp. Tier attribution is largely invariant to where in the incident lifecycle you are — different from phase prediction, which is strongly timestep-driven.
4. Context features (incident_phase, sensitivity tier) and engineered composites are recovered by the trees from raw inputs. They are retained in the pipeline as a documented baseline reference but contribute essentially zero on their own.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 3 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 28 → 128 → 64 → 3, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters are held internally by XpertSystems.

Limitations

This is a baseline reference, not a production insider-threat detection system.

1. The dataset has 3 tiers, not 4. The CYB007 README claims a 4-tier scheme including compromised_account but the sample contains only negligent_user, malicious_employee, and privileged_insider. If your work requires the 4th tier, request regeneration.

2. Volume-feature dominance is a property of the dataset. Real insider-threat telemetry has more variance — some negligent users accidentally trigger large data downloads, some privileged insiders work patiently with small transfers. The sample's per-tier volume distributions overlap, but not as much as in real environments. Buyers should test the model on their own data before assuming the 0.86 accuracy transfers.

3. MLP modestly outperforms XGBoost. With 22,750 training rows, the MLP has enough data to compete favorably. On smaller training sets (n < 1k rows) we would expect XGBoost to be stronger.

4. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to insider-threat research benchmarks (CERT Insider Threat Center, Verizon DBIR, IBM Cost of Insider Threats, Ponemon Institute, MITRE ATT&CK, NIST SP 800-53 / SP 800-207, Securonix, Forrester UEBA, Gartner ZTNA, CrowdStrike, Mandiant). Real insider telemetry has different noise characteristics, and adversarial insiders may deliberately mimic negligent-user patterns. Do not assume metrics transfer.

5. Adversarial robustness not evaluated. The dataset does not simulate insiders deliberately spoofing a different tier's behavioural footprint to evade attribution.

6. The 75-incident test fold is robust but not large. Multi-seed std of 0.012 on accuracy confirms the metric is stable, but full confidence intervals for downstream production decisions should come from the full ~4,800-incident product.

Notes on dataset schema

The CYB007 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

None of these affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Evaluating fit of the CYB007 dataset for your insider-threat research
• Baseline reference for new model architectures (sequence models, graph models considering collusion structure)
• Teaching and demo for multi-class tabular classification on insider-threat telemetry
• Feature engineering reference for per-timestep insider activity

Out-of-scope use

• Production insider-threat detection on real telemetry
• HR investigation or employment decisions
• Adversarial-evasion evaluation (dataset not adversarially generated)
• Any operational or legal decision affecting actual persons

Reproducibility

Outputs above were produced with seed = 42 (published artifact), group-aware nested GroupShuffleSplit (70/15/15 by incident_id), on the published sample (xpertsystems/cyb007-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems.

Files in this repo

Contact and full product

The full CYB007 dataset contains ~335,000 rows across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative insider-threat research sources (CERT Insider Threat Center, Verizon DBIR, IBM Cost of Insider Threats, Ponemon Institute, MITRE ATT&CK, NIST SP 800-53 / SP 800-207, Securonix, Forrester UEBA, Gartner ZTNA, CrowdStrike, Mandiant M-Trends). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb007-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
  • https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)
  • https://huggingface.co/xpertsystems/cyb005-baseline-classifier (ransomware actor-tier attribution)
  • https://huggingface.co/xpertsystems/cyb006-baseline-classifier (user risk tier + leakage diagnostic)

Citation

@misc{xpertsystems_cyb007_baseline_2026,
  title  = {CYB007 Baseline Classifier: XGBoost and MLP for Insider Threat Type Classification},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb007-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb007-sample}
}