cyb008-baseline-classifier

CYB008 Baseline Classifier

SOC alert triage classifier trained on the CYB008 synthetic SOC alert sample. Predicts which of 5 triage outcome classes (auto_resolved_soar / duplicate_merged / false_positive_closed / true_positive_remediated / true_positive_escalated) an alert will reach, from per-alert features. ALSO ships a leakage diagnostic for the three structural-oracle columns dropped from the feature pipeline.

Read this first. This repo ships two related artifacts: (1) a working baseline classifier for resolution_outcome (the primary product), and (2) a leakage_diagnostic.json file documenting (a) the three structural oracle columns that were dropped from the feature set, and (b) the separate finding that the README's first suggested use case — MITRE ATT&CK tactic classification — is not learnable on this sample. Both files matter; the diagnostic is required reading for anyone evaluating CYB008 for a triage product.

Model overview

Why this task — and what was dropped

The CYB008 README lists alert triage (TP vs FP prediction) as its first suggested use case and MITRE ATT&CK tactic classification as its second. We piloted both on the sample dataset:

• Triage outcome: works honestly. After dropping 3 structural oracle columns, the model achieves acc 0.777 ± 0.007, ROC-AUC 0.955 ± 0.003 on 5-class classification. This is the primary baseline.

• MITRE tactic classification: does NOT work on this sample. Without mitre_technique_id (which is a perfect ATT&CK-by-design oracle), the per-tactic feature distributions are nearly identical (raw_score 0.37–0.39 across all 12 tactics, similar for enriched score and fatigue). A trained XGBoost achieves accuracy 0.08, below the majority baseline of 0.14. The README's stated use case cannot be honestly demonstrated on the sample. See leakage_diagnostic.json for the full finding and our recommendation to the dataset author.

The three structural oracle columns (dropped)

CYB008 has three columns that structurally encode the resolution_outcome label:

With all three present, plain XGBoost achieves 100% test accuracy across all seeds — mechanical, not learned. With all three dropped, accuracy is 0.79 with ROC-AUC 0.96: real learning on a non-trivial 5-class task. The published baseline trains with these three columns excluded.

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal:

• model_xgb.json — gradient-boosted trees
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

On CYB008 the MLP slightly outperforms XGBoost on the test fold (0.767 vs 0.766 accuracy, 0.955 vs 0.952 ROC-AUC at seed 42) — only the second SKU in the XpertSystems baseline catalog where this happens (after CYB007).

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb008-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import transform_single, load_meta, INT_TO_LABEL

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Predict (see inference_example.ipynb for the full pattern)
# Note: do NOT include alert_lifecycle_phase, automation_resolved, or
# escalation_flag in your record - those were the oracle columns.
X = transform_single(my_alert_record, meta)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB008, 9,200 per-alert records:

Stratified split (no natural group key)

CYB008 does not have a natural row-level group key for group-aware splitting:

• 25 analysts — group-aware split would yield only ~4 test analysts
• 5 SOCs — would yield 1 test SOC
• 589 incidents — only 9% of alerts have a non-null incident_id

Alerts are essentially independent given features, so we use StratifiedShuffleSplit (nested 70/15/15), the same approach as CYB001 for network flow classification:

Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 53 features survive after encoding, drawn from:

• Per-alert numeric (9): raw_score, enriched_score, time_in_phase_minutes, queue_depth_at_ingestion, soar_playbook_triggered, sla_breached_flag, mttd_minutes, mttr_minutes, fatigue_score_at_alert
• Per-alert categorical (5, one-hot): alert_severity (7 values), alert_source (8 values), mitre_tactic (12 values), analyst_tier (3 values), siem_platform (8 values)
• Engineered (6): enrichment_lift, log_mttr, log_mttd, queue_pressure, enrichment_per_minute, is_high_confidence

Excluded columns

Oracle columns (dropped to allow honest evaluation):

High-cardinality columns (dropped for tractability):

Partial-oracle features (kept as legitimate observables)

soar_playbook_triggered is a necessary but not sufficient condition for auto_resolved_soar — when 0, the alert is never auto-resolved; when 1, the outcome is auto-resolved 68% of the time but can also be TP-remediated, TP-escalated, FP-closed, or duplicate-merged. This is a legitimate observable that downstream operators would already have on hand at decision time. KEPT in the pipeline.

Evaluation

Test-set metrics, seed 42 (n = 1,380 alerts)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact) — slightly outperforms XGBoost

With 6,440 training rows and 53 features, the MLP has enough data to compete favorably with boosted trees. Both models are published.

Multi-seed robustness (XGBoost, 10 seeds)

Very stable performance — std 0.007 on accuracy is among the tightest in the XpertSystems catalog:

Full per-seed results in multi_seed_results.json. All 10 seeds yielded all 5 classes in the test fold (stratified split guarantees this).

Per-class F1 (seed 42)

The model performs best on false_positive_closed (clearest behavioural profile — low scores, fast resolution by L1 analysts) and duplicate_merged (smallest class but distinctive — duplicate-suppressed severity is a strong tell). The hardest discrimination is between true_positive_remediated and true_positive_escalated — both are genuine threats, differing primarily by whether the alert was closed by the original analyst or passed to a higher tier. In production this matters less because both are TP outcomes; binary TP-vs-FP recall is much higher.

Ablation: which feature groups matter

Four findings:

1. Alert severity carries the dominant signal (drops 25 pp accuracy, 22 pp ROC-AUC). This is intuitive: severity directly drives triage priority, which drives outcome. false_positive severity → false_positive_closed; duplicate_suppressed severity → duplicate_merged.
2. soar_playbook_triggered is the second-strongest signal (drops 15 pp accuracy). It's a partial oracle for the auto_resolved_soar outcome class.
3. MITRE tactic and analyst tier contribute essentially nothing. The model performs marginally better without them — they add noise that the trees over-fit on the training set.
4. Engineered features and timing features are near-flat. The trees recover composites from raw inputs. Kept in the pipeline as a documented baseline reference.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 5 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 53 → 128 → 64 → 5, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters are held internally by XpertSystems.

Limitations

This is a baseline reference, not a production SOC triage system.

1. MITRE tactic classification is unlearnable on this sample. The README lists it as a suggested use case but the per-tactic feature distributions are too similar (raw_score 0.37–0.39 across all 12 tactics). See leakage_diagnostic.json for the full audit. Real SOC data has stronger per-tactic feature signatures.

2. TP-remediated vs TP-escalated is the hardest discrimination. F1 0.56 on TP-escalated is the weakest per-class result. Both are genuine threats; the difference is workflow rather than threat nature. For most operational uses (TP-vs-FP recall, SLA-breach reduction), this confusion does not matter.

3. MLP modestly outperforms XGBoost. Both are shipped; we recommend running both and treating disagreement as a triage triage signal. The boost is modest enough that for production deployment, the choice between them is essentially an engineering preference.

4. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to 12 SOC-operations benchmarks (SANS SOC Survey, IBM Cost of Data Breach, Mandiant M-Trends, Forrester Wave SOAR, Gartner SIEM Magic Quadrant, SOC.OS, CrowdStrike, Splunk State of Security, Verizon DBIR). Real SOC telemetry has different noise characteristics and the structural-oracle pattern documented above (alert_lifecycle_phase deterministically encoding outcome) would not be present in real data — real lifecycle phases transition stochastically. Do not assume metrics transfer end-to-end.

5. 9,200 alerts is a modest training set. The 1,380-alert test fold yields stable multi-seed metrics (std 0.007), but full confidence intervals for downstream production decisions should come from the full ~280k-alert product.

Notes on dataset schema

The CYB008 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

None of these affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Evaluating fit of the CYB008 dataset for your SOC-triage research
• Baseline reference for new model architectures
• Reference example of structural-leakage diagnostics in synthetic SOC datasets — the diagnostic methodology is reusable
• Feature engineering reference for per-alert SOC telemetry

Out-of-scope use

• Production SOC triage decisions on real telemetry
• MITRE ATT&CK tactic prediction (this baseline establishes that task is unlearnable on the sample)
• SLA-breach prediction (also tested as unlearnable on the sample — acc 0.68 vs majority 0.82)
• Any operational decision affecting actual security operations without further validation on your own data

Reproducibility

Outputs above were produced with seed = 42 (published artifact), nested StratifiedShuffleSplit (70/15/15), on the published sample (xpertsystems/cyb008-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems.

Files in this repo

Contact and full product

The full CYB008 dataset contains ~335,000 rows across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative SOC operations and threat intelligence sources (SANS SOC Survey, IBM Cost of Data Breach, Mandiant M-Trends, Forrester Wave SOAR, Gartner SIEM Magic Quadrant, SOC.OS, CrowdStrike, Splunk State of Security, Verizon DBIR). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb008-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
  • https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)
  • https://huggingface.co/xpertsystems/cyb005-baseline-classifier (ransomware actor-tier attribution)
  • https://huggingface.co/xpertsystems/cyb006-baseline-classifier (user risk tier + leakage diagnostic)
  • https://huggingface.co/xpertsystems/cyb007-baseline-classifier (insider threat type)

Citation

@misc{xpertsystems_cyb008_baseline_2026,
  title  = {CYB008 Baseline Classifier: XGBoost and MLP for SOC Alert Triage Outcome Classification, with Structural-Leakage and Unlearnable-Target Diagnostic},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb008-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb008-sample}
}