cyb009-baseline-classifier

CYB009 Baseline Classifier

Vulnerability classification baseline (8-class) trained on the CYB009 synthetic vulnerability intelligence sample. The primary artifact value of this repo is leakage_diagnostic.json — the most comprehensive structural-leakage audit in the XpertSystems baseline catalog, documenting 8 oracle paths and 6 unlearnable README-suggested targets. The classifier itself is the catalog's weakest baseline by design (acc 0.244 vs majority 0.176), included to show that vulnerability_class is the ONLY README-headline target that learns honestly on this sample.

Read this first. This repo ships three artifacts in priority order:

1. leakage_diagnostic.json — comprehensive audit of 8 oracle paths discovered on CYB009 and 6 README-suggested targets that are unlearnable on the sample after honest leak removal.
2. A working classifier for vulnerability_class 8-class — the only README target that learns honestly on this sample, and the weakest baseline in the XpertSystems catalog by design.
3. A feature engineering reference (feature_engineering.py).

If you came here looking for a strong baseline, you will be disappointed. If you came here to understand why the CYB009 sample has hard-to-detect structural label-feature determinism, the diagnostic is exactly the artifact you need.

Model overview

Why this task — and the journey to get here

The CYB009 README lists 11 suggested use cases. We piloted every README-headline target and found pervasive structural leakage. The abandoned candidates, in order of how we discovered them:

Initial candidate: exploit_maturity_final 4-class (ABANDONED)

The most natural target — 4-class (unproven/PoC/functional/weaponised), n=2638 well-balanced (36/27/25/12%), maps directly to EPSS calibration. Initial feasibility hit acc 0.74, macro-F1 0.72, ROC-AUC 0.91 vs majority 0.36 — a +38pp lift looked excellent.

Then we found the leak. cvss_temporal_score_final divided by cvss_base_score clusters near-deterministically per maturity tier:

This is exactly the CVSS v3.1 Exploit Code Maturity multiplier (unproven 0.91 / PoC 0.94 / functional 0.97 / high or weaponised 1.00), combined with other near-constant Temporal factors (Remediation Level, Report Confidence). The cvss_temporal/cvss_base ratio uniquely identifies the maturity tier.

Drop cvss_temporal_score_final → accuracy collapses to 0.31 (below majority 0.36). The target is structurally unlearnable on the sample once the oracle is removed.

Other 5 candidates: also unlearnable after honest leak removal

All five rare-event binaries are oracled by time_to_exploit_days (-1 sentinel) or time_to_remediate_days (120 sentinel) at full features; after honest leak removal, all are at-or-below majority.

Per-timestep multi-class targets: state-machine oracles

lifecycle_phase, patch_status, and remediation_status on vulnerability_records.csv form a tightly-coupled state machine:

• lifecycle_phase = residual_risk_review → 100% remediated
• lifecycle_phase = discovery → 100% undetected
• lifecycle_phase = remediation_deployment → 100% in_remediation
• patch_status = deployed → 100% remediated

Naive evaluation on these targets reaches accuracy 0.95-0.98, but any two of the three deterministically pin the third. None of these is a viable independent ML target on the sample.

severity_class: 100% mechanical CVSS function

Observed cvss_base_score ranges per severity match CVSS v3.1 exactly: critical [9.0, 10.0], high [7.0, 9.0], medium [4.0, 7.0], low [1.8, 4.0]. Predicting severity is trivial with CVSS; below majority (acc 0.55 vs 0.51) without it.

vulnerability_class 8-class: the only honest target — and the baseline ships

After exhausting the README-suggested targets, vulnerability_class is the only one that learns honestly:

• acc 0.244 ± 0.023, macro-F1 0.230 ± 0.024, ROC-AUC 0.687 ± 0.014
• +7pp lift over majority (the catalog's smallest)
• All 8 classes represented (per-class F1 0.09-0.33)
• No oracle feature — modest signal genuinely spread across CVSS, EPSS, asset context, and binary flags

This is the weakest baseline in the XpertSystems catalog by design. The full ~487k-row product would tighten per-class signal materially. The dataset roadmap recommendations in leakage_diagnostic.json describe what would make CYB009's headline targets viable on the sample.

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download, snapshot_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb009-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import (
    transform_single, load_meta, build_asset_lookup, INT_TO_LABEL,
)

meta = load_meta(paths["feature_meta.json"])

# Asset features are joined from asset_inventory.csv at inference time
ds = snapshot_download("xpertsystems/cyb009-sample", repo_type="dataset")
asset_lookup = build_asset_lookup(f"{ds}/asset_inventory.csv")

xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Predict (see inference_example.ipynb for the full pattern)
# Note: do NOT include exploit_maturity_final, cvss_temporal_score_final,
# time_to_exploit_days, time_to_remediate_days, patch_lag_days, or
# risk_score_composite - those were the outcome-leak columns.
X = transform_single(my_vuln_record, meta, asset_lookup=asset_lookup)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB009, 2,638 per-vulnerability records:

Stratified split

Per-vulnerability task (one row per vuln in vuln_summary.csv), StratifiedShuffleSplit nested 70/15/15:

Class imbalance addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical recipe. 57 features survive after encoding, drawn from:

• Per-vulnerability numeric (10): cvss_base_score, epss_score_final, plus 8 binary post-hoc flags
• Per-vulnerability categorical (1, one-hot): severity_class (4 values, CVSS-derived but useful as feature)
• Asset features (joined from asset_inventory.csv): 8 numeric
  • 4 categorical (asset_type, criticality_tier, environment_type, os_family)
• Engineered (5): log_epss, is_high_cvss, exposure_severity_composite, risk_flag_count, epss_x_base

Excluded columns (outcome leaks)

Evaluation

Test-set metrics, seed 42 (n = 396 vulnerabilities)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact)

MLP and XGBoost are within noise of each other on this task — both are publishing the same modest honest signal.

Multi-seed robustness (XGBoost, 10 seeds)

All 10 seeds yielded all 8 classes in the test fold (stratified split guarantees this). Full per-seed results in multi_seed_results.json.

Per-class F1 (seed 42)

memory_corruption (highest mean CVSS at 8.3) and information_disclosure (lowest mean CVSS at 5.4) are the most distinctive classes. logic_flaw is the hardest — its feature distribution overlaps closely with everything else.

Ablation: which feature groups matter

Three findings:

1. No feature group is dominant. Largest single drop is 2.5pp (CVSS features). Every group contributes a little; nothing contributes a lot. The signal is genuinely diffuse.
2. CVSS and asset features carry the most signal (~2pp each), consistent with the observation that per-class CVSS means differ (5.4 to 8.3) and asset features modestly inform class.
3. EPSS features slightly hurt on this task (+1pp without them). EPSS is intended for exploitation prediction, not class prediction; on this sample it acts as small additional noise.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 8 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 57 → 128 → 64 → 8, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters are held internally by XpertSystems.

Limitations

This is a baseline reference, not a production vulnerability classifier.

1. The headline finding is the leakage diagnostic, not the classifier. Read leakage_diagnostic.json first. The classifier demonstrates that vulnerability_class is the only README-suggested target that learns honestly on the sample.

2. Per-class F1 ranges 0.09–0.33. The model is more confident on memory_corruption and information_disclosure than on logic_flaw and auth_access_control. For production use, expect different error patterns by class.

3. No feature group contributes more than 3pp accuracy. The model has no single decisive signal; instead it integrates many weakly-informative features. Removing any one group has minimal impact.

4. Synthetic-vs-real transfer. The dataset is synthetic, calibrated to 12 benchmarks from authoritative vulnerability intelligence sources (NIST NVD, EPSS v3, CISA KEV, Mandiant, Verizon DBIR, Rapid7, Qualys, Tenable). Real vulnerability telemetry has different noise characteristics — in particular, the structural-oracle patterns documented in leakage_diagnostic.json (CVSS temporal multipliers, sentinel-coded time fields, lifecycle state-machine determinism) would not be present in real data with comparable density. Real data has stochastic transitions and observation noise.

5. 2,638 vulnerabilities is a modest training set for 8 classes. The 396-vulnerability test fold yields stable multi-seed metrics (std 0.023) but per-class confidence intervals are wide. The full ~487k-row product has materially more data per class.

Notes on dataset schema

The CYB009 sample dataset README describes some fields differently from the actual schema. This note helps buyers reconcile what they read with what they receive.

None of these affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

• Reading the leakage diagnostic — the primary value of this repo. Reusable methodology for any synthetic vulnerability dataset.
• Evaluating fit of the CYB009 dataset for your research, with open knowledge of the structural-oracle patterns
• Honest baseline reference for the only README-suggested target that learns on the sample
• Feature engineering reference for per-vulnerability ML

Out-of-scope use

• Production vulnerability triage on real telemetry
• Exploit maturity prediction — README headline target, unlearnable on the sample after honest leak removal
• Zero-day / KEV / supply-chain prediction — README headline targets, unlearnable as rare-event binaries on the sample
• SLA breach prediction — README headline target, unlearnable after honest leak removal
• Any operational security decision without further validation on real data

Reproducibility

Outputs above were produced with seed = 42 (published artifact), nested StratifiedShuffleSplit (70/15/15), on the published sample (xpertsystems/cyb009-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits (std 0.023 on accuracy).

The training script itself is private to XpertSystems.

Files in this repo

Contact and full product

The full CYB009 dataset contains ~487,000 vulnerability records across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative vulnerability intelligence sources (NIST NVD, EPSS v3, CISA KEV, Mandiant, Verizon DBIR, Rapid7, Qualys, Tenable). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb009-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
  • https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
  • https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)
  • https://huggingface.co/xpertsystems/cyb005-baseline-classifier (ransomware actor-tier attribution)
  • https://huggingface.co/xpertsystems/cyb006-baseline-classifier (user risk tier + leakage diagnostic)
  • https://huggingface.co/xpertsystems/cyb007-baseline-classifier (insider threat type)
  • https://huggingface.co/xpertsystems/cyb008-baseline-classifier (SOC alert triage + leakage diagnostic)

Citation

@misc{xpertsystems_cyb009_baseline_2026,
  title  = {CYB009 Baseline Classifier: XGBoost and MLP for Vulnerability Classification, with the XpertSystems Catalog's Most Comprehensive Structural-Leakage Audit},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb009-baseline-classifier},
  note   = {Reference baseline + 8-oracle-path leakage diagnostic on xpertsystems/cyb009-sample}
}