cyb003-baseline-classifier

CYB003 Baseline Classifier

Malware execution-phase classifier trained on the CYB003 synthetic malware behaviour sample. Predicts which of 10 execution phases a per-timestep telemetry record belongs to, from observable behavioural and PE-static features.

Baseline reference, not for production use. This model demonstrates that the CYB003 sample dataset is learnable end-to-end and gives prospective buyers a working starting point. It is not a production sandbox, EDR, or threat-detection system. See Limitations.

Model overview

Why this task instead of malware family classification?

The CYB003 dataset README leads with "training malware family classifiers" as a suggested use case. We piloted that target first and found it is not learnable from the sample dataset under proper group-aware evaluation: with only 100 unique samples spread across 10 families, XGBoost on per-timestep features lands at ~15% accuracy and ROC-AUC ~0.58 — at majority baseline. Per-sample aggregation gives the same result.

This is a sample-size constraint, not a feature-engineering failure. With ~7 samples per family on average, a held-out test set of 15 samples covers at most ~8 families and yields a model that cannot generalize. The full 280k-row CYB003 product, with ~28 samples per family at the sample's distribution, will not have this constraint.

We pivoted to execution_phase prediction, which has 6,000 rows of per-timestep data and learns cleanly: 91% accuracy, ROC-AUC 0.98, stable across seeds. This is a legitimate SOC use case — dynamic-analysis tools and EDR systems regularly need to tag what phase of execution observed malware activity belongs to — and it shows the dataset is well-calibrated even when the headline product use case needs more data.

Two model artifacts are published. They are designed to be used together — disagreement is a useful triage signal:

• model_xgb.json — gradient-boosted trees, primary recommendation
• model_mlp.safetensors — PyTorch MLP in SafeTensors format

Quick start

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb003-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import transform_single, load_meta, INT_TO_LABEL

meta = load_meta(paths["feature_meta.json"])
xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Predict (see inference_example.ipynb for the full pattern)
X = transform_single(my_timestep_record, meta)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB003, 6,000 per-timestep telemetry rows from 100 malware samples (60 timesteps per sample):

Group-aware split

A single malware sample generates 60 highly-correlated timesteps. Random row-level splitting would put timesteps from the same sample in both train and test, inflating metrics in a way that does not generalize to new samples.

This release uses GroupShuffleSplit by sample_id (nested, 70/15/15):

All test samples are completely unseen during training. Class imbalance is addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical feature recipe. 69 features survive after encoding, drawn from:

• Per-timestep numeric (10): timestep, api_call_rate, registry_write_count, network_connection_count, process_injection_flag, c2_beacon_interval_sec, av_signature_hit_flag, sandbox_evasion_flag, lateral_propagation_count, privilege_escalation_flag
• PE static features (11): pe_entropy_mean, pe_entropy_std, import_hash_cluster, section_count, packed_section_ratio, string_entropy_mean, byte_histogram_chi2, code_section_rx_ratio, resource_section_entropy, suspicious_import_count, packer_detected_flag
• Categorical (6, one-hot encoded): malware_family, threat_actor_tier, target_platform, obfuscation_technique, detection_outcome, ep_stack
• Engineered (6): api_burst_score, is_c2_active, is_high_net_volume, is_stealth_step, is_destructive_step, lateral_activity_score

Leakage audit

No categorical feature has phase->phase purity above 0.17 (uniform random baseline is 0.10), so nothing in the dataset is an oracle for the target. The model relies on a mix of timestep (strong but not deterministic) and behavioural features.

Evaluation

Test-set metrics, seed 42 (n = 900 timesteps from 15 disjoint samples)

XGBoost (the published model_xgb.json artifact)

MLP (the published model_mlp.safetensors artifact)

Multi-seed robustness (XGBoost, 10 seeds)

Accuracy and ROC-AUC are tight across seeds — the task is genuinely learnable, not seed-lucky:

Full per-seed results in multi_seed_results.json. All 10 seeds yielded all 10 classes in the test fold, supporting clean multi-class ROC-AUC computation.

Per-class F1 (seed 42) — where the signal is and isn't

Seven phases are near-trivially classified because they sit in tight timestep windows with characteristic behavioural signatures. Three phases — dormancy_dwell, sandbox_evasion_stall, self_destruct_cleanup — scatter across the full 0–59 timestep range and lack distinctive behavioural features (idle/evasion phases have low activity by design), so a flat-tabular event-level model can't reliably disambiguate them. Sequence models that consider neighbouring timesteps would help here.

Ablation: which feature groups matter

Three clear findings:

1. timestep is by far the dominant feature (drops 22 pp when removed, ROC-AUC still 0.93). Malware execution progresses in time, and where you are in that timeline carries most of the phase signal.
2. PE static features are barely used for phase prediction. This is honest: PE features (entropy, packed sections, import hashes) inform family classification, not phase classification. A buyer doing family work should expect to use them; for phase work they can be dropped.
3. Engineered features and behavioural features each contribute ~1 pp. Trees recover most of the engineered features on their own.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 10 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 69 → 128 → 64 → 10, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters (learning rate, batch size, n_estimators, early-stopping patience, weight decay, class-weighting strategy) are held internally by XpertSystems and are not part of this release.

Limitations

This is a baseline reference, not a production sandbox or threat detector.

1. Three phases are genuinely hard at sample size. dormancy_dwell, sandbox_evasion_stall, and self_destruct_cleanup span the full 0–59 timestep range and have low row counts. Per-class F1 = 0.13–0.53. These are the phases by design lacking distinctive moment-to-moment features (the malware is being quiet to evade detection). Sequence models or per-sample aggregation would substantially improve these.

2. The pivot away from malware family classification is dataset-limited, not method-limited. Family classification on 100 samples with 10 classes is at majority baseline. The full 280k-row CYB003 product provides ~5,600 samples and supports proper family classification.

3. Synthetic-vs-real transfer. The dataset is synthetic and calibrated to threat-intelligence and AV-testing benchmark targets (VirusTotal, AV-TEST, MITRE ATT&CK Evaluations, Mandiant M-Trends, CrowdStrike GTR, Verizon DBIR). Real malware telemetry has different noise characteristics, adversary adaptation, and instrumentation gaps. Do not assume metrics transfer.

4. Adversarial robustness not evaluated. The dataset is not adversarially generated; the model has not been red-teamed against evasive samples.

5. MLP brittleness on OOD inputs. With ~4k training timesteps, the MLP can produce confidently-wrong predictions on hand-crafted records far from the training manifold. XGBoost is more robust. Use both; treat disagreement as a signal for human review.

6. timestep dominance is a property of the dataset. Real malware in production doesn't have a clean "timestep" feature on a per-sample 60-step normalized timeline — that's a simulator artifact. A buyer transferring this baseline to real sandbox traces would need to recover an equivalent temporal-position feature from execution-trace timestamps relative to detonation.

Notes on dataset schema

The CYB003 sample dataset README describes some fields differently from the actual schema. The model was trained on the actual schema; this note helps buyers reconcile what they read with what they receive.

The actual per-timestep table also contains rich PE-static features not listed in the README: import_hash_cluster, section_count, packed_section_ratio, string_entropy_mean, byte_histogram_chi2, code_section_rx_ratio, resource_section_entropy, suspicious_import_count. These are excellent features for family classification work and are documented in the model's feature_engineering.py.

None of these discrepancies affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns, not the README descriptions.

Intended use

• Evaluating fit of the CYB003 dataset for your malware-analysis or sandbox-detection research
• Baseline reference for new model architectures (especially sequence models, which should beat this baseline on the late/scattered phases)
• Teaching and demo for tabular classification on malware telemetry
• Feature engineering reference for per-timestep behavioural data

Out-of-scope use

• Production sandbox analysis on real malware
• EDR phase tagging on real systems
• Family attribution (this baseline does not address that task; see why above)
• Adversarial-evasion evaluation (dataset not adversarially generated)
• Any operational security decision

Reproducibility

Outputs above were produced with seed = 42 (published artifact), group-aware nested GroupShuffleSplit (70/15/15 by sample_id), on the published sample (xpertsystems/cyb003-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits.

The training script itself is private to XpertSystems. The published artifacts contain the feature pipeline, model weights, scaler, metadata, and validation results — sufficient to reproduce inference but not training.

Files in this repo

Contact and full product

The full CYB003 dataset contains ~349,000 rows across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative threat intelligence and AV-testing sources (VirusTotal, AV-TEST, MITRE ATT&CK Evaluations, Mandiant, CrowdStrike, Verizon). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

• 📧 [email protected]
• 🌐 https://xpertsystems.ai
• 🗂 Dataset: https://huggingface.co/datasets/xpertsystems/cyb003-sample
• 🤖 Companion models:
  • https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
  • https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)

Citation

@misc{xpertsystems_cyb003_baseline_2026,
  title  = {CYB003 Baseline Classifier: XGBoost and MLP for Malware Execution Phase Classification},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb003-baseline-classifier},
  note   = {Baseline reference model trained on xpertsystems/cyb003-sample}
}