cti-attack-mapper-modernbert

cti-attack-mapper

Maps sentences from cyber threat intelligence reports to MITRE ATT&CK technique IDs. 49 techniques, multi-label, sentence level.

Read this before you trust the numbers. On the leak-free split, a fine-tuned ModernBERT scores 0.445 macro-F1. A plain TF-IDF baseline scores 0.454. The transformer does not beat it; they are tied inside the noise of a 151-report corpus. What does beat both is a blend of the two: 0.474 macro-F1 (5-fold mean, standard deviation 0.022), about 0.04 ahead of each base model at the same threshold setting, and ahead in all five folds. One caveat: against TF-IDF at its own best threshold setting the blend's lead shrinks to 0.02 and is no longer outside the noise. Every number here is written by a script in the repo. See Results.

Defensive use only. It labels adversary behavior that has already been written up in public threat reports. It produces no offensive capability.

---

What this repo is for

There are already several ATT&CK classifiers on the Hub. This one exists for three reasons those usually leave out.

1. The baselines are published. Most cyber-ML papers never say what a TF-IDF plus logistic regression floor scores. This one does. On the honest split that floor ties the fine-tuned transformer, and only a blend of the two gets past it. The repo shows by how much and with what spread.
2. The test split is leak-free, and the naive alternative is measured. The corpus is 19k sentences drawn from only 151 reports. The sentence-level random split that most work uses scatters near-identical prose from one report across train and test. That inflates macro-F1 by about 12 percent. Both splits ship here.
3. Everything regenerates. Every number below is written by scripts/04_report.py from JSON that the training and eval scripts produce. None of it is typed by hand.

Quick start

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "ctokx/cti-attack-mapper-modernbert"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

text = "The implant establishes persistence by creating a scheduled task that runs at logon."
with torch.no_grad():
    probs = torch.sigmoid(model(**tok(text, return_tensors="pt", truncation=True)).logits)[0]

for i, p in enumerate(probs):
    if p > 0.5:
        print(model.config.id2label[i], round(float(p), 3))
# T1053.005 0.94

Or use the repo's wrapper, which loads the tuned per-class thresholds and resolves technique names:

from cti_attack.predict import AttackMapper

mapper = AttackMapper("models/modernbert__document")
mapper.predict("The dropper base64-encodes its configuration before writing it to disk.")
# [Prediction(T1027, 'Obfuscated Files or Information', 0.912)]

Results

<!-- BEGIN GENERATED RESULTS - scripts/04_report.py -->

Test-set results

Per-class thresholds tuned on dev. Macro-F1 is the headline metric. It weights all 49 techniques equally, so a handful of common ones cannot cover for a weak long tail.

Same table, single global threshold

Ensemble + cross-validated results

One 70/15/15 split of a 151-document corpus is one draw, so the gap between models can be noise. To check it, here is 5-fold cross-validation grouped by document. Each report lands in exactly one test fold. The blend weight and thresholds are tuned on each fold's own dev set, never on its test set.

Paired per-fold test, per-class thresholds on both sides:

• ensemble vs TF-IDF, same per-class regime: +0.0412, 5/5 folds positive, p = 0.005
• ensemble vs ModernBERT, same per-class regime: +0.0475, 5/5 folds positive, p = 0.010
• ensemble vs TF-IDF at its stronger global regime: +0.0217, 4/5 folds positive, p = 0.100

On the original single 70/15/15 document split, the same ensemble (70% TF-IDF, 30% ModernBERT) scores 0.4965 macro-F1, against 0.4536 for TF-IDF and 0.4454 for ModernBERT.

How to read this: the ensemble has the highest mean and the lowest variance of everything tested. Compared against each base model at the same threshold setting, it wins in all five folds. The one comparison it does not clearly win is against TF-IDF at its own best threshold setting, where the lead is small enough to be noise over five folds. That row is in the table on purpose.

ModernBERT per-technique, document split (12 most frequent)

Head vs tail, document split

Corpus

• 19,178 raw sentences, 18,437 after cleaning and dedup
• 4,024 carry at least one technique (21.8%)
• 49 techniques across 151 source documents
• 714 duplicate sentences removed (370 of them appeared in more than one document)
• dropped for having too few documents to split: T1557.001

<!-- END GENERATED RESULTS -->

How to read this

• No single model beats the linear baseline; a blend of two does. On the leak-free split TF-IDF scores 0.454 and ModernBERT 0.445. The 0.009 gap is inside the noise you get from a 151-document corpus, so they are tied, not that TF-IDF wins. Blending their probabilities (about 70 percent TF-IDF) reaches 0.474 in 5-fold cross-validation and beats each base model at the same threshold setting in all five folds. The one place it does not clearly win is against TF-IDF at its own best threshold setting, where the 0.02 lead is within noise. The blend is also the steadiest model tested, with the smallest fold-to-fold spread.
• The random column is the number you would have reported by accident. It runs about 12 percent higher for the same models on the same data. If you are comparing against work that split at sentence level, that column is the comparable one, and it is not the true one.
• The keyword row is the control. It never trains, so leaked phrasing cannot help it. It moves 2.4 percent between splits, which is the size of plain test-set-composition noise. The trained models move 8 to 19 percent. That difference is what makes the leakage claim a measurement, not an assertion.
• One claim this data does not support: that leakage inflates high-capacity models more than linear ones. Under per-class thresholds TF-IDF inflates more (11.5 percent versus 8.3 percent); under a global threshold the order flips (14.7 percent versus 18.6 percent). A result that changes sign with the threshold setting is not a result, and it is not claimed here.
• The tail is where this model is weak. 0.570 mean F1 on the 15 techniques with at least 20 test examples, 0.391 on the 34 rarer ones. Micro-F1 hides this, which is why macro-F1 leads.

Intended use

In scope. A triage aid for threat-intelligence and detection-engineering work:

• First-pass ATT&CK tagging of a report, for an analyst to correct
• Deciding which reports to read first when clearing a backlog
• Rough coverage analysis: which techniques a body of reporting talks about
• A baseline to beat, with a published harness for beating it

Out of scope.

• Unreviewed labeling. Output is candidates, not conclusions. At macro-F1 around 0.45, roughly half the predictions on rarer techniques are wrong.
• Compliance, audit, or attestation evidence.
• Detecting techniques outside the 49 covered.
• Anything but English prose describing adversary behavior. It is not a malware classifier; it does not read binaries, logs, or code.

Limitations

• 49 techniques, not the full ATT&CK matrix. Anything outside the label set is invisible, including techniques the text plainly describes. No prediction is not evidence of absence.
• 2 of the 49 labels are revoked in current ATT&CK. T1562.001 and T1574.002 were valid when the TRAM corpus was annotated and MITRE has since revoked them. Map them forward before comparing output to current ATT&CK.
• Sentence-level context only. Techniques you can only infer from surrounding paragraphs are under-represented.
• Long-tailed. T1027 has 678 training instances; the rarest retained techniques have about 20.
• 151 source documents. Even the leak-free split is one draw from a small pool. Treat a point or two between models as noise, including the ModernBERT/TF-IDF gap above.
• Annotation is not exhaustive. Some unlabelled sentences do describe techniques, so measured recall is lower than the truth.
• Domain shift is untested. The training text is vendor threat-report prose. Behavior on incident tickets, chat logs, or non-native-English reporting is unknown and probably worse.
• Per-class thresholds are fitted. 49 thresholds tuned on a dev set with few positives per class can overfit, so the single-threshold numbers are reported next to them and you can see the size of the effect.

Training details

The pos_weight is doing real work. 78 percent of sentences carry no label, and without it the model learns to predict nothing.

Training loss reached 0.010 while dev macro-F1 peaked at 0.414. The model memorised the training set instead of generalising from it. With 2,684 labelled training sentences across 49 techniques, that is what you would expect, and it is the likely reason a single transformer does not pull ahead of the linear baseline. A blend of the two does, which points at data, not architecture, as the limit.

Dataset

See DATASET_CARD.md. In short:

• 19,178 raw sentences, 18,437 after cleaning and dedup
• 4,024 carry at least one technique (21.8%)
• 49 techniques across 151 source documents
• 714 duplicate sentences removed. 370 of them appeared in more than one document, which is the leakage the random split exposes.
• T1557.001 dropped: it appears in exactly one document and cannot be split leak-free

Reproducing

pip install -r requirements.txt
python scripts/reproduce_all.py              # ModernBERT only, ~40 min on an RTX 4060
python scripts/reproduce_all.py --all-models # adds DeBERTa-v3 and SecureBERT

Individual steps:

python scripts/01_build_dataset.py                              # build both splits
python scripts/02_run_baselines.py                              # frequency, keyword, TF-IDF
python scripts/03_train.py --model modernbert --scheme document # fine-tune
python scripts/05_ensemble.py --scheme document                 # blend TF-IDF + ModernBERT
python scripts/06_cv.py --folds 5                               # 5-fold document CV + error bars
python scripts/04_report.py                                     # regenerate the tables above
pytest tests/ -q                                                # invariants + smoke test

The published weights are the ModernBERT model. The blend is those weights plus a TF-IDF and logistic regression model, which scripts/05_ensemble.py refits from the included dataset in a few seconds, so the best-scoring setup reproduces without shipping a second binary. The blend weight and every threshold are tuned on dev, never on test.

The test suite checks the claims this card makes: that no document spans two splits, that all 49 techniques reach every split, that no duplicate sentences survive, and that the model loads and returns well-formed output.

Licence and attribution

Apache-2.0. Derived from MITRE CTID TRAM (Apache-2.0). Technique names come from MITRE ATT&CK STIX data under the ATT&CK Terms of Use.

ATT&CK® is a registered trademark of The MITRE Corporation. This project is not affiliated with, endorsed by, or sponsored by The MITRE Corporation.

See NOTICE for full attribution.

Citation

@misc{cti_attack_mapper,
  title  = {cti-attack-mapper: sentence-level MITRE ATT&CK classification
            with leak-free evaluation},
  author = {Varol Cagdas Tok},
  year   = {2026},
  url    = {https://huggingface.co/ctokx/cti-attack-mapper-modernbert},
  note   = {Derived from MITRE CTID TRAM, Apache-2.0}
}