vulnerability-attack-technique-classification-roberta-base-llm-expanded

vulnerability-attack-technique-classification-roberta-base-llm-expanded

This is a negative-result comparison checkpoint, published for reproducibility. For applications, use CIRCL/vulnerability-attack-technique-classification-roberta-base.

A multi-label classifier that suggests MITRE ATT&CK (Enterprise) techniques from a free-text vulnerability description. It is identical to the released gold-only model — same base model (roberta-base), same 53-technique label vocabulary, same seed, same evaluation protocol — except for one thing: its training set folds 984 additional LLM-labeled CVEs (CIRCL/vulnerability-attack-techniques-llm-scaling, labeled by qwen3.5:122b at ≈0.39 agreement with the expert gold labels) into the 972 expert-labeled training rows.

The paper Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion (arXiv:2607.25572) uses this pair of checkpoints to answer the question "can LLM-assisted labeling extend a small expert gold set?" — and the answer is no, not at this agreement level: no reliable ranking improvement at any expansion size from 100 to 984 CVEs, and measurable degradation of rare-technique coverage at scale.

DOI: 10.57967/hf/9624

What this checkpoint shows

Five seeds, corrected protocol (checkpoint selection on the validation split), identical test split — gold-only vs. this configuration (gold + 984 LLM rows):

The pattern: the noisy labels concentrate mass on frequent, "obvious" techniques (micro-F1 up a little) while deflating exactly the rare-technique coverage the expert labels paid for (macro-F1 down ≈3 SEM, no recall@5 gain). On CVE-2021-44077, for example, this checkpoint is more confident than the gold model about T1190 (Exploit Public-Facing Application) but drops the analyst-credited T1505 (Server Software Component) below the prediction threshold and pushes tail techniques such as T1136 (Create Account) from rank 18 to 32. Section 6 of the paper gives the full account, including why an earlier apparent gain turned out to be evaluation noise.

How to use

Same interface as the gold-only model:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

description = "..."  # free-text vulnerability description
inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

for i in probs.argsort(descending=True)[:5]:
    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")

Or side by side with the released model on a live CVE:

vulntrain-infer-attack-classification --cve CVE-2021-44077 \
    --model CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded

Intended uses & limitations

Intended: reproducing and extending the paper's expansion experiments — e.g. contrasting its per-technique behaviour with the gold-only checkpoint, or as a baseline for better silver-labeling strategies (higher-agreement labelers, agreement-weighted losses, human-in-the-loop curation).

Not intended: production use. It is strictly dominated by the gold-only model on ranking and rare-technique metrics, which is why Vulnerability-Lookup deploys the gold-only checkpoint. All limitations of the gold-only model (53-technique vocabulary, KEV-skewed data, English only, 512-token truncation, uncalibrated scores, unverified suggestions) apply here too.

Training and evaluation data

• Expert rows: the 972-row train split of CIRCL/vulnerability-attack-techniques (MITRE CTID gold mappings).
• LLM rows (train only): 984 CVEs from CIRCL/vulnerability-attack-techniques-llm-scaling, labeled by qwen3.5:122b (Ollama, assertive single-call prompt following the CTID methodology) — the best configuration of the paper's labeler benchmark, at ≈0.39 F1 agreement with held-out expert labels.
• The label vocabulary stays frozen to the gold train split, and the validation (106) and test (118) splits contain only expert-labeled rows; checkpoint selection uses the validation split.

Training procedure

Binary cross-entropy over 53 sigmoid outputs with balanced per-label pos_weight, trained with vulntrain-train-attack-classification (VulnTrain), like the gold-only model — only the training set differs (1,956 rows instead of 972).

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 40
• max_length: 512
• loss: BCEWithLogitsLoss, balanced pos_weight
• checkpoint selection: best macro-F1 on the validation split

Training results

Framework versions

• Transformers 5.13.0
• Pytorch 2.12.1+cu130
• Datasets 4.8.5
• Tokenizers 0.22.2

Related artifacts

Citation

@misc{bonhomme2026mappingcvesmitreattck,
      title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion},
      author={Cédric Bonhomme and Alexandre Dulaunoy},
      year={2026},
      eprint={2607.25572},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.25572},
}

Acknowledgements

Developed at CIRCL in the context of the AIPITCH project, co-funded by the European Union.