cti-ner-securebert

cti-ner-securebert (clean LLM-gold retrain, v2)

SecureBERT (ehsanaghaei/SecureBERT, RoBERTa architecture, domain-pretrained on ~2M security documents) fine-tuned as a BIO token classifier for cyber threat intelligence NER on CISA advisories: 16 entity types (attack technique id/name, CVE/CWE identifiers, file hash/name/path, IP address, email address, malware family, threat actor group, hacking tool, tactic, TLP classification, YARA rule name, affected software product).

This version replaces the earlier checkpoint trained on dictionary-silver labels. Training gold is LLM-proposed (Claude Sonnet 5, exhaustive extraction under per-label span-format validators) with the evaluation split human-curated. See the paper in the repository below for the methodology and the circularity analysis that motivated the switch.

Evaluation (measured on this checkpoint)

Document-level span scoring on the held-out 8-advisory test split (strict = exact span + label):

Best of the four fine-tuned architectures studied (RoBERTa-base 0.659, RoBERTa-large 0.648, GLiNER 0.560 on the identical gold and harness).

Per-label strict F1 highlights: attack technique id 0.99, cwe identifier 1.00, tlp classification 0.97, email address 0.98, cve identifier 0.95, ip address 0.91; hardest labels remain threat actor group 0.32 and hacking tool 0.48.

Training details

• Data: 66 training advisories (2,799 chunks), 8 val, 8 test; document-level split committed in the repository (data/gold_v2/split_full.json).
• 8 epochs, lr 2e-5, batch size 16, seed 0, wordpiece-budget sliding windows; best epoch by validation strict F1 (epoch 7, val 0.664).
• Trained on Apple Silicon (MPS). Reproduces the paper's CUDA-trained result (0.712 strict micro) within seed/hardware variance; the paper's committed numbers come from data/gold_v2/results/trained-securebert.json.

Usage

Inference should replicate the sliding-window word-tagging procedure in ner/finetune_roberta.py (see notebooks/hf_models_demo.ipynb in the repository for a worked example); a naive single-pass tokenizer call truncates long, hash-heavy advisory text.

Links

• Code, gold data, paper: https://github.com/dandye/nlp_capstone
• Companion models: pzryathzsdhc/cti-ner-roberta-large, pzryathzsdhc/cti-ner-gliner