Downloads · 30 days
0
cassandra-anon/cassandra-usenix-asl-tram2
cassandra-usenix-asl-tram2 is a text classification model from cassandra-anon. Use it when you need a label for a piece of text. The card lists the license as other.
Six independently trained CTI-BERT checkpoints that extract MITRE ATT&CK techniques from cyber threat intelligence (CTI) reports. This repository holds the recommended TRAM2 deployment system from the CASSANDRA paper,…
Downloads · 30 days
0
Access
Public
Updated Aug 17, 2026
Repo size
3 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3 GB · 99%
From the Hugging Face model README
Six independently trained CTI-BERT checkpoints that extract MITRE ATT&CK techniques from cyber
threat intelligence (CTI) reports. This repository holds the recommended TRAM2 deployment
system from the CASSANDRA paper, together with the per-sentence predictions needed to
reproduce the TRAM2 ASL operating points below without a GPU. It does not reproduce the whole
paper; the companion cassandra-usenix-evaluation-artifacts repository does that.
An extractor here is a complete configuration, not a checkpoint: encoder, classification head, loss, decision threshold, and how many checkpoints are averaged at inference. All three are stated for every number below, because checkpoint count changes both the score and the cost.
Per-report F1 on the TRAM2 test split of Büchel et al., averaged over the 30 test reports that have at least one in-vocabulary technique (31 reports total; one has no in-vocabulary ground truth and is excluded, as in the source protocol).
| Configuration | Checkpoints | Threshold | Report F1 | 95% CI |
|---|---|---|---|---|
| Recommended deployment (seeds 42, 123, 456) | 3 | fixed τ = 0.50 | 76.70 | [70.43, 82.27] |
| All checkpoints | 6 | fixed τ = 0.50 | 76.91 | [70.77, 82.48] |
| Maximum observed | 6 | dev-selected τ = 0.63 | 77.43 | [71.57, 82.65] |
Intervals are percentile bootstrap over the scored reports, 2,000 resamples, seeded with
numpy.random.default_rng(0). The bundled verify.py ships that exact implementation and
reproduces every point estimate and every interval endpoint in this table; it asserts them and
fails on any mismatch. The τ = 0.63 threshold was selected on a held-out development split, never
on test.
Per-checkpoint scores at τ = 0.50, with the weight variant selected on development macro-F1:
| Seed | Report F1 | Weights selected |
|---|---|---|
| 42 | 72.92 | SWA |
| 123 | 74.36 | EMA |
| 456 | 73.33 | base |
| 789 | 73.11 | SWA |
| 2024 | 70.31 | SWA |
| 3141 | 69.63 | EMA |
Three checkpoints, not six, is the recommendation. Across all 63 non-empty subsets of these six checkpoints, mean F1 rises 72.28 → 76.34 from one to three checkpoints (+4.07, 95% CI [2.70, 5.58], p < 0.001) but only 76.34 → 76.91 from three to six (+0.57, 95% CI [−0.51, 1.63], p = 0.317). Three checkpoints capture 87.7% of the available aggregation gain for half the storage and half the encoder passes.
The gain from averaging is driven primarily by precision. Precision rises 65.4% → 72.9% from one checkpoint to six, while recall varies little and without trend (86.7%–87.5%). A report's prediction is the union over its sentence-level decisions, so a single spurious firing anywhere in a report inserts a technique that no other sentence can retract. Averaging independently trained checkpoints attenuates firings that are specific to one training run while leaving evidence that several runs agree on, which is consistent with the effect being variance reduction — and with its returns diminishing as checkpoints are added.
The published Llama-3.1-8B result on this benchmark is 72.50 report-F1. The three-checkpoint configuration here evaluates 373.4M parameters per sentence against that model's 8B, roughly 21× fewer. This is a published reference rather than a controlled experiment: training and inference implementations differ, so the comparison should not be read as isolating model scale or architecture.
AMD Ryzen 7 7700X, 16 threads, batch size 16, sequential checkpoint inference, seven timed repeats after warm-up, on real TRAM2 sentences.
| Checkpoints | Storage | Peak RSS | 30 sentences | Throughput |
|---|---|---|---|---|
| 1 | 498 MB | 1.39 GB | 0.93 s | 32.2 sent/s |
| 3 (recommended) | 1.49 GB | 2.45 GB | 2.66 s | 11.3 sent/s |
| 6 | 2.99 GB | 3.88 GB | 5.21 s | 5.8 sent/s |
The full 3,820-sentence TRAM2 test set labels in under six minutes on this CPU at the recommended operating point. Latency and memory are implementation- and machine-specific; between-session timing varied by 25–30%.
ibm-research/CTI-BERT. The shared base encoder is 124,441,344 parameters;
with the label-aware head each released TRAM2 checkpoint totals 124,480,513.[CLS] summary. The head is custom, so loading requires
the bundled modeling.py.DataParallel replica threads, the classification head trained in float32.random_state=0,
identical across seeds.pip install -r requirements.txt
python inference_example.py --threshold 0.5 # predetermined 3-checkpoint system (76.70)
python inference_example.py --all-seeds # all six checkpoints (76.91)
inference_example.py defaults to the predetermined three checkpoints (seeds 42, 123, 456) —
the recommended deployment system, and the configuration reported as 76.70. Pass --all-seeds for
the six-checkpoint system at roughly twice the storage and encoder passes per sentence.
from modeling import load_seed, predict_ensemble
seeds = [load_seed(f"seeds/seed-{s}") for s in (42, 123, 456)] # recommended: 3 checkpoints
preds = predict_ensemble(
seeds,
["The malware created a scheduled task to maintain persistence."],
threshold=0.5,
)
Use τ = 0.50 with three checkpoints for the recommended operating point, or τ = 0.63 with all six to reproduce the maximum observed score.
Every seed directory contains seed_probs.npz with the per-sentence sigmoid probabilities, the
test gold labels, and the report boundaries. This is sufficient to recompute every F1, confidence
interval, threshold sweep, and subset score in this card without running the model:
python verify.py # requires only numpy
It recomputes 76.70, 76.91, and 77.43, the per-seed scores, the subset curve, and the confidence intervals from the released predictions, and fails loudly on any mismatch. It verifies the TRAM2 ASL operating points only; the companion artifacts repository verifies the paper's other results.
The code in this repository (modeling.py, inference_example.py, verify.py) is released
under Apache-2.0; see LICENSE.
The weights are a derivative of ibm-research/CTI-BERT. That model card currently declares no
license metadata, so we cannot assert that Apache-2.0 extends to the derived weights. Users
intending to redistribute or deploy commercially should confirm the upstream terms with the base
model's authors. We make no license claim over the fine-tuned weights beyond the code.
cassandra-usenix-evaluation-artifacts — all preserved prediction outputs, ablation and matched
controlled comparisons, subset analysis, measured deployment costs, and analysis scripts.Paper under anonymous review; citation will be added on acceptance.