nci-technique-classifier-v5.2

NCI Technique Classifier v5.2

Multi-label propaganda technique classifier based on ModernBERT, trained to identify 18 propaganda techniques from the SemEval-2020 Task 11 taxonomy.

Model Description

This model is part of the NCI (Narrative Coordination Index) Protocol for detecting coordinated influence operations. It classifies text into 18 propaganda techniques with well-calibrated probability outputs.

Key Improvements in v5.2

• Reduced False Positives: Scientific/factual content false positive rate reduced from 35% (v4) to 8.8%
• Better Calibration: ASL loss with clip=0.02 provides more discriminative probability outputs
• Hard Negatives Training: Trained on v5 dataset with 1000+ hard negative examples (scientific, business, factual content)
• Document-Level Analysis: Works well with full documents, no need for sentence-level splitting

Training Details

• Base Model: answerdotai/ModernBERT-base
• Dataset: synapti/nci-propaganda-v5 (24,037 samples)
• Loss Function: Asymmetric Loss (ASL)
  • gamma_neg: 4.0
  • gamma_pos: 1.0
  • clip: 0.02 (reduced from 0.05 to minimize probability shifting)
• Training: 3 epochs, lr=2e-5, batch_size=16
• Validation: 4/7 tests passed (57%)

Techniques Detected

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "synapti/nci-technique-classifier-v5.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "This is OUTRAGEOUS! They are LYING to you. WAKE UP!"

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.sigmoid(outputs.logits)[0]

# Get techniques with probability > 0.5
LABELS = [
    "Loaded_Language", "Appeal_to_fear-prejudice", "Exaggeration,Minimisation",
    "Repetition", "Flag-Waving", "Name_Calling,Labeling", "Reductio_ad_hitlerum",
    "Black-and-White_Fallacy", "Causal_Oversimplification",
    "Whataboutism,Straw_Men,Red_Herring", "Straw_Man", "Red_Herring", "Doubt",
    "Appeal_to_Authority", "Thought-terminating_Cliches", "Bandwagon", "Slogans",
    "Obfuscation,Intentional_Vagueness,Confusion"
]

for i, (label, prob) in enumerate(zip(LABELS, probs)):
    if prob > 0.5:
        print(f"{label}: {prob:.1%}")

Performance

Validation Results

Key Metrics

• Scientific Report FPR: 8.8% (vs 35% in v4) - 75% reduction
• Factual News FPR: 4.6% (vs 29% in v4) - 84% reduction
• Propaganda Detection: Maintained (73.7% max confidence on propaganda)

Citation

@inproceedings{da-san-martino-etal-2020-semeval,
    title = "{S}em{E}val-2020 Task 11: Detection of Propaganda Techniques in News Articles",
    author = "Da San Martino, Giovanni and others",
    booktitle = "Proceedings of the 14th International Workshop on Semantic Evaluation",
    year = "2020",
}

License

Apache 2.0