Emc

Model Card for Engineering Misinformation Detection Models

This repository provides a suite of models trained on the Engineering Misinformation Corpus (EMC), introduced in our paper "The Brittleness of Transformer Feature Fusion: A Comparative Study of Model Robustness in Engineering Misinformation Detection (2025)".

The models are designed to detect AI-generated misinformation in safety-critical engineering documents. Each represents a different paradigm: traditional feature-based learning, end-to-end Transformers, and hybrid fusion models.

---

Model Details

Models Included

1. XLM-R (Text Only)

   • Transformer-based classifier (XLM-RoBERTa) trained end-to-end on raw text.
   • Files: config.json, model.safetensors, sentencepiece.bpe.model, tokenizer_config.json, tokenizer.json.

2. Simple Fusion (Text + Features, Concatenation)

   • Combines XLM-R embeddings with a 12-dimensional engineered feature set via naive concatenation.
   • Files: fusion_simple.pt, scaler.pkl.

3. Gated Fusion (Text + Features, Dynamic Arbitration)

   • Combines XLM-R embeddings with engineered features using a gating mechanism to dynamically arbitrate signals.
   • Files: fusion_gated.pt, scaler.pkl.

4. XGBoost + Features (Feature-Only Baseline)

   • Lightweight tree-based model trained solely on 12D engineered features.
   • Files: xgb_model.json, scaler.pkl.

• Developed by: Steve Nwaiwu and collaborators
• Model type: Comparative benchmark (feature-based, Transformer-based, and hybrid fusion approaches)
• Languages: Primarily English, with experimental subsets in ES/FR/DE
• License: CC-BY 4.0
• Finetuned from: xlm-roberta-base (for text-based components)

---

Uses

Direct Use

• Detect AI-generated misinformation in engineering and technical documentation.
• Benchmark adversarial robustness across modeling paradigms.
• Educational use for comparing model interpretability and brittleness.

Downstream Use

• Adaptation to domain-specific technical corpora (e.g., biomedical, aerospace, safety reports).
• Fusion-based approaches can be extended with domain-specific structured features.

Out-of-Scope Use

• Not suitable for general misinformation detection (e.g., politics, social media).
• Not validated for non-technical or informal text domains.
• Should not be used as a standalone safety system without human oversight.

---

Bias, Risks, and Limitations

• Bias in training corpus: While curated for engineering, domain coverage may be uneven across subfields.
• Fusion brittleness: Naive feature fusion models fail under semantic adversarial attacks (e.g., synonym swaps).
• Language limits: Non-English results were confounded by dataset artifacts; multilingual robustness is not yet validated.

Recommendations

• Use XLM-R when robustness to adversarial perturbations is critical.
• Use XGBoost in resource-constrained environments, but pair with human-in-the-loop oversight.
• Avoid naive fusion in safety-critical deployment.

Training Details

• Training Data

• Dataset: Engineering Misinformation Corpus (EMC)

• Splits: Train / Validation / Test

Includes both real engineering documents and AI-generated misinformation.

• Training Procedure

• XLM-R models: finetuned with AdamW optimizer, learning rate warmup, max length 256.

• Fusion models: trained with joint optimization of Transformer encoder + fusion classifier.

• XGBoost: trained with grid-searched hyperparameters (depth, learning rate, estimators).

Technical Specifications

• Architectures

• XLM-RoBERTa: Transformer encoder (base size)

• Simple Fusion: Concatenation of Transformer pooled embedding + 12D features

• Gated Fusion: Transformer pooled embedding + gated feature projection (64D)

• XGBoost: Gradient-boosted decision trees

Software

• PyTorch, Hugging Face Transformers, scikit-learn, XGBoost

Evaluation

• Metrics

• Macro F1, Micro F1, and AP (Average Precision).

• Adversarial Robustness tested under:

• Structural perturbations (typos, casing)

• Semantic/Cue perturbations (synonym swap, term masking)

• Feature-targeted perturbations (numeric & unit corruption)

---

How to Get Started with the Models

Example: Load XGBoost + Features

import joblib, xgboost as xgb
import numpy as np

# Load scaler + model
scaler = joblib.load("scaler.pkl")
model = xgb.XGBClassifier()
model.load_model("xgb_model.json")

# Example features (12D vector extracted with your pipeline)
x = np.array([[120, 30, 5, 0.0, 0.0, 0.05, 10, 3, 1, 0, 12.5, 0.33]])
x_scaled = scaler.transform(x)
pred = model.predict(x_scaled)
print("Predicted class:", pred[0])

### Example: Load XLM-R (Text Only)
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("your-username/emc-models", subfolder="xlmr_text_only")
model = AutoModelForSequenceClassification.from_pretrained("your-username/emc-models", subfolder="xlmr_text_only")

text = "For safety, follow the shutdown procedure."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
print(outputs.logits.softmax(-1))

### Example: Load GATED FUSION
import torch
import joblib
import numpy as np
import pandas as pd
from transformers import AutoTokenizer, XLMRobertaModel

# ---- Load tokenizer and scaler ----
tokenizer = AutoTokenizer.from_pretrained("your-username/emc-models/xlmr_text_only")
scaler = joblib.load("scaler.pkl")

# ---- Define Gated Fusion model (must match training definition) ----
import torch.nn as nn

class GatedFusion(nn.Module):
    def __init__(self, model_name: str, n_feats: int = 12, n_labels: int = 2, feat_proj: int = 64):
        super().__init__()
        self.encoder = XLMRobertaModel.from_pretrained(model_name)
        H = self.encoder.config.hidden_size
        self.fe_proj = nn.Sequential(nn.Linear(n_feats, feat_proj), nn.ReLU())
        self.gate    = nn.Sequential(
            nn.Linear(H + n_feats, 64), nn.ReLU(),
            nn.Linear(64, 1), nn.Sigmoid()
        )
        self.dropout = nn.Dropout(0.1)
        self.classifier = nn.Linear(H + feat_proj, n_labels)

    def forward(self, input_ids, attention_mask, feats):
        out = self.encoder(input_ids=input_ids, attention_mask=attention_mask, return_dict=True)
        mask = attention_mask.unsqueeze(-1).float()
        pooled = (out.last_hidden_state * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-6)
        alpha = self.gate(torch.cat([pooled, feats], dim=1))
        ef    = self.fe_proj(feats)
        fused = torch.cat([pooled, alpha * ef], dim=1)
        return self.classifier(self.dropout(fused))

# ---- Load trained weights ----
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = GatedFusion("xlm-roberta-base", n_feats=12, n_labels=2).to(device)
model.load_state_dict(torch.load("fusion_gated.pt", map_location=device), strict=False)
model.eval()

# ---- Example input ----
text = "For safety, follow the shutdown procedure."
features = np.array([[120, 30, 5, 0.0, 0.0, 0.05, 10, 3, 1, 0, 12.5, 0.33]])  # replace with your feature extractor
features_scaled = scaler.transform(features)

# ---- Tokenize text ----
enc = tokenizer(text, truncation=True, padding="max_length", max_length=256, return_tensors="pt")
input_ids = enc["input_ids"].to(device)
attn_mask = enc["attention_mask"].to(device)
feats = torch.tensor(features_scaled, dtype=torch.float32).to(device)

# ---- Run model ----
with torch.no_grad():
    logits = model(input_ids, attn_mask, feats)
    probs = torch.softmax(logits, dim=-1).cpu().numpy()[0]

print("Prediction probs:", probs)
print("Predicted class:", probs.argmax())  # 0 = Real, 1 = Misinformation


# ---- Citation -----

@article{NWAIWU2025107783,
title = {The brittleness of transformer feature fusion: A comparative study of model robustness in engineering misinformation detection},
journal = {Results in Engineering},
volume = {28},
pages = {107783},
year = {2025},
issn = {2590-1230},
doi = {https://doi.org/10.1016/j.rineng.2025.107783},
url = {https://www.sciencedirect.com/science/article/pii/S2590123025038368},
author = {Steve Nwaiwu and Nipat Jongsawat and Anucha Tungkasthan}}