toxic-comment-moderator

lakresi/toxic-comment-moderator

A DistilBERT model fine tuned for multi label toxic comment classification across six categories. Trained on the Jigsaw Toxic Comment Classification dataset. Deployed as a production REST API.

Model Details

Model Description

distilbert-base-uncased fine tuned for multi label classification of toxic content. A linear classification head maps the CLS token representation to six independent output logits, one per toxicity category. Sigmoid activation produces per label probabilities between 0 and 1. Each label is classified independently so a comment can be flagged for multiple categories simultaneously.

Trained with BCEWithLogitsLoss and per label pos_weight to handle significant class imbalance. Per label decision thresholds were tuned on the validation set post training and stored in thresholds.json.

• Developed by: Lloyd Akresi
• Model type: Text classification, multi label
• Language: English
• License: Apache 2.0
• Finetuned from: distilbert-base-uncased

Model Sources

• Repository: GitHub
• Demo: Live API | Interactive Docs

Uses

Direct Use

Classifying text across six toxicity categories for content moderation pipelines. The model accepts raw English text and returns per label probabilities and binary flags based on tuned decision thresholds.

Downstream Use

Can be used as a baseline for further fine tuning on domain specific moderation tasks. The model weights and tokenizer can be loaded directly via HuggingFace Transformers as shown in the How to Get Started section below.

Out of Scope Use

• Automated content removal without human review for borderline or LGBTQ+ identity related sentences. See the bias section below.
• Languages other than English.
• Deployment without awareness of the known bias issues documented below.

Bias, Risks, and Limitations

The model was evaluated against a synthetic probe set of 1,400 sentences spanning 25 demographic groups across religion, race and ethnicity, gender and sexuality, and nationality.

12 of 1,400 sentences (0.9%) were flagged. All flags were on the toxic label only and concentrated entirely in three groups: Gay, Lesbian, and Transgender. No sentence from any of the other 22 groups tested was flagged in any category.

The effect is driven by a copula construction stating identity directly. "The new manager is Gay" scored 0.818 toxic probability. The structurally identical "The new manager is Muslim" scored 0.007. This is a documented artifact of the Jigsaw training corpus where constructions like "that's so gay" are disproportionately represented as insults, causing the model to associate the is/are + LGBTQ+ identity term pattern with toxicity independent of surrounding sentiment or context.

Additional limitations:

• Comments exceeding 512 tokens are truncated and content beyond the limit is not considered
• Trained on Wikipedia comments so performance may vary on platform specific language or slang
• Rare categories threat and severe_toxic have lower F1 scores due to data scarcity and should be treated as indicative rather than definitive

Recommendations

Do not use this model for automated content removal on sentences that state an LGBTQ+ identity using an is/are construction. Route these to human review instead. A post hoc calibration layer or allowlist rule for this specific construction is a reasonable short term mitigation before any automated action is taken.

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from huggingface_hub import hf_hub_download
import torch
import json

model_name = "lakresi/toxic-comment-moderator"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

file_path = hf_hub_download(repo_id=model_name, filename="thresholds.json")
with open(file_path) as f:
    thresholds = json.load(f)

label_cols = ["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]

def classify(text: str):
    inputs = tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        padding="max_length",
        max_length=512
    )
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = torch.sigmoid(logits).squeeze().tolist()
    return {
        label: {
            "probability": round(prob, 4),
            "flagged": prob >= thresholds[label]
        }
        for label, prob in zip(label_cols, probs)
    }

result = classify("You are absolutely worthless and should be ashamed of yourself")
print(result)

Training Details

Training Data

Jigsaw Toxic Comment Classification Challenge dataset. Approximately 145k training examples and 16k validation examples. Wikipedia comments labeled across six binary toxicity categories.

Class distribution is highly imbalanced: toxic appears in ~10.2% of comments, obscene in ~5.9%, insult in ~4.9%, severe_toxic in ~1.0%, identity_hate in ~0.9%, and threat in ~0.3%. Per label pos_weight was applied inversely proportional to each label's positive frequency to address this.

Training Procedure

Preprocessing

Text tokenized with the distilbert-base-uncased tokenizer. Maximum sequence length of 512 tokens with truncation and padding. Comments exceeding 512 tokens are truncated from the right.

Training Hyperparameters

• Training regime: fp32
• Epochs: 5
• Batch size: 256 effective (128 per device across 2x RTX 3090)
• Optimizer: AdamW
• Loss function: BCEWithLogitsLoss with per label pos_weight
• Primary metric for best model: F1 Macro

Speeds, Sizes, Times

• Hardware: 2x NVIDIA RTX 3090 (24GB VRAM each) via Vast.ai
• Training time: ~34 minutes (2,845 steps at ~1.57 iterations per second)
• Evaluation speed: ~410 samples per second on validation set

Evaluation

Testing Data, Factors and Metrics

Testing Data

16,180 held out examples from the Jigsaw Toxic Comment Classification dataset, stratified by label to preserve class distribution.

Factors

Performance disaggregated by toxicity category. Rare labels evaluated separately given their low positive frequency.

Metrics

F1 score per label and overall macro F1. ROC AUC to measure discriminative capability independent of threshold. Both metrics reported because the model can have high ROC AUC (strong ranking ability) while having lower F1 macro (threshold sensitivity), which is exactly what was observed here.

Results

Epoch by epoch training dynamics:

Per label results after threshold tuning:

Summary

Strong performance on frequent labels (toxic, obscene, insult) with F1 above 0.76. Rare labels (threat, severe_toxic, identity_hate) underperform due to data scarcity rather than modeling limitations, as evidenced by the consistently high ROC AUC of 0.987 across all labels. Per label threshold tuning meaningfully improved F1 over the default 0.5 cutoff, particularly for rare labels where lower thresholds improve recall at acceptable precision cost.

Environmental Impact

• Hardware type: 2x NVIDIA RTX 3090 via Vast.ai
• Hours used: ~0.57 hours (34 minutes)
• Cloud provider: Vast.ai
• Compute region: Not specified
• Carbon emitted: Estimated minimal given short training duration

Technical Specifications

Model Architecture and Objective

distilbert-base-uncased with a single linear classification head. Input text is tokenized and passed through 6 transformer layers. The CLS token representation from the final layer is passed through the linear head to produce 6 logits. Sigmoid activation converts logits to independent per label probabilities. Binary cross entropy loss with logits and per label positive weighting.

Compute Infrastructure

Hardware

2x NVIDIA RTX 3090 (24GB VRAM each) via Vast.ai. Multi GPU training managed by HuggingFace Accelerate.

Software

• Python 3.10
• PyTorch
• HuggingFace Transformers
• HuggingFace Accelerate
• HuggingFace Trainer

More Information

Full technical breakdown including training code, bias analysis methodology, API documentation, and Docker deployment instructions are available in the GitHub repository.

Model Card Authors

Lloyd Akresi

Model Card Contact

GitHub