distilbert-hatespeech-classifier

Ethical-Content-Moderation

Fine-Tuning DistilBERT for Ethical Content Moderation

Live Demo

Try the model directly in your browser here:
➡️ Ethical Content Moderator Space

Model description

This model fine-tunes distilbert-base-uncased on the Davidson et al. (2017) hate speech and offensive language dataset loaded from HuggingFace. The classifier predicts whether a tweet is:

• (a) hate speech
• (b) offensive but not hate
• (c) neither

Using a frozen DistilBERT base and a custom dense head.

The architecture consists of three dense layers (256 → 128 → 32, LeakyReLU and Swish activations), with dropout and batch normalization to improve generalization.

Intended uses & limitations

Intended uses

• As a starting point for transfer learning in NLP and AI ethics projects

• Academic research on hate speech and offensive language detection

• As a fast, lightweight screening tool for moderating user-generated content (e.g., tweets, comments, reviews)

Limitations Not suitable for real-time production use without further robustness testing

Trained on English Twitter data (2017) — performance on other domains or languages may be poor

Does not guarantee removal of all forms of bias or unfairness; see Fairness & Bias section

Training and evaluation data

Dataset: Davidson et al., 2017 (24K+ English tweets, labeled as hate, offensive, or neither)

Class distribution: Imbalanced (majority: “offensive”; minority: “hate”)

Split: 80% training, 20% validation (stratified)

Training procedure

Frozen base: DistilBERT transformer weights frozen; only dense classifier head is trained.

Loss: Sparse categorical crossentropy

Optimizer: Adam (learning rate = 3e-5)

Batch size: 16

Class weighting: Used to compensate for class imbalance (higher weight for “hate”)

Early stopping: Custom callback at val_accuracy ≥ 0.92

Hardware: Google Colab (Tesla T4 GPU)

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': np.float32(3e-05), 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-07, 'amsgrad': False}
• training_precision: float32

Training results

Best validation accuracy:

0.7402 at epoch 15

Environmental Impact

Training emissions: Estimated at 0.0273 kg CO₂ (CodeCarbon, Colab T4 GPU)

Fairness & Bias

Bias/fairness audit: The model was evaluated on synthetic gender pronoun tests and showed relatively balanced outputs, but biases may remain due to dataset limitations. See Appendix B of the project report for details.

If you use this model, please cite:

Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017). Automated Hate Speech Detection and the Problem of Offensive Language. ICWSM 2017.

William Radiyeh. DistilBERT Hate Speech Classifier (2025). https://huggingface.co/will-rads/distilbert-hatespeech-classifier

Framework versions

• Transformers 4.51.3
• TensorFlow 2.18.0
• Datasets 3.6.0
• Tokenizers 0.21.1