toxishield

Model Card for toxishield

<!-- Provide a quick summary of what the model is/does. -->

A bert-base-uncased model fine-tuned for binary toxicity classification (TOXIC / NON-TOXIC) of GitHub pull request review comments.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This model classifies a single GitHub pull request comment as toxic or non-toxic. It was fine-tuned from google-bert/bert-base-uncased on 38,761 labelled PR comments (the "38k detection dataset") as part of the ToxiShield project, which studies and filters toxicity in software engineering communication. The model is evaluated with stratified 10-fold cross-validation and also exported to ONNX (INT8-quantized) for lightweight/in-browser inference.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: Transformer encoder (BERT), fine-tuned for binary sequence classification
• Language(s) (NLP): English (software-engineering / code-review domain text)
• License: [More Information Needed] (base model google-bert/bert-base-uncased is released under Apache-2.0; license for the fine-tuned weights has not been set)
• Finetuned from model [optional]: google-bert/bert-base-uncased

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: huggingface.co/imranraad/toxishield
• Paper [optional]: https://arxiv.org/abs/2604.14408 (FSE 2026)

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Classifying individual GitHub PR/code-review comments as TOXIC or NON-TOXIC, e.g. via transformers.pipeline("text-classification", ...), to flag potentially toxic comments for human review.

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Integration into CI bots, code-review dashboards, or moderation tooling that triages or surfaces potentially toxic PR comments before/alongside human moderators. The ONNX INT8 export is intended for low-latency or in-browser inference in such tooling.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

• Not intended for languages other than English.
• Not validated on text outside the GitHub PR/code-review domain (e.g. social media, forums, chat, general web text) — the training distribution is short technical comments (median ~80 characters).
• Not intended as the sole basis for moderation, disciplinary, employment, or legal decisions — outputs should be reviewed by a human.
• Not a general-purpose toxicity/hate-speech classifier.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Class imbalance: the training data is 73.9% non-toxic / 26.1% toxic (28,641 vs 10,120 of 38,761 comments), which can bias the model toward the majority (non-toxic) class.
• Recall on the toxic class is the weaker metric: across 10-fold CV, mean recall on toxic comments is 0.954 (± 0.010) vs. mean precision of 0.975 (± 0.003) — i.e. the model is somewhat more likely to miss a toxic comment than to wrongly flag a non-toxic one.
• Manual review of false positives (n=23 sampled) attributes most errors to:
  • Nuance/context misreads, e.g. sarcasm, dry humor, mockery (12/23, ~52%)
  • Technical jargon or inline code snippets read as hostile (6/23, ~26%)
  • Self-deprecating humor (2/23, ~9%)
  • General negative sentiment without toxicity (2/23, ~9%)
• The model was fine-tuned and evaluated on data from GitHub PR comments only, so its calibration will not necessarily transfer to other codebase-hosting platforms or review tools.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Use as a triage/assistive signal rather than an automated blocking mechanism, particularly given the lower recall on the toxic class. Expect false positives on sarcastic, self-deprecating, or jargon-heavy comments, and route model decisions through human review before any moderation action.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline

classifier = pipeline("text-classification", model="<path-or-hub-id-of-saved-model>")
classifier("this the stupidest code ever")[0]["label"]

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

38,761 labelled GitHub PR review comments (38k-detection-dataset, dataset handle [ANONYMIZED-ORG]/38k-dataset-labelled, not yet published): 28,641 NON-TOXIC (label 0) and 10,120 TOXIC (label 1). Comment length ranges from 5 to 998 characters (median ~80). Split into an 80% train / 10% test CSV for the single-run fine-tune; the full dataset is additionally used for stratified 10-fold cross-validation.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

Comments are tokenized with the bert-base-uncased WordPiece tokenizer with truncation. The single-run fine-tune uses dynamic padding (DataCollatorWithPadding); the 10-fold cross-validation run uses fixed max_length=128 padding.

Training Hyperparameters

• Single-run fine-tune: learning rate 2e-5, per-device batch size 16, weight decay 0.01, 1 epoch, evaluate/save every epoch, best checkpoint restored at the end.
• 10-fold cross-validation run: learning rate 2e-5, per-device batch size 16, gradient accumulation 8 (effective batch size 128), weight decay 0.01, up to 20 epochs with early stopping (patience 3), stratified 10-fold split (each fold further split ~89/11 into train/validation).
• Training regime: fp16 mixed precision on GPU when available, fp32 on CPU. <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

bert-base-uncased has ~110M parameters. The best checkpoint is additionally exported to ONNX and INT8-quantized for lighter-weight/in-browser inference. [More Information Needed] on wall-clock training time.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

Stratified 10-fold cross-validation over the full 38,761-sample dataset (per-fold results in results/kfold-metrics/cross_validation_results.csv); the single-run fine-tune is additionally evaluated on the held-out 10% test split.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

No subpopulation disaggregation performed; results are reported per cross-validation fold and averaged.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Accuracy, precision, recall, and F1 (binary, positive class = TOXIC), chosen to capture both overall correctness and toxic-class-specific performance given the class imbalance.

Results

10-fold cross-validation (mean ± std over 10 folds):

Baseline comparison — GPT-4o, zero-shot prompted, on the held-out test split (comparison/openai-detection-inference/):

Summary

The fine-tuned BERT model substantially outperforms zero-shot GPT-4o prompting on this task, most notably on toxic-class recall (0.95 vs. 0.49) — GPT-4o zero-shot misses roughly half of toxic comments, while the fine-tuned model catches the large majority at comparable or better precision.

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: 2× NVIDIA RTX 6000 Ada Generation (49 GB), local workstation
• Hours used: [More Information Needed]
• Cloud Provider: N/A (local/on-prem)
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

bert-base-uncased (12-layer transformer encoder, 110M parameters) with a linear classification head over 2 labels (NON-TOXIC = 0, TOXIC = 1), fine-tuned with cross-entropy loss for binary sequence classification.

Compute Infrastructure

Hardware

2× NVIDIA RTX 6000 Ada Generation (49 GB each).

Software

Python 3.11, PyTorch 2.5.1 (CUDA 12.1 build), Transformers 4.57.6, Datasets 5.0.0, 🤗 Evaluate, scikit-learn, Optimum/ONNX Runtime (for INT8 export).

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

• PR: Pull request (GitHub code-review unit).
• TOXIC / NON-TOXIC: The two output labels (id 1 / id 0 respectively).

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]