distilbert-dbpedia-14-97acc

rpudathu/distilbert-dbpedia-14-97acc

Model Description

This model is a fine-tuned version of DistilBERT for multi-class text classification on the DBpedia 14 dataset. It classifies short text passages into one of 14 ontology-based categories such as Company, Artist, Athlete, etc. The model achieves high accuracy (~97%) and is optimized for fast inference with reduced computational cost compared to standard BERT.

• Developed by: PUDATHU RAMA KRISHNA
• Funded by: Ardentix
• Model type: DistilBERT (Transformer-based text classifier)
• Language(s): English
• License: Apache 2.0
• Finetuned from model: distilbert-base-uncased

---

Training Procedure

Training Data

The model was trained on the DBpedia 14 dataset, which contains Wikipedia-derived structured content categorized into 14 classes.

Preprocessing

• Removed noise (HTML tags, special characters)
• Lowercased all text
• Tokenization using DistilBERT tokenizer
• Synthetic data augmentation applied to improve generalization

Training Hyperparameters

• Epochs: 3
• Batch size: 16
• Learning rate: 2e-5
• Optimizer: AdamW
• Max sequence length: 256
• Training regime: fp16 mixed precision
• Loss function: CrossEntropyLoss

Training Platform

• Environment: Google Colab
• Framework: Hugging Face Transformers + PyTorch

---

Evaluation

Testing Data

• Standard DBpedia 14 test split (~70k samples)

Metrics

• Accuracy
• Precision
• Recall
• F1 Score

Results

• Accuracy: 97.0%
• F1 Score (macro): 0.97
• Precision (macro): 0.97
• Recall (macro): 0.97

Summary

The model performs strongly across all classes with balanced precision and recall, making it suitable for general-purpose topic classification tasks.

---

Model Examination

• Captures semantic meaning effectively using transformer attention
• Works well on structured and semi-structured text
• Limitations:
  • Struggles with very short or ambiguous inputs
  • May not generalize well to out-of-domain data

---

Environmental Impact

• Hardware Type: NVIDIA Tesla T4 GPU
• Hours used: ~2 hours
• Cloud Provider: Google Colab
• Compute Region: Likely US-based
• Carbon Emitted: ~0.02 kg CO2eq (estimated)

---

Technical Specifications

Architecture

• DistilBERT (6-layer transformer)
• Hidden size: 768
• Task: Sequence classification (14 classes)

Software

• Python 3.10
• PyTorch
• Hugging Face Transformers
• Datasets library

---

Usage

from transformers import pipeline

classifier = pipeline("text-classification", model="rpudathu/distilbert-dbpedia-14-97acc")

result = classifier("Apple is a technology company based in California.")
print(result)