human-vs-ai-text-classifier

Human vs. AI Text Classifier

Model Description

A comprehensive ensemble-based text classification system that distinguishes between human-written and AI-generated text with high accuracy. This implementation combines traditional machine learning (Logistic Regression, Random Forest, SVM, XGBoost) and deep learning approaches (BiLSTM with Attention, BERT) using advanced ensemble techniques.

Key Features:

• 6 diverse classifiers (4 traditional ML + 2 deep learning)
• 5,015-dimensional hybrid feature space (5,000 TF-IDF + 15 linguistic features)
• 4 ensemble strategies (Hard/Soft Voting, Weighted Average, Stacking)
• 99.59% F1-score with weighted ensemble
• Balanced performance (99.59% precision, recall, and accuracy)
• Trained on 52,452 samples from HC3 dataset

Model Architecture

Input Text
    ↓
[Feature Engineering]
    ├─→ TF-IDF Vectorization (5,000 features)
    │   - Unigrams & Bigrams
    │   - Max DF: 0.95, Min DF: 2
    │
    └─→ Linguistic Features (15 features)
        - Text length, word count, sentence count
        - Lexical diversity (TTR)
        - Stopword/punctuation ratios
        - Statistical text properties
    ↓
Multi-Modal Feature Vector (5,015 dimensions)
    ↓
┌──────────────────────────────────────────────┐
│         Base Classifiers (Parallel)          │
├──────────────────────────────────────────────┤
│ Traditional ML          │ Deep Learning      │
├─────────────────────────┼────────────────────┤
│ • Logistic Regression   │ • BERT             │
│ • Random Forest (200)   │   (bert-base)      │
│ • SVM (RBF kernel)      │ • BiLSTM+Attention │
│ • XGBoost (200 trees)   │   (64 units)       │
└─────────────────────────┴────────────────────┘
    ↓
[Ensemble Aggregation]
    ├─→ Hard Voting (Majority vote)
    ├─→ Soft Voting (Probability averaging)
    ├─→ Weighted Average (Optimized weights)
    └─→ Stacking (Meta-learner: Logistic Regression)
    ↓
Final Prediction: Human (0) or AI (1)

Individual Model Specifications:

Performance

Individual Models

Ensemble Methods

Optimized Ensemble Weights:

• XGBoost: 0.25
• Logistic Regression: 0.20
• BERT: 0.20
• SVM: 0.15
• Random Forest: 0.10
• BiLSTM: 0.10

Confusion Matrix (Weighted Ensemble):

              Predicted
              Human    AI
Actual Human  3918     16
       AI     16       3918

• Total Errors: 32 / 7,868 (0.41%)
• False Positives: 16 (0.20%)
• False Negatives: 16 (0.20%)

Training Details

Dataset:

• Name: HC3 (Human-ChatGPT Comparison Corpus)
• Total Samples: 52,452 balanced pairs
• Training: 36,716 (70%)
• Validation: 7,868 (15%)
• Test: 7,868 (15%)
• Domains: Finance, Medicine, Open QA, Reddit ELI5, Wikipedia CS/AI
• Minimum Length: 50 characters
• Balance: 50-50 (Human-AI)

Feature Engineering:

• TF-IDF: 5,000 dimensions (unigrams + bigrams)
• Linguistic: 15 handcrafted features
  • Text statistics (length, word/sentence counts)
  • Lexical diversity (Type-Token Ratio)
  • Character ratios (stopwords, punctuation, digits, capitals)
  • Structural patterns (long/short words, question/exclamation marks)

Training Configuration:

Traditional ML Models:

• Framework: scikit-learn 1.3+
• Cross-validation: 5-fold (for stacking)
• Class balance: Maintained via stratified splitting

Deep Learning Models:

• BiLSTM: 10 epochs (early stopped at 4), batch=64, Adam optimizer (LR=1e-3)
• BERT: 2 epochs, batch=16, AdamW optimizer (LR=2e-5), warmup=500 steps

Hardware:

• Training: CPU/GPU compatible
• BiLSTM training time: 3,406 seconds (4 epochs)
• BERT training time: Variable (depends on GPU)

Usage

Installation

git clone https://github.com/huzaifanasir95/Human-vs-AI-Classifier.git
cd Human-vs-AI-Classifier
pip install -r requirements.txt

Download Models

from huggingface_hub import hf_hub_download
import pickle
import torch

# Download traditional ML models
models = ['logistic_regression', 'random_forest', 'svm', 'xgboost']
for model_name in models:
    model_path = hf_hub_download(
        repo_id="huzaifanasirrr/human-vs-ai-text-classifier",
        filename=f"models/{model_name}.pkl"
    )
    with open(model_path, 'rb') as f:
        model = pickle.load(f)

# Download deep learning models
bert_path = hf_hub_download(
    repo_id="huzaifanasirrr/human-vs-ai-text-classifier",
    filename="models/bert_best.pt"
)
bilstm_path = hf_hub_download(
    repo_id="huzaifanasirrr/human-vs-ai-text-classifier",
    filename="models/bilstm_best.h5"
)

Inference (Weighted Ensemble)

from src.feature_extractor import FeatureExtractor
from src.models.ensemble import WeightedEnsemble
import numpy as np

# Initialize feature extractor
feature_extractor = FeatureExtractor(
    max_features=5000,
    ngram_range=(1, 2)
)

# Extract features from text
text = "Your text to classify here..."
features = feature_extractor.extract(text)  # Shape: (5015,)

# Load ensemble
ensemble = WeightedEnsemble(
    models=[lr_model, rf_model, svm_model, xgb_model, bert_model, bilstm_model],
    weights=[0.20, 0.10, 0.15, 0.25, 0.20, 0.10]
)

# Predict
prediction = ensemble.predict(features)
probability = ensemble.predict_proba(features)

if prediction == 0:
    print(f"Human-written (confidence: {probability[0]:.2%})")
else:
    print(f"AI-generated (confidence: {probability[1]:.2%})")

Single Model Inference

# Using XGBoost (best individual model)
xgb_prediction = xgb_model.predict(features.reshape(1, -1))
xgb_proba = xgb_model.predict_proba(features.reshape(1, -1))

print(f"Prediction: {'AI' if xgb_prediction[0] else 'Human'}")
print(f"Confidence: {xgb_proba[0][xgb_prediction[0]]:.2%}")

Key Innovations

1. Hybrid Feature Engineering: Combines vocabulary-based TF-IDF with linguistic style features
2. Multi-Paradigm Ensemble: Integrates linear models, tree ensembles, kernel methods, and neural networks
3. Optimized Weighting: Performance-based weight assignment for ensemble members
4. Balanced Performance: Equal precision and recall (99.59%) indicates no systematic bias
5. Domain Diversity: Trained across 5 different text domains for robust generalization

Feature Importance

Based on XGBoost analysis:

Insight: Vocabulary patterns dominate, but linguistic features provide crucial complementary information.

Limitations

• Dataset Specificity: Trained on ChatGPT-generated text; may not generalize to other LLMs (GPT-4, Claude, Gemini)
• Domain Dependency: Best performance on domains similar to training data
• Temporal Drift: As LLMs evolve, detection patterns may become obsolete
• Adversarial Vulnerability: Not evaluated against deliberate evasion attempts
• Language: English-only (no multilingual support)
• Computational Cost: Full ensemble requires running 6 models (mitigated by optimized weights)

Citation

If you use this model in your research, please cite:

@article{nasir2025humanaiclassifier,
  title={Human vs. AI Text Classification: A Comprehensive Study Using Machine Learning and Deep Learning Approaches},
  author={Nasir, Huzaifa},
  institution={National University of Computer and Emerging Sciences, Pakistan},
  year={2025},
  note={Hugging Face: https://huggingface.co/huzaifanasirrr/human-vs-ai-text-classifier}
}

HC3 Dataset:

@article{guo2023hc3,
  title={How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author={Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and ... and Wu, Yupeng},
  journal={arXiv preprint arXiv:2301.07597},
  year={2023}
}

Model Files

• models/*.pkl - Traditional ML models (Logistic Regression, Random Forest, SVM, XGBoost)
• models/bert_best.pt - Fine-tuned BERT model checkpoint
• models/bilstm_best.h5 - BiLSTM with Attention model
• results/*.json - Comprehensive performance metrics
• data/feature_info.json - Feature vocabulary and metadata
• visualizations/*.png - Training curves, confusion matrices, ROC curves, comparisons
• config.yaml - Configuration settings
• research_paper.tex - Full research paper (SPRINGER LNCS format)

Ethical Considerations

⚠️ Important Notice:

This model is designed for research and educational purposes. When deploying for real-world applications:

• Transparency: Inform users when text is subject to AI detection
• Fairness: Evaluate for bias against non-native speakers or specific writing styles
• Privacy: Respect user privacy and data protection regulations
• Accuracy: Do not use as definitive proof; false positives (0.2%) can occur
• Context: Use as one signal among many, not as sole evidence
• Appeals: Provide mechanisms for users to contest decisions

Detection systems should support human judgment, not replace it.

Author

Huzaifa Nasir
📧 [email protected]
🎓 National University of Computer and Emerging Sciences (FAST-NUCES), Pakistan
🔗 GitHub Repository
🆔 ORCID: 0009-0000-1482-3268

License

MIT License - See LICENSE file for details.

Acknowledgments

This project builds upon:

• HC3 Dataset: Human-ChatGPT comparison corpus (Guo et al., 2023)
• BERT: Pre-trained language model (Devlin et al., 2018)
• XGBoost: Gradient boosting framework (Chen & Guestrin, 2016)
• Scikit-learn: Machine learning library (Pedregosa et al., 2011)

Research conducted at FAST-NUCES Islamabad. Special thanks to the open-source community.

---

Status: ✅ Production-ready | Last updated: January 2025