modernbert-discussion-classifier

ModernBERT Reddit Discussion Classifier

A lightweight, high-throughput ModernBERT-based model for classifying constructive vs non-constructive conversations in online forums like Reddit. Optimized for processing vast amounts of Reddit discussion data efficiently.

Model Description

This model is a QLoRA (Quantized LoRA) fine-tuned version of answerdotai/ModernBERT-base specifically designed as a lightweight solution for large-scale Reddit discussion analysis.

• Model Type: Text Classification (Binary)
• Base Model: answerdotai/ModernBERT-base
• Training Method: QLoRA with self-training
• Task: Binary classification of conversation constructiveness
• Language: English

Model Source

• Repository: https://github.com/Niklas257/Reddit-Constructiveness-Classification.git

Intended Uses

Primary Use Case

• Classifying Reddit discussions as constructive or non-constructive
• Content moderation assistance
• Large-scale conversation quality analysis
• Social media research

Direct Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
import torch

# Load base model and tokenizer
base_model_name = "answerdotai/ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    base_model_name,
    num_labels=2
)

# Load the fine-tuned adapters
model = PeftModel.from_pretrained(model, "NiklasKoch/modernbert-discussion-classifier")
model.eval()

# Classify text (optimized for batch processing)
def classify_text(text):
    inputs = tokenizer(
        text, 
        return_tensors="pt", 
        truncation=True, 
        padding=True, 
        max_length=4096
    )
    
    # Move inputs to same device as model (important for GPU usage)
    inputs = {k: v.to(next(model.parameters()).device) for k, v in inputs.items()}
    
    with torch.no_grad():
        outputs = model(**inputs)
        predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
        
    # 0 = non-constructive, 1 = constructive
    predicted_class = torch.argmax(predictions, dim=-1).item()
    confidence = predictions[0][predicted_class].item()
    
    return {
        'class': 'constructive' if predicted_class == 1 else 'non-constructive',
        'confidence': confidence,
        'scores': {
            'non-constructive': predictions[0][0].item(),
            'constructive': predictions[0][1].item()
        }
    }

# Example usage - Reddit discussion
text = "[author0] LEGO: What do you think you're doing?!? [author1] I don't get it did he reveal bionicle reboot or smthn? [author2] Not really, he did announce something but was super vague, seems like a sort of passion project we wants to do with the community, he even said it might not even be bionicle. [author1] So is that image fan made or is it one of his passion projects [author2] Those pictures are real and on his insta, he did a stream talking about it I'm sure you can find somewhere, search up Fabre bionicle stream 2020 or something. [author1] OK thanks"
result = classify_text(text)
print(result)

Training Details

Training Data

• Source: https://archive.org/download/pushshift_reddit_200506_to_202212/
• Size: ~1.4 million Reddit threads filtered for English language and minimum 2 authors
• Labels: Binary (constructive/non-constructive conversations)
• Additional Data: YNACC and IAC datasets for initial supervised training

Training Procedure

• Training Method: Self-training
• Quantization: 4-bit QLoRA for efficiency
• LoRA Config:
  • r: 16
  • lora_alpha: 32
  • lora_dropout: 0.1
  • Target modules: Wqkv, Wo, Wi, dense
• Loss Function: Focal Loss with class weighting
• Max Sequence Length: 4096 tokens
• Batch Size: 64
• Learning Rate: 2e-6

Training Hardware

• 48 hours on 4x NVIDIA A100 40GB GPUs

Performance

Evaluation Results

YNACC:
Accuracy: 0.63
Precision: 0.63
F1-Score: 0.65

IAC:
Accuracy: 0.79
Precision: 0.85
F1-Score: 0.87

Reddit:
Accuracy: 0.57
Precision: 0.74
F1-Score: 0.67

Limitations and Bias

• Language: English only
• Bias: May reflect biases present in Reddit discussions and training data

Ethical Considerations

• Human oversight is recommended for important moderation decisions

Technical Specifications

• Model Architecture: ModernBERT + Classification Head
• Parameters: ~150M base + LoRA adapters + classification head
• Precision: 4-bit quantized base model with full-precision adapters
• Framework: PyTorch, Transformers, PEFT (any recent version - you may see harmless warnings about configuration parameters)

Model Card Authors

Niklas Koch, Georg August University of Göttingen

Model Card Contact

[email protected]