fineweb-edu-complexity-classifier

Reasoning Complexity Classifier

A ModernBERT-base model fine-tuned to predict the reasoning complexity of educational text on a continuous 1–4 scale. Trained on FineWeb-Edu documents labeled by GPT-5-nano via the OpenAI Batch API (~$20 in credits).

Model Description

This is a regression model (num_labels=1, problem_type="regression") that outputs a continuous score. The score can be rounded to the nearest integer to obtain a discrete complexity level. Level 5 (Formal/Abstract reasoning) was excluded from training due to data scarcity; the model's effective range is 1.0–4.0.

Complexity Levels

Training Details

Data

• Source: FineWeb-Edu — a curated subset of Common Crawl filtered for educational content.
• Labeling: ~100,000 documents reservoir-sampled from ~6,000 records per subject category, then labeled with GPT-5-nano via the OpenAI Batch API using structured output (integer 1–5).
• Splits: 80% train / 10% validation / 10% test (stratified by integer complexity level).
• Preprocessing: Texts truncated to 8,000 characters before labeling; tokenized to 512 tokens during training with dynamic padding.
• Level 5 exclusion: Rows labeled as level 5 were excluded from the training set.

Hyperparameters

Training History

The best checkpoint (by validation MAE) was saved at epoch 2.

Evaluation Results

Evaluated on a held-out test set:

Interpretation: The model achieves a Spearman correlation of ~0.75 with gold labels, indicating strong ordinal ranking ability. The MAE of ~0.51 means predictions are on average within half a level of the true score when treated as a continuous signal.

Output Interpretation

Clip and round the raw float output to [1, 4] for a discrete level.

Architecture

Based on answerdotai/ModernBERT-base:

• Layers: 22 transformer layers (alternating full and sliding attention)
• Hidden size: 768
• Attention heads: 12
• Intermediate size: 1,152
• Max position embeddings: 8,192
• Classifier pooling: mean
• Classifier activation: GELU

Limitations

• Labels are silver-standard (GPT-5-nano), not human-annotated; label noise may affect the ~1.5% of ambiguous texts.
• Texts are truncated to 512 tokens; very long documents are judged on their first ~512 tokens only.
• Trained primarily on English educational web text; performance may degrade on other domains or languages.

Intended Use

Designed for data curation pipelines that need to filter or balance training corpora by reasoning complexity — for example, constructing curriculum-ordered datasets for language model training.