gaperon-mmbert-english

gaperon-mmbert-english

A multi-head text-quality classifier for English, fine-tuned from jhu-clsp/mmBERT-base (a ModernBERT-architecture multilingual encoder) as part of the Gaperon data-curation pipeline. Given a passage of text, it predicts a low / medium / high rating along six quality dimensions, intended for scoring and filtering web/corpus text before it's used for further model training.

This is the best checkpoint (selected on validation mean_macro_f1) from training run mmbert_english_v1, saved at epoch 3.

Model description

• Backbone: jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, hidden size 768, 8192 max sequence length, mean pooling over token representations).
• Heads: six independent linear heads (Linear(768 → 3)), one per label dimension, applied to the mean-pooled backbone output. Each head is a 3-way classifier over {low: 0, medium: 1, high: 2}.
• Label dimensions: clarity, coherence, depth, grammar, usefulness, overall.

The backbone weights are in model.safetensors / config.json (standard HF format, loadable with AutoModel). The six classification heads are not part of the HF backbone class and are stored separately in heads.pt (see usage below).

Intended use

Rating the quality of English text (e.g. web-scraped documents) along the six dimensions above, for filtering/weighting examples in a training corpus. Not intended as a general-purpose sentence encoder or for tasks unrelated to the quality-rating schema it was trained on.

Training data

Fine-tuned on an in-house tagged corpus of English text (data_gaperon/tagged_v2/english), labeled with low/medium/high ratings for each of the six dimensions.

Hyperparameters

Evaluation results (best checkpoint, epoch 3)

Macro-F1 / accuracy per dimension:

Usage

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

repo_id = "<your-username>/gaperon-mmbert-english"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
backbone = AutoModel.from_pretrained(repo_id)

heads_path = hf_hub_download(repo_id, filename="heads.pt")
ckpt = torch.load(heads_path, map_location="cpu")
label_dims = ckpt["label_dims"]              # ["clarity", "coherence", "depth", "grammar", "usefulness", "overall"]
label2id = ckpt["label2id"]                  # per-dimension {"low": 0, "medium": 1, "high": 2}
heads_state = ckpt["heads_state_dict"]        # one Linear(768, 3) per dimension

heads = {dim: torch.nn.Linear(768, 3) for dim in label_dims}
for dim, head in heads.items():
    head.weight.data = heads_state[f"{dim}.weight"]
    head.bias.data = heads_state[f"{dim}.bias"]
    head.eval()

backbone.eval()
text = "Your input passage here."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    hidden = backbone(**inputs).last_hidden_state          # (1, seq_len, 768)
    mask = inputs["attention_mask"].unsqueeze(-1)
    pooled = (hidden * mask).sum(1) / mask.sum(1)           # mean pooling

id2label = {dim: {v: k for k, v in mapping.items()} for dim, mapping in label2id.items()}
for dim, head in heads.items():
    pred_id = head(pooled).argmax(-1).item()
    print(dim, "->", id2label[dim][pred_id])

Limitations

• Trained and evaluated only on English text matching the tagging schema of the source corpus; quality ratings may not transfer to other domains or genres.
• The 3-class (low/medium/high) granularity is coarse; F1 scores in the 0.55-0.67 range indicate the heads are useful as a filtering signal but should not be treated as ground truth.
• heads.pt requires custom loading code (shown above) — it is not loadable via AutoModelForSequenceClassification.