bert-marker-category

Havelock Marker Category Classifier

ModernBERT-based binary classifier that determines whether a rhetorical span is oral or literate, grounded in Walter Ong's Orality and Literacy (1982).

This is the coarsest level of the Havelock span classification hierarchy. Given a text span that has been identified as a rhetorical marker, the model classifies it into one of two categories: oral (characteristic of spoken, performative discourse) or literate (characteristic of written, analytic discourse).

Model Details

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "HavelockAI/bert-marker-category"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

span = "Tell me, O Muse, of that ingenious hero"
inputs = tokenizer(span, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits
    pred = torch.argmax(logits, dim=1).item()

label_map = {0: "oral", 1: "literate"}
print(f"Category: {label_map[pred]}")

Training

Data

22,367 span-level annotations from the Havelock corpus with marker types normalized against a canonical taxonomy at build time. Spans are drawn from documents sourced from Project Gutenberg, textfiles.com, Reddit, and Wikipedia talk pages. A stratified 80/10/10 train/val/test split was used with swap-based optimization. The test set contains 1,609 spans (1,162 oral, 447 literate).

Hyperparameters

Training Metrics

Best checkpoint selected at epoch 13 by missing-label-primary, F1-tiebreaker (0 missing, F1 0.850).

<details><summary>Click to show per-epoch metrics</summary> </details>

Test Set Classification Report

              precision    recall  f1-score   support

        oral      0.953     0.798     0.868      1162
    literate      0.631     0.897     0.741       447

    accuracy                          0.825      1609
   macro avg      0.792     0.847     0.804      1609
weighted avg      0.863     0.825     0.833      1609

The model achieves high precision on oral spans (0.953) and high recall on literate spans (0.897). The precision gap on literate (0.631) indicates some oral spans are misclassified as literate — expected given the class imbalance (72% oral in test).

Limitations

• Class imbalance: The test set is 72% oral / 28% literate, reflecting the corpus distribution. Literate precision suffers as a result.
• Span-level only: This model classifies pre-extracted spans. It does not detect span boundaries — pair it with a span detection model (e.g., HavelockAI/bert-token-classifier) for end-to-end use.
• 128-token context window: Longer spans are truncated.
• Domain: Trained on historical/literary and web text. Performance on other domains is untested.

Theoretical Background

The oral–literate distinction follows Ong's framework. Oral markers include features like direct address, formulaic phrasing, parataxis, repetition, and sound patterning. Literate markers include features like subordination, abstraction, hedging, passive constructions, and textual apparatus (citations, cross-references). This binary classifier serves as the top level of a three-tier taxonomy: category → type → subtype.

Related Models

Citation

@misc{havelock2026category,
  title={Havelock Marker Category Classifier},
  author={Havelock AI},
  year={2026},
  url={https://huggingface.co/HavelockAI/bert-marker-category}
}

References

• Ong, Walter J. Orality and Literacy: The Technologizing of the Word. Routledge, 1982.
• Lee, C. et al. "Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models." ICLR 2020.
• Warner, A. et al. "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference." 2024.

---

Trained: February 2026