modernbert-seeks_guidance

modernbert-seeks_guidance

Fine-tuned ModernBERT-base classifier that detects whether a user in a multi-turn conversation is seeking legal guidance.

Part of the Legal QA collection · Try the interactive demo →

Model description

Stage 1 of a two-model encoder routing pipeline:

modernbert-seeks_guidance uses ModernBERT's 8192-token context window (trained with max_length=4096), so long WildChat threads are classified without the 512-token truncation limit of classic BERT encoders.

Results

Joint pipeline (with modernbert-primary_topic, max length 4096, full input, test): legal 87.9% · topic 78.5% · joint 78.5%.

Input-mode ablation (same checkpoint, test): full conversation 87.6% F1 vs.\ user-only 82.3% F1.

Usage

Pipeline (recommended)

Run both classifiers on a conversation stored as a list of {"role": "...", "content": "..."} messages:

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

def serialize(messages, input_mode="full"):
    lines = []
    for msg in messages:
        role = msg["role"]
        if input_mode == "user" and role != "user":
            continue
        lines.append(f"{role.capitalize()}: {msg['content']}")
    return "\n".join(lines)

def predict(model_id, text, input_mode="full"):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForSequenceClassification.from_pretrained(model_id)
    enc = tokenizer(text, truncation=True, max_length=4096, return_tensors="pt")
    with torch.no_grad():
        logits = model(**enc).logits
        pred_id = logits.argmax(dim=-1).item()
    return model.config.id2label[str(pred_id)]

conversation = [
    {"role": "user", "content": "Can my landlord evict me without notice?"},
    {"role": "assistant", "content": "Eviction rules depend on your jurisdiction..."},
    {"role": "user", "content": "I'm in California on a month-to-month lease."},
]

seeks = predict(
    "AmirMohseni/modernbert-seeks_guidance",
    serialize(conversation, input_mode="full"),
)
topic = predict(
    "AmirMohseni/modernbert-primary_topic",
    serialize(conversation, input_mode="full"),
)
print(seeks, topic)  # e.g. True HOUSING

Single model

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="AmirMohseni/modernbert-seeks_guidance",
)
text = "User: Can my landlord evict me?\nAssistant: It depends on your lease.\nUser: I'm in California."
print(classifier(text))

Intended uses & limitations

Use for: routing or filtering English multi-turn chat logs before legal QA, topic assignment, or human review.

Do not use for: legal advice, high-stakes decisions without human review, or non-English / jurisdiction-specific deployment without evaluation.

Caveats: silver labels from GPT-5.4; English only; trained on a balanced eval set — real traffic may be skewed.

Training data

Conversations come from allenai/WildChat-1M, labeled for legal guidance, topic, and uncertainty, then resampled into a balanced set:

Dataset: AmirMohseni/WildChat-Legal-Classification-V2-Balanced

• Equal seeks_legal_guidance=true/false rows with uncertainty-balanced non-legal sampling
• Splits: train 1909 · val 106 · test 107
• Input for modernbert-seeks_guidance: all turns serialized as Role: content lines (see usage example above)

Training procedure

<details> <summary>Full training log</summary> </details>

Framework versions

• Transformers 5.8.1 · PyTorch 2.10.0 · Datasets 4.8.5