mmbert-intent-classifier-lora

mmBERT Intent Classifier (LoRA Adapter)

A multilingual intent classification model based on mmBERT (Multilingual ModernBERT) with LoRA adapters for efficient inference.

Model Description

This model classifies text into 14 MMLU-Pro academic categories using a LoRA-enhanced mmBERT backbone. It supports 1800+ languages through mmBERT's multilingual pretraining.

Categories

• biology, business, chemistry, computer science, economics
• engineering, health, history, law, math
• other, philosophy, physics, psychology

Performance

Training Details

• Base Model: jhu-clsp/mmBERT-base
• LoRA Rank: 32
• LoRA Alpha: 64
• Trainable Parameters: 6.8M / 314M (2.2%)
• Epochs: 10
• Batch Size: 64
• Learning Rate: 2e-5
• Dataset: TIGER-Lab/MMLU-Pro (9,144 samples)

Usage

from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load base model and LoRA adapter
base_model = AutoModelForSequenceClassification.from_pretrained(
    "jhu-clsp/mmBERT-base",
    num_labels=14
)
model = PeftModel.from_pretrained(base_model, "vllm-sr/mmbert-intent-classifier-lora")
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")

# Classify
text = "What are the legal requirements for forming a corporation?"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
outputs = model(**inputs)
predicted_class = outputs.logits.argmax(-1).item()

Multilingual Support

This model supports cross-lingual transfer:

• Fine-tuned on English MMLU-Pro data
• Can classify queries in 1800+ languages
• Best performance on English, good transfer to Chinese, Spanish, French, German, etc.

Part of vLLM Semantic Router

This model is part of the vLLM Semantic Router project - a Mixture-of-Models (MoM) router that understands request intent.

Citation

@misc{mmbert-intent-classifier,
  author = {vLLM Semantic Router Team},
  title = {mmBERT Intent Classifier with LoRA},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/vllm-sr/mmbert-intent-classifier-lora}
}

License

Apache 2.0