mmbert32k-intent-classifier-lora

mmBERT-32K Intent Classifier (LoRA Adapter)

LoRA adapter for intent classification based on mmBERT-32K-YaRN (32K context, multilingual).

Model Details

• Base Model: vllm-sr/mmbert-32k-yarn
• Training Method: LoRA (Low-Rank Adaptation)
• LoRA Rank: 32
• LoRA Alpha: 64
• Trainable Parameters: 6.8M (2.2% of base model)
• Adapter Size: 27 MB

Training Data

• Primary: TIGER-Lab/MMLU-Pro (~12K academic questions)
• Supplement: vllm-sr/category-classifier-supplement (653 samples including casual "other" examples)

Categories (14 classes)

biology, business, chemistry, computer science, economics, engineering, health, history, law, math, other, philosophy, physics, psychology

Performance

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model and LoRA adapter
base_model = AutoModelForSequenceClassification.from_pretrained(
    "vllm-sr/mmbert-32k-yarn", num_labels=14
)
model = PeftModel.from_pretrained(base_model, "vllm-sr/mmbert32k-intent-classifier-lora")
tokenizer = AutoTokenizer.from_pretrained("vllm-sr/mmbert32k-intent-classifier-lora")

# Inference
inputs = tokenizer("How do neural networks learn?", return_tensors="pt")
outputs = model(**inputs)
predicted_class = outputs.logits.argmax().item()

Training Configuration

• Epochs: 5
• Batch Size: 16
• Learning Rate: 2e-4
• Weight Decay: 0.1
• Optimizer: AdamW with cosine LR scheduler