mmbert-feedback-detector-lora

mmBERT Feedback Detector (LoRA Adapter)

A lightweight LoRA adapter for multilingual 4-class feedback classification, fine-tuned on mmBERT-base using AMD MI300X GPU.

Model Description

This is a LoRA adapter (27 MB) that can be loaded on top of mmBERT-base for efficient inference and further fine-tuning.

Labels

Performance

LoRA Configuration

Training

• Dataset: 51,694 examples from feedback-detector-dataset
• Epochs: 5
• Batch Size: 64
• Learning Rate: 2e-5

Hardware

Usage

With PEFT

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    "jhu-clsp/mmBERT-base",
    num_labels=4
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "vllm-sr/mmbert-feedback-detector-lora"
)
tokenizer = AutoTokenizer.from_pretrained("vllm-sr/mmbert-feedback-detector-lora")

# Classify
labels = ["SAT", "NEED_CLARIFICATION", "WRONG_ANSWER", "WANT_DIFFERENT"]
inputs = tokenizer("Thank you, that was helpful!", return_tensors="pt")
outputs = model(**inputs)
pred = outputs.logits.argmax(-1).item()
print(f"Label: {labels[pred]}")

Merge LoRA Weights

from peft import PeftModel

# Merge for faster inference
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged_model")

Advantages of LoRA

• Small Size: 27 MB vs 1.2 GB (full model)
• Fast Training: Only 2.15% of parameters trained
• Easy Deployment: Can swap adapters without reloading base model
• Further Fine-tuning: Can continue training on domain-specific data

Related Models

• Merged Model: vllm-sr/mmbert-feedback-detector (full weights, ready to use)
• Base Model: jhu-clsp/mmBERT-base

Citation

@model{mmbert_feedback_detector_lora,
  title={mmBERT Feedback Detector LoRA},
  author={LLM Semantic Router Team},
  year={2025},
  url={https://huggingface.co/vllm-sr/mmbert-feedback-detector-lora}
}

License

Apache 2.0