mpc-router-mrpc

MPC-Router MRPC checkpoint

A trained GatedBertPool (cost-aware Mixture-of-BERT-Experts) checkpoint for the GLUE MRPC paraphrase classification task.

Architecture

• Router stem: first 2 layers of prajjwal1/bert-tiny
• MLP router: 128 → 256 → 3
• Expert pool (heterogeneous, fine-tuned on MRPC):
  • M-FAC/bert-tiny-finetuned-mrpc (cost = 2)
  • howey/bert-base-uncased-mrpc (cost = 7)
  • yoshitomo-matsubara/bert-large-uncased-mrpc (cost = 13)

Headline numbers (GLUE MRPC validation, 408 samples)

File contents

This checkpoint is a slim PyTorch state-dict (no optimizer state):

ckpt = torch.load('mrpc_best_model.pt', map_location='cpu', weights_only=False)
ckpt.keys()                # ['model_state_dict', 'config']

Usage

from huggingface_hub import hf_hub_download
from mpc_router import GatedBertPool
import torch

ckpt_path = hf_hub_download('yu505948/mpc-router-mrpc', 'mrpc_best_model.pt')
ckpt   = torch.load(ckpt_path, map_location='cpu', weights_only=False)
config = ckpt['config']
model  = GatedBertPool(
    router_model_name  = config['model']['router_model_name'],
    router_stem_layers = config['model']['router_stem_layers'],
    expert_model_names = config['model']['expert_model_names'],
    num_labels         = config['task']['num_labels'],
    task_type          = config['task']['task_type'],
    router_hidden_size = config['model']['router_hidden_size'],
    router_dropout     = config['model']['router_dropout'],
)
model.load_state_dict(ckpt['model_state_dict'])

See tutorials/Demo.ipynb in the source repo for an end-to-end Colab demo.

Source

• Repo: https://github.com/wocaishiniliu/MPC_Router_with_LLM_pool
• Trained with mpc-router-train --config configs/task_mrpc.yaml (batch=16, epochs=9, lr=2e-6, alpha=0.01, beta=0.03)