mcq-scratch-lstm-transformer

MCQ Scratch Solver — BiLSTM + Transformer

A custom multiple-choice QA model built from scratch (no pretrained backbone) for the Smart MCQ Solver Challenge on Kaggle.

Architecture

Embedding → BiLSTM (bidirectional) → LayerNorm → Transformer Encoder (self-attention) → Mean+Max Pooling → Linear classifier

Each (prompt, option) pair is scored independently; the 5 option scores per question form the final logits, ranked to produce a top-3 prediction (MAP@3 metric).

Files

This repo stores weights as a plain PyTorch state dict (pytorch_model.bin) rather than safetensors, due to shared-storage tensors in the LSTM layer that are incompatible with safetensors' serialization checks. Config (architecture hyperparameters) is stored separately in config.json.

Training

• Tokenizer: bert-base-uncased (WordPiece, used for input encoding only)
• Embedding dim: 128, LSTM hidden: 128, Attention heads: 4, Transformer layers: 2
• Optimizer: AdamW, lr=1e-3, weight decay 0.01
• Loss: CrossEntropyLoss with label smoothing (0.1)
• Epochs: 5, batch size: 16
• Validation MAP@3: 1.0

Usage

import torch, json
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import torch.nn as nn

class HybridQAModel(nn.Module):
    def __init__(self, vocab_size, embed_dim, lstm_hidden, num_heads, num_layers):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, lstm_hidden, batch_first=True, bidirectional=True)
        self.layer_norm = nn.LayerNorm(lstm_hidden * 2)
        encoder_layer = nn.TransformerEncoderLayer(d_model=lstm_hidden * 2, nhead=num_heads, batch_first=True)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
        self.classifier = nn.Linear(lstm_hidden * 4, 1)

    def forward(self, input_ids):
        b, n, s = input_ids.shape
        flat = input_ids.view(b * n, s)
        mask = flat == 0
        emb = self.embedding(flat)
        out, _ = self.lstm(emb)
        out = self.layer_norm(out)
        enc = self.transformer(out, src_key_padding_mask=mask)
        pooled = torch.cat([enc.mean(1), enc.max(1)[0]], dim=1)
        return self.classifier(pooled).view(b, n)

repo_id = "pranay-aggarwal/mcq-scratch-lstm-transformer"
config_path = hf_hub_download(repo_id=repo_id, filename="config.json")
weights_path = hf_hub_download(repo_id=repo_id, filename="pytorch_model.bin")

with open(config_path) as f:
    config = json.load(f)

model = HybridQAModel(**config)
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

tokenizer = AutoTokenizer.from_pretrained(repo_id)

Live demo

mcq-scratch-solver Space

Author

Built as part of BSDA2001P: Introduction to DL and GenAI (project coursework).