Banking-Custom-SLM

Banking Custom SLM

Overview

This is a custom-built, 6-layer Transformer-based Small Language Model (SLM), trained specifically for the banking domain. This model was developed as part of an end-to-end Retrieval-Augmented Generation (RAG) pipeline designed to automate the summarization and interpretation of Reserve Bank of India (RBI) regulatory circulars.

Unlike general-purpose LLMs, this model focuses on domain-specific vocabulary and factual grounding through a custom TF-IDF retrieval engine.

Model Architecture

The model implements a custom BankingTransformer class from scratch using PyTorch:

• Architecture: 6-layer Transformer Encoder
• Embedding Dimension: 384
• Attention Heads: 8
• Vocabulary Size: 30,000
• Training Framework: PyTorch

How to Load the Model

Since this is a custom architecture, you can load it using the model_architecture.py file included in this repository:

import torch
from model_architecture import BankingTransformer

# Configuration
vocab_size = 30000
embed_dim = 384
num_layers = 6
num_heads = 8

# Initialize model
model = BankingTransformer(vocab_size, embed_dim, num_layers, num_heads)

# Load weights
model.load_state_dict(torch.load("banking_model_epoch_3.pth", map_location='cpu'))
model.eval()

print("Model loaded successfully!")