modular-gpt-multicorpus

Modular GPT Multi-Corpus (17.45M Parameters)

This is a custom PyTorch-native Modular GPT model trained on a balanced multi-corpus spanning English (TinyStories), Arabic (Wikipedia/Literature), and Python Code.

Architecture & Hyperparameters

• Model Parameters: 17,451,520 (17.45 Million)
• Vocabulary Size: 628 characters (shared multilingual/code character vocab)
• Embedding Dimension ($d_{\text{model}}$): 512
• Attention: Multi-Head Attention (8 heads)
• Positional Encoding: Sinusoidal
• Feed-Forward: SwiGLU (Hidden Dimension: 2048)
• Normalization: RMSNorm
• Layers: 4 Transformer blocks

Training Optimizations

• Distributed Data Parallel (DDP)
• Balanced Corpus Sampler: Equal 1/3 sampling weight across English, Arabic, and Code to prevent language imbalance
• Automatic Mixed Precision (AMP)
• Cosine Learning Rate Schedule with Warmup (Max LR: 5e-4, Min LR: 1e-5)
• Gradient Clipping: 1.0 norm

Usage with zahidgpt Python Library

pip install git+https://github.com/Zahid-coder-17/transformers

from zahidgpt import generate

# Generate Code
print(generate("def fibonacci(", model_type="multicorpus"))

# Generate English
print(generate("Once upon a time", model_type="multicorpus"))

# Generate Arabic
print(generate("مرحبا", model_type="multicorpus"))

Quick CLI Usage

python generate.py --prompt "def train_model(" --max_tokens 200