PharmaGPT-336M

PharmaGPT-336M

A 336M parameter GPT language model trained entirely from scratch on 200K synthetic pharmaceutical documents across 6 manufacturing domains.

No pre-trained weights. No fine-tuning. Every component built from scratch: custom BPE tokenizer, full transformer architecture (RoPE + RMSNorm + SwiGLU), training loop, and evaluation pipeline.

Paper: ArXiv preprint (coming soon)
Blog: Medium article
Code: Included in this repository

---

Quick Start

import torch
from tokenizers import Tokenizer

# Download model files from this repo, then:
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)

# Reconstruct model from saved config
from model import GPT  # model.py included in this repo
model = GPT(ckpt["model_config"])
model.load_state_dict(ckpt["model"])
model.eval()

# Load tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")

# Generate pharmaceutical text
prompt = "<|deviation|>\nDuring manufacturing of Batch B-NDL-2026"
ids = torch.tensor([tok.encode(prompt).ids])
output = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50)
print(tok.decode(output[0].tolist()))

Generation with Different Domains

prompts = {
    "deviation": "<|deviation|>\nDuring routine inspection of the tablet coating line",
    "batch_record": "<|batch_record|>\nBATCH PRODUCTION RECORD\nProduct: Metformin HCl 500mg Tablets",
    "sop": "<|sop|>\nSOP-ENV-205 | Environmental Monitoring Program",
    "stability": "<|stability_study|>\nSTABILITY STUDY REPORT\nProduct: Adalimumab 40mg/0.8mL",
    "pharmacovigilance": "<|icsr|>\nA 72-year-old female patient with history of diabetes",
    "scientific": "<|scientific_paper|>\nObjective: To evaluate the impact of granulation",
}

for domain, prompt in prompts.items():
    ids = torch.tensor([tok.encode(prompt).ids])
    out = model.generate(ids, max_new_tokens=150, temperature=0.8, top_k=50)
    print(f"\n{'='*60}\n[{domain.upper()}]\n{'='*60}")
    print(tok.decode(out[0].tolist()))

---

Model Details

Architecture Highlights

This model implements the same architectural innovations found in LLaMA/Mistral, all coded from scratch:

• RoPE (Rotary Positional Embeddings) — encodes relative position through rotation of Q/K vectors
• RMSNorm — faster, simpler alternative to LayerNorm (no mean subtraction)
• SwiGLU — gated feed-forward network with Swish activation
• No bias in any linear layer — modern simplification
• Weight tying — token embedding and output projection share parameters
• Pre-norm architecture — normalize before attention/FFN, not after

---

Training Details

Training Results

Note: Low perplexity reflects the structured/templated nature of synthetic training data. Real-world pharmaceutical text would yield higher perplexity.

---

Training Data: 6 Pharmaceutical Domains

The model was trained on 200K synthetic documents (~32M tokens) generated across six pharmaceutical manufacturing domains:

1. Manufacturing Deviation Reports (~33K samples)

Equipment failures, process excursions, out-of-specification results, root cause analysis (Ishikawa, 5-Why), CAPA documentation following ICH Q10.

2. Batch Production Records (~33K samples)

Raw material dispensing, process step documentation, in-process controls, critical process parameters (CPPs), yield calculations, lot disposition decisions.

3. Standard Operating Procedures (~33K samples)

Cleaning validation, environmental monitoring, aseptic processing, water system maintenance (WFI, PW), equipment qualification — in Q&A format.

4. Stability Studies (~33K samples)

ICH Q1A(R2) study designs, accelerated (40°C/75% RH) and long-term (25°C/60% RH) conditions, assay trending, degradation products, shelf-life determination.

5. Pharmacovigilance Case Reports (~33K samples)

Individual Case Safety Reports (ICSRs), adverse event narratives, MedDRA coding, WHO-UMC causality assessment (certain/probable/possible/unlikely).

6. Scientific Writing (~33K samples)

Formulation development, Design of Experiments (DoE), analytical method development/validation, dissolution studies, results and discussion sections.

---

Special Tokens

---

Repository Contents

├── best_model.pt          # Full checkpoint (model weights + config + metadata)
├── config.json            # Architecture specification (JSON)
├── tokenizer/
│   └── tokenizer.json     # Trained BPE tokenizer (32K vocab)
├── model.py               # Complete model source code (GPT + all components)
├── tokenizer.py           # Tokenizer training/loading utilities
└── README.md              # This file

Loading Without model.py

If you want to inspect the architecture without running the custom code:

import torch, json

# Load config
with open("config.json") as f:
    config = json.load(f)
print(config)
# {'vocab_size': 32000, 'n_embd': 1024, 'n_head': 16, 'n_layer': 24, ...}

# Load checkpoint metadata
ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
print(f"Keys: {ckpt.keys()}")
print(f"Val loss: {ckpt.get('best_val_loss')}")
print(f"Iteration: {ckpt.get('iter_num')}")

---

Intended Use

Primary Use Cases

• Educational: Understanding how modern GPT architectures work end-to-end
• Research baseline: Starting point for pharmaceutical NLP research
• Template generation: Generating draft pharmaceutical document structures
• Domain adaptation: Fine-tuning on real pharmaceutical data for production use

Out-of-Scope Uses

• Clinical decision-making: This model generates plausible but NOT factually verified content
• Regulatory submissions: Generated text requires expert review and verification
• Production deployment without validation: The model was trained on synthetic data only
• General-purpose chat: This is a domain-specific completion model, not a chatbot

---

Limitations and Risks

Ethical Considerations

• Generated pharmaceutical content should NEVER be used for actual drug manufacturing without expert review
• The model may reproduce biases present in the synthetic data templates
• Not intended as a replacement for qualified pharmaceutical professionals

---

Citation

If you use PharmaGPT in your research, please cite:

@misc{chaturvedi2026pharmagpt,
  title={PharmaGPT: A Domain-Specific Language Model for Pharmaceutical Manufacturing Intelligence Trained from Scratch on Synthetic Data},
  author={Chaturvedi, Parth},
  year={2026},
  howpublished={\url{https://huggingface.co/ParthChat1802/PharmaGPT-336M}},
}

---

Technical Notes for Reproducibility

Checkpoint Format

The best_model.pt file is a PyTorch checkpoint dictionary containing:

{
    "model": OrderedDict,        # model.state_dict()
    "model_config": GPTConfig,   # dataclass with architecture params
    "config": dict,              # training configuration
    "iter_num": int,             # iteration at save time
    "best_val_loss": float,      # best validation loss achieved
}

System Requirements

• Inference: Any machine with 2GB+ RAM and PyTorch installed
• Training (reproduce): NVIDIA GPU with 8GB+ VRAM, or Apple M-series with 16GB+ unified memory
• Dependencies: torch>=2.0, tokenizers>=0.13

Reproducing Training

git clone <source-repo>
cd gpt-from-scratch
pip install -r requirements.txt

# Generate data
python -m data.generators.master_generator

# Train tokenizer + model
python -m src.train_pharma

Or use the Kaggle notebook for GPU-accelerated training (see repository).

---

License

Apache 2.0 — Use freely for any purpose (commercial, research, educational). Attribution appreciated but not legally required beyond the license notice.