Downloads · 30 days
33
1% of all-time downloads
Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens
bharat-minigpt-350m-pretrain-3b-tokens is a text generation model from Harshhvm. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Bharat MiniGPT 350M is a custom GPT-style causal language model trained from scratch by Harshvardhan Mishra using modern LLM architecture components such as RoPE, RMSNorm, SwiGLU, and SDPA Attention.
Downloads · 30 days
33
1% of all-time downloads
All-time downloads
4.4K
Public
Parameters
354M
2.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors1.4 GB · 100%
From the Hugging Face model README
Bharat MiniGPT 350M is a custom GPT-style causal language model trained from scratch by Harshvardhan Mishra using modern LLM architecture components such as RoPE, RMSNorm, SwiGLU, and SDPA Attention.
This is not a fine-tuned GPT-2 or LLaMA variant. The architecture and training pipeline were implemented manually in PyTorch and later integrated into the HuggingFace ecosystem.
A better version with more tokens and fine tune version available SOON.
Best suited for:
Knowledge completion
Educational prompts
Article continuation
English prose generation
This is a pretrained foundation model and is not instruction tuned yet.
Explore More:https://iotbyhvm.ooo/bharat-minigpt-350m-a-custom-gpt-style-llm-built-from-scratch-in-india/
| Component | Details |
|---|---|
| Layers | 24 Transformer Blocks |
| Heads | 16 Attention Heads |
| Embedding Size | 1024 |
| Context Length | 768 Tokens |
| Vocabulary Size | 50,257 |
| Position Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Feed Forward | SwiGLU |
| Attention | SDPA / Flash Attention Compatible |
| Weight Tying | Yes |
| Precision | FP16 Training |
The model was trained using a weighted mixture of:
| Dataset | Weight |
|---|---|
| HuggingFaceFW/fineweb (sample-10BT) | 40% |
| HuggingFaceFW/fineweb-edu (sample-10BT) | 30% |
| Wikimedia Wikipedia (20231101.en) | 30% |
| Setting | Value |
|---|---|
| Optimizer | AdamW |
| Learning Rate | 3e-4 |
| Min LR | 3e-5 |
| Warmup Steps | 51,200 |
| LR Scheduler | Cosine Decay |
| Gradient Accumulation | 128 |
| Mixed Precision | FP16 |
| Gradient Clipping | 1.0 |
generate() supportEvaluated using: EleutherAI LM Evaluation Harness
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_easy | 1 | none | 0 | acc | ↑ | 0.3312 | ± | 0.0097 |
| none | 0 | acc_norm | ↑ | 0.3413 | ± | 0.0097 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2650 | ± | 0.0044 |
| none | 0 | acc_norm | ↑ | 0.2636 | ± | 0.0044 | ||
| piqa | 1 | none | 0 | acc | ↑ | 0.5631 | ± | 0.0116 |
| none | 0 | acc_norm | ↑ | 0.5533 | ± | 0.0116 |
Notes:
pip install transformers torch
Load Model from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained( "Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens", trust_remote_code=True )
tokenizer = AutoTokenizer.from_pretrained( "Harshhvm/bharat-minigpt-350m-pretrain-3b-tokens", trust_remote_code=True )
import torch
prompt = "India is a land of"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad(): outputs = model.generate( **inputs,
max_new_tokens=80,
temperature=0.45,
top_p=0.82,
top_k=40,
repetition_penalty=1.35,
no_repeat_ngram_size=4,
do_sample=True,
use_cache=True,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))