codebharat-100m

CodeBharat-100M

CodeBharat-100M is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).

Model Details

• Architecture: Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
• Parameters: 100,679,424 (100.68M)
• Vocabulary: 49,152 tokens (Byte-level BPE)
• Context Window: 1,024 tokens
• Training Device: NVIDIA GeForce RTX 5050 GPU
• Final Validation Loss: 1.3891

Quickstart Usage

Native PyTorch / Tokenizers Usage

import torch
from tokenizers import Tokenizer
from pathlib import Path

# Load tokenizer and model weights
tokenizer = Tokenizer.from_file("tokenizer.json")
weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")

Run Inference via CLI

python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"

Model Card Info

• Developed by: CodeBharat Team
• Model Type: Causal Language Model for Code
• License: Apache 2.0