SBD_o1

SBD 1o

A 355M parameter GPT-2 model, built from scratch in PyTorch following Sebastian Raschka's Build a Large Language Model (From Scratch), starting from OpenAI's pretrained GPT-4 medium weights and instruction fine-tuned on the Stanford Alpaca dataset (52k instruction/response pairs).

This is a learning project, not a production model — it follows simple instructions reasonably well but has limited knowledge and reasoning ability compared to modern LLMs.

How to use it

This uses a custom architecture, not the standard transformers GPTModel, so you'll need the llms_from_scratch package to load it:

```bash pip install llms_from_scratch tiktoken torch huggingface_hub ```

```python from huggingface_hub import hf_hub_download import torch import tiktoken from llms_from_scratch.ch04 import GPTModel from llms_from_scratch.ch05 import generate, text_to_token_ids, token_ids_to_text

model_path = hf_hub_download(repo_id="YOUR-USERNAME/my-tiny-gpt2", filename="gpt2-medium355M-sft-standalone.pth")

GPT_CONFIG_355M = { "vocab_size": 50257, "context_length": 1024, "emb_dim": 1024, "n_heads": 16, "n_layers": 24, "drop_rate": 0.0, "qkv_bias": True }

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = GPTModel(GPT_CONFIG_355M) model.load_state_dict(torch.load(model_path, map_location=device, weights_only=True)) model.to(device) model.eval()

tokenizer = tiktoken.get_encoding("gpt2")

def format_input(instruction): return ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request." f"\n\n### Instruction:\n{instruction}\n\n### Response:\n" )

prompt = format_input("Name three colors") token_ids = generate( model=model, idx=text_to_token_ids(prompt, tokenizer).to(device), max_new_tokens=100, context_size=GPT_CONFIG_355M["context_length"], top_k=50, temperature=0.7, eos_id=50256, ) response = token_ids_to_text(token_ids, tokenizer)[len(prompt):].strip() print(response) ```

Training

• Base model: GPT-2 medium (355M), OpenAI pretrained weights
• Fine-tuning: supervised instruction fine-tuning, Alpaca-style prompt format
• Dataset: 52k examples from Stanford Alpaca
• Trained on: RTX 4060