llm-114m-trained

LLM 114M Parameters

This is a 114M parameter language model trained on 10 billion tokens.

Model Details

• Parameters: 114,150,144
• Training tokens: 10 billion
• Architecture: GPT-style transformer

Usage

import torch
from Model import LLM, params
import tiktoken

# Load tokenizer
tokenizer = tiktoken.get_encoding("gpt2")

# Load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model = LLM(params, tokenizer, device)
model.load_state_dict(torch.load("best_model_state-114m.bin"))
model.eval()

# Generate text
output = model.generate(
    "Hello, I am",
    max_gen_len=50,
    temperature=0.6,
    top_p=0.9,
    top_k=32
)
print(output)