Downloads · 30 days
15
14% of all-time downloads
Sherwinroger002/gpt-124m-fineweb-edu-10m-tokens
gpt-124m-fineweb-edu-10m-tokens is a text generation model from Sherwinroger002. Use it when you need the model to write or continue text. It is set up for pytorch. The card lists the license as mit.
A custom GPT-style autoregressive transformer language model implemented from scratch in PyTorch.
Downloads · 30 days
15
14% of all-time downloads
All-time downloads
108
Public
Parameters
163M
653 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors653 MB · 99%
From the Hugging Face model README
A custom GPT-style autoregressive transformer language model implemented from scratch in PyTorch.
This project contains:
The model was trained on a subset of FineWeb-Edu using a GPT-2 tokenizer.
Model configuration:
{
"vocab_size": 50257,
"context_length": 256,
"emb_dim": 768,
"n_heads": 12,
"n_layers": 12,
"drop_rate": 0.1,
"qkv_bias": False
}
Approximate parameter count:
Architecture components:
Training setup:
Hardware used:
Dataset:
Tokenizer:
Install dependencies:
pip install torch transformers safetensors
import json
import torch
from safetensors.torch import load_file
from transformers import AutoTokenizer
from model import GPTModel
# load config
with open("config.json") as f:
cfg = json.load(f)
# create model
model = GPTModel(cfg)
# load weights
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()
# tokenizer
tokenizer = AutoTokenizer.from_pretrained(".")
from model import generate_and_print_sample
print(generate_and_print_sample(model, tokenizer, "cuda", "The world is big"))
Example generations from early-stage training:
"The world is big and is a whole for children. The best part of which has been made in the lives, and the state is an ideal man, but also the same one is in the world. “The only one has been created by people,” said the new study of the journal In the past, it is the best “s not people who have no longer to have not been seen in a few years.” “The only one who have one, the most famous in the country has no one at least three years. “If you’re very low, it is not a big or less than one’s risk.” The study is a study of people who have already been reported that the risk of people who are diagnosed with HIV-S"
The model currently demonstrates:
model.py # GPT architecture
model.safetensors # trained weights
config.json # model configuration
tokenizer files # GPT2 tokenizer assets
README.md # project documentation
This is a custom PyTorch implementation and is not directly compatible with Hugging Face AutoModelForCausalLM.
Users should load the model using the provided model.py architecture.
MIT License.