Downloads · 30 days
13
4% of all-time downloads
gpjt/jax-with-mha-bias-no-dropout-extended
jax-with-mha-bias-no-dropout-extended is a text generation model from gpjt. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is gpjt/jax-with-mha-bias-no-dropout-extended, a trained-from-scratch base model using the GPT-2-style architecture from Sebastian Raschka's book "Build a Large Language Model (from Scratch)".
Downloads · 30 days
13
4% of all-time downloads
All-time downloads
343
Public
Parameters
176M
702 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors702 MB · 99%
From the Hugging Face model README
This model is gpjt/jax-with-mha-bias-no-dropout-extended, a trained-from-scratch base model using the GPT-2-style architecture from Sebastian Raschka's book "Build a Large Language Model (from Scratch)".
The model was trained in JAX using a black-box reimplementation of Raschka's original PyTorch code, but the safetensors have been converted back to a PyTorch-compatible format for convenience -- if you run this model, it will be in PyTorch.
This model was deliberately overtrained on approximately 40 tokens per parameter (over one epoch).
Don't have high expectations for the model! It has only 163M parameters (the GPT-2 "small" size) and was trained on roughly twice the Chinchilla-optimal number of tokens (~40x the number of parameters), which means that it doesn't know many facts and is not terribly smart. If you want to do serious work, use a serious model (I like Qwen's). But if you want to build on this and see what you can do with a 2020-vintage LLM, please do feel free to play with it!
You can download and run the model for inference directly:
from transformers import pipeline
pipe = pipeline("text-generation", model="gpjt/jax-with-mha-bias-no-dropout-extended", trust_remote_code=True)
out = pipe(
"Every effort moves you",
max_new_tokens=20,
do_sample=True,
temperature=1.4,
top_k=25,
)
print(out[0]["generated_text"])
Note that because it uses custom code, you'll need to set trust_remote_code to True.
It supports AutoTokenizer, AutoModel and AutoModelForCausalLM:
>>> from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("gpjt/jax-with-mha-bias-no-dropout-extended")
>>> model = AutoModel.from_pretrained("gpjt/jax-with-mha-bias-no-dropout-extended", trust_remote_code=True)
>>> llm_model = AutoModelForCausalLM.from_pretrained("gpjt/jax-with-mha-bias-no-dropout-extended", trust_remote_code=True)
You can also fine-tune it; this notebook has an example.
Again, don't expect too much from this model! It's a 163M-parameter GPT-2 one, trained on a limited number of tokens. It's both dumb and ignorant ;-)