Downloads · 30 days
16
4% of all-time downloads
gpjt/jax-with-mha-bias-larger-chinchilla-1
jax-with-mha-bias-larger-chinchilla-1 is a text generation model from gpjt. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model is gpjt/jax-with-mha-bias-larger-chinchilla-1, a trained-from-scratch base model using the GPT-2-style architecture from Sebastian Raschka's book "Build a Large Language Model (from Scratch)".
Downloads · 30 days
16
4% of all-time downloads
All-time downloads
375
Public
Parameters
251M
1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1 GB · 100%
From the Hugging Face model README
This model is gpjt/jax-with-mha-bias-larger-chinchilla-1, a trained-from-scratch base model using the GPT-2-style architecture from Sebastian Raschka's book "Build a Large Language Model (from Scratch)".
The model was trained in JAX using a black-box reimplementation of Raschka's original PyTorch code, but the safetensors have been converted back to a PyTorch-compatible format for convenience -- if you run this model, it will be in PyTorch.
This model had its number of parameters scaled up so that if it was trained with the Chinchilla-optimal
number of tokens, it would approximately the same compute cost to train as a 2x overtrained
model like gpjt/jax-with-mha-bias-no-dropout-extended or
gpjt/jax-with-mha-bias-no-dropout-2-epoch.
So it's somewhere between GPT-2 small and GPT-2 medium in its number of parameters.
Don't have high expectations for the model! It has only 163M parameters (the GPT-2 "small" size) and was trained on roughly the Chinchilla-optimal number of tokens (~20x the number of parameters), which means that it doesn't know many facts and is not terribly smart. If you want to do serious work, use a serious model (I like Qwen's). But if you want to build on this and see what you can do with a 2020-vintage LLM, please do feel free to play with it!
slightly-larger in the text.You can download and run the model for inference directly:
from transformers import pipeline
pipe = pipeline("text-generation", model="gpjt/jax-with-mha-bias-larger-chinchilla-1", trust_remote_code=True)
out = pipe(
"Every effort moves you",
max_new_tokens=20,
do_sample=True,
temperature=1.4,
top_k=25,
)
print(out[0]["generated_text"])
Note that because it uses custom code, you'll need to set trust_remote_code to True.
It supports AutoTokenizer, AutoModel and AutoModelForCausalLM:
>>> from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("gpjt/jax-with-mha-bias-larger-chinchilla-1")
>>> model = AutoModel.from_pretrained("gpjt/jax-with-mha-bias-larger-chinchilla-1", trust_remote_code=True)
>>> llm_model = AutoModelForCausalLM.from_pretrained("gpjt/jax-with-mha-bias-larger-chinchilla-1", trust_remote_code=True)
You can also fine-tune it; this notebook has an example.
Again, don't expect too much from this model! It's a 236M-parameter GPT-2 one, trained on a limited number of tokens. It's both dumb and ignorant ;-)