crumbly

horizon-25m-v0

crumbly/horizon-25m-v0

horizon-25m-v0 is a text generation model from crumbly. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.

A modified GPT2 architecture with 25m non-embedding parameters, no biases, embedding-ln, scaled sin position embeddings, and a modification that makes the model's transformer run over the sequence four times before go…