crumb

cramped-94m-8btok

crumb/cramped-94m-8btok

cramped-94m-8btok is a text generation model from crumb. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.

A modified GPT-2 model with ScaledSinusoidal position embeddings, no biases, embedding layernorm, and one shared MLP layer, with 94 million non-embedding params, that beats most similarly sized and slightly larger mod…