NeelNanda

SoLU_12L_v23_old

NeelNanda/SoLU_12L_v23_old

SoLU_12L_v23_old is a machine learning model from NeelNanda. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

A GPT-2 Medium sized SoLU model trained on 11.7B tokens of the Pile (training crashed because of dodgy data loaders at 11B, and wasn't resumed, so this is shorter than the others). 12 layers, dmodel=1536.