crumb

test-00-switchllama-i3b-f10b-e4-init

crumb/test-00-switchllama-i3b-f10b-e4-init

test-00-switchllama-i3b-f10b-e4-init is a text generation model from crumb. Use it when you need the model to write or continue text. It is set up for transformers.

this is the initial model warmstarted from openllama 3b v2 with the expert layers copied from models trained with lora on only the mlp layers, if you can find the problem with the modeling code i will send you a pictu…