Cloyster-1280m-8x160

(WIP) 8 pythia-160m models ("cortices") making up a mixture of experts adding to a total storage of 1280m (1.28b) parameters, trained on crumb/C4-K8-1M for 1 epoch (way too little, bigger dataset soon).

Predicted Expert Domains

Automatically generated classes (based on a very small subset of data) (actually the overarching class is just "ads" it's all "ads" there's so many ads in c4. Future datasets will be based on the pile.):

1. Promotional Material / Local News
2. News, promotional material, social media, survey questions, and informational material
3. Food / Cooking / Dental (?)
4. Entertainment, spirituality, personal hobbies and interests
5. Business communication and digital marketing
6. Commerce and consumerism
7. Economics, politics, sports, and current events
8. Science and Technology

Example Output

Better evaluation coming soon (tm). Uniform sampling was used (sum(logits)/8) with temperature 0.7, top_p 0.95, top_k 28. Prompt "Once upon a time," max_new_tokens=128

Once upon a time, I’ve made it a point to clean dishes in front of an illuminated grout pan for cooktop preparation, and I’ve just been enjoying that process on my back. The inner work of cleaning a grout pan is more creative and requires less than the usual amount of patience and desire. This is one of the reasons my kitchen becomes so refined and inflexible with the same grout pan.
The latest gadget in this latest trend has been Vivalant, or of course Vivalant-based. Vivalant allows you to use grout at its cooking and utensils, because it’s a

Once upon a time, as a regular walk through the area, I had no idea how very many things were going on.
I was surprised to find this same area looking much different after a few days.
Just what I expected when I went to the closest city of Marietta. It's been a few weeks now since I visited Marietta and was once again struck by its weather.
I don't know what's better, but the beautiful white gauze in this photograph is gorgeous!
I look at the weather, and the growing shapes of the sky as one moves toward the shore.
Perhaps the harbor is an air-conditioned place,

Once upon a time, we weren’t really talking about one of the broads most popular mini-games on the planet. Even the Red Dead Redemption is known for being as varied as half-teaser Monopoly, F2P and Secret Weapon. But does anyone know the half-truth of their power source that the Red Dead Redemption is built on? What’s behind the story of the new stealth film?
The game is no different. It is known for its zombies that capture some magic and take them on the battlefield.
However, I think the answer may be no. I think I’ve never seen the