aurelian-FAILED-70b-rope8-32K-fp16

This is a FAILED training checkpoint encountered while training the next iteration of Aurelian 70 32K (previous alpha version is here).

It follows same format and setup as the previous version (visit that page for more details), but was trained on a lot more huiman-generated creative writing data, and a lot more cleaning up and quality control run on the datasets. It also includes some DPO experiments to bias the model away from ChatGPT's creative writing style and positive bias.

However, something went wrong during training and while it writes much better prose than the alpha (IMO), it lost alpha's ability to pay attention to long contexts and generate consistent responses over many 1000s of tokens, Q&A over documents, etc. Basically it hallucinates and writes stories all the time, every prompt.

I'm uploading it anyway, if anyone has any weird use-case for it, like merging.

Meanwhile, I've rolled back to an earlier checkpoint and have addressed the issue here.