MaryGPT

MaryGPT Model Card

MaryGPT is a is a text generation model and a fine-tuned version of GPT-J 6B.

This model is fine-tuned exclusively on text from Mary Shelley's 1818 novel "Frankenstein; or, The Modern Prometheus".

This will be used as a base model for AI Artist Yuma Kishi👤’s activity, including art creation and exhibition curation.

<p align="right"><em>Portrait of Mary Shelley (1840, by Richard Rothwell, in the collection of the National Portrait Gallery)</em></p>

Training Data Sources

All data was obtained ethically and in compliance with the site's terms and conditions. No copyright texts are used in the training of this model without the permission.

• GPT-J 6B was trained on the Pile, a large-scale curated dataset created by EleutherAI.
• Frankenstein; or, The Modern Prometheus, Mary Shelley, 1818 (Public domain)

Training procedure

This model was trained for 402 billion tokens over 383,500 steps on TPU v3-256 pod. It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token correctly.

How to use

This model can be easily loaded using the AutoModelForCausalLM functionality:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("obake2ai/MaryGPT")
model = AutoModelForCausalLM.from_pretrained("obake2ai/MaryGPT")

Developed by

MaryGPT

• Yuma Kishi

GPT-J

• Ben Wang, developer of GPT-J
• James Bradbury for valuable assistance with debugging JAX issues.
• Stella Biderman, Eric Hallahan, Kurumuz, and Finetune for converting the model to be compatible with the transformers package.
• Leo Gao for running zero shot evaluations for the baseline models for the table.
• Laurence Golding for adding some features to the web demo.
• Aran Komatsuzaki for advice with experiment design and writing the blog posts.
• Janko Prester for creating the web demo frontend.