poetlm-29m-pretrain

poetlm-29m-pretrain

28,845,568-parameter decoder trained from scratch on public-domain English poetry. Stage: pretrain. Val loss 3.6293.

Llama-style: RMSNorm, RoPE, GQA (8Q/2KV), QK-norm, SwiGLU, tied embeddings. d_model=512, n_layers=8, vocab=12288. Not a transformers architecture.

Data provenance

biglam/gutenberg-poetry-corpus (CC0-1.0), filtered twice: prose removed (the corpus is lines from books filed under poetry, so prefaces and boilerplate ride along), then rights-checked against the Project Gutenberg catalogue — only authors who died 1955 or earlier, which clears life+70 jurisdictions rather than relying on "public domain in the USA". Anthologies and translations were used for pretraining but carry no author label: the editor is not the voice. The catalogue is a build-time filter; none of it enters the weights.

Limitations

Surface style only — diction, line length, rhyme density, archaic forms. No factual grounding, no structure past a few lines. Pre-1955 verse carries the attitudes of its period. Small models on small corpora can memorise; check outputs before publishing.

Licence

Weights under apache-2.0. The corpus is CC0 and the poems are public domain, so the data imposes no downstream terms — but PD is jurisdictional, and the 1955 cutoff targets life+70 countries. Confirm against yours.