lovelace-medium-alpha1

Avelina/lovelace-medium-alpha1

lovelace-medium-alpha1 is a text generation model from Avelina. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as bsd-3-clause.

551M parameter Transformer-XL style model trained on 100B tokens of The Pile!

Downloads · 30 days

5% of all-time downloads

All-time downloads

412

Public

Parameters

551M

1.1 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors1.1 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: bsd-3-clause
Model type: lsw_transformer
Access: Public
Created: Apr 26, 2024
Updated: May 31, 2024
SHA: a893f574

Try a prompt

Datasets

EleutherAI/pile

Model card

From the Hugging Face model README

Lovelace Medium Alpha1

551M parameter Transformer-XL style model trained on 100B tokens of The Pile!

This model was originally trained for the "Direct Prefrence Heads" paper, but will also be used as the basis for much of my future research. All code used to train and run these models is available here: https://github.com/Avelina9X/direct-preference-heads and our paper is available here: https://arxiv.org/abs/2405.20053

Model Architecture

Name	Value
Total Parameters	551M
Non-Embedding Parameters	512M
Vocab Size	50272
\(d_\text{vocab}\)	768
\(d_\text{model}\)	1536
\(n_\text{layers}\)	18
FFN Activation	SwiGLU
\(d_\text{ffn}\)	4096
Attention Type	Full
Positon Embedding	Reversed RoPE with ABF
\(n_\text{heads}\)	24
\(d_\text{key}\)	64
Trained Context	2048
Trained Memory	2048
Max Inference Context	4096

Model Collection

Model	Link
Pre-Trained Model	lovelace-medium-alpha1
Fine-Tuned Model	lovelace-medium-alpha1-sft
DPH Aligned Model	lovelace-medium-alpha1-dph

Task: Text Generation
Library: transformers
Type: lsw_transformer
License: bsd-3-clause
Languages: en
Created: Apr 26, 2024
Updated: May 31, 2024