Downloads · 30 days
0
mem-1.3b
opentensor/mem-1.3b
mem-1.3b is a machine learning model from opentensor. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as bigscience-openrail-m.
⚠️🪖 This model has not been tested in nakamoto. Usage is at your own risk AND NOT RECOMMENDED. Please see the disclaimer below ⚠️🪖
Access
Public
Updated Oct 31, 2022
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.md1.8 KB · 55%
At a glance
- License
- bigscience-openrail-m
- Access
- Public
- Created
- Oct 31, 2022
- Updated
- Oct 31, 2022
- SHA
- c2bbccf5
Model card
From the Hugging Face model README
MegaEfficentModel-PyTorch
⚠️🪖 This model has not been tested in nakamoto. Usage is at your own risk AND NOT RECOMMENDED. Please see the disclaimer below ⚠️🪖
current version : 0.1
sequence length : 512
layers : 24
attention heads : 24
dimension : 2048
learning rate : 2e-4
trained steps : 383000
GPT architectures have proven quite useful in many areas of research and industry, yet their usage is confined to high end NVIDIA GPUs. This prevents many researchers and enthusiasts from performing rapid experimentation and development on large language models.
Our mission at Opentensor Cortex is to develop the critical infrastructure and tooling to enable researchers, enthusiasts, and server operators to run large language models on consumer graphics cards.
MegaEfficentModel (MEM for short) incorporates a new type of attention from HazyResearch called FlashAttention. This new type of attention uses a Fused Kernel to decrease the time to perform the attention operation, while allowing for longer sequence lengths greater than 2048 in a memory efficient manner.
Anecdotal VRAM usage of MEM-Pytorch compared to GPT-Neo-1.3b showed an improvement from 8GB to 3GB. Autocast may offer additional memory improvements.
DISCLAIMER This model has many bugs that need to be squashed, optimizations to be performed, and lacks proper benchmarking. These will be added over the coming months, but are not currently reflected in the codebase. Please reach out to the @Cortex team in the discord for more questions.
Credits
Opentensor Foundation : provided the compute to train these models.
Lucidrains : MEM is inspired from their work on flash attention
- License
- bigscience-openrail-m
- Created
- Oct 31, 2022
- Updated
- Oct 31, 2022