Downloads · 30 days
13
13% of all-time downloads
Moah-MoE-1.58b-1B
Ostixe360/Moah-MoE-1.58b-1B
Moah-MoE-1.58b-1B is a text generation model from Ostixe360. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This Model is a first test to combine Jamba architecture with 1.58 bits linear layers and mixture of attention head.
All-time downloads
104
Public
Parameters
1.1B
4.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Apr 12, 2024
- Updated
- Apr 12, 2024
- SHA
- a783b086
Try a prompt
Datasets
Model card
From the Hugging Face model README
Model Card for Model ID
Model Details
Model Description
<!-- Provide a longer summary of what this model is. -->This Model is a first test to combine Jamba architecture with 1.58 bits linear layers and mixture of attention head.
The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.
- Model type: Mixture of attention head and mixture of expert 1.58bit linear layers
- License: Apache licence 2.0
Model Sources [optional]
- Repository: https://github.com/ostix360/optimized-LLM
How to Get Started with the Model
If you want to test this model please look at this repo at this commit
Training Details
Training Data
We use the first 100k data of Locutusque/UltraTextbooks to train this model
Training Procedure
We use adam-8 bits with default betas and epsilon values
Preprocessing [optional]
The data fit the model max length i.e. 512 tokens
Training Hyperparameters
Please look at this file to see the hyperparameters
Technical Specifications [optional]
Compute Infrastructure
Hardware
- one 4070 ti GPU
Software
- pytorch, transformers etc
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Languages
- en
- Created
- Apr 12, 2024
- Updated
- Apr 12, 2024