Downloads · 30 days
18
6% of all-time downloads
Taiwan-Lawstral
lopentu/Taiwan-Lawstral
Taiwan-Lawstral is a text generation model from lopentu. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Continually pretrained from MediaTek-Research/Breeze-7B-32k-Base-v10 Base model with no instruction fine-tuning
All-time downloads
289
Public
Parameters
7.5B
29.4 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors15 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- May 28, 2024
- Updated
- May 29, 2024
- SHA
- 3a009846
Model card
From the Hugging Face model README
Model Details
- Continually pretrained from MediaTek-Research/Breeze-7B-32k-Base-v1_0
- Base model with no instruction fine-tuning
Training Details
- Hardware: 2 NVIDIA DGX H100 nodes on Taipei-1
- Software: NVIDIA NeMo and NVIDIA NeMo Framework Launcher
- Training time: 7 days
- Context length: 8192 tokens
- Steps: 81783
- Global batch size: 112
- Epochs: 3
- Tokens trained on: 75B+
- Learning rate:
- Schedule: Cosine
- Warmup steps: 107
- Constant steps: 11873
- Max LR: 1e-4
- Min LR: 1e-5
Dataset Details
- Trained mostly on Taiwan legal documents
- Tokens: 25B
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Languages
- zh
- Created
- May 28, 2024
- Updated
- May 29, 2024