Downloads · 30 days
25
5% of all-time downloads
Ichigo-llama3.1-s-base-v0.3
Menlo/Ichigo-llama3.1-s-base-v0.3
Ichigo-llama3.1-s-base-v0.3 is a audio-text-to-text model from Menlo. Use it for the audio-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
We have developed and released the family llama3s. This family is natively understanding audio and text input.
All-time downloads
470
Public
Parameters
8B
16.1 GB on disk
Likes
4
Public
Repo makeup
Click a slice to open those files.
.safetensors16.1 GB · 100%
At a glance
- Task
- Audio-Text-to-Text
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Sep 9, 2024
- Updated
- Nov 19, 2024
- SHA
- 796adc31
Model card
From the Hugging Face model README
Model Details
We have developed and released the family llama3s. This family is natively understanding audio and text input.
We continual pretrain on the expanded vocabulary homebrewltd/llama3.1-s-whispervq-init with 900M tokens from homebrewltd/raw-speech-whispervq-v1 dataset.
Model developers Homebrew Research.
Input Text and sound.
Output Text.
Model Architecture Llama-3.
Language(s): English.
Intended Use
Intended Use Cases This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.
Out-of-scope The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.
Training process
Training Metrics Image: Below is a snapshot of the training loss curve visualized.
MMLU:
| Model | MMLU Score |
|---|---|
| llama3.5-instruct-8b | 69.40 |
| ichigo-llama3.1-s-v0.3: phase 3 | 63.79 |
| ichigo-llama3.1-s-v0.3: phase 2 | 63.08 |
| ichigo-llama3.1-s-base-v0.3 | 42.11 |
| llama3.5-instruct-v0.2 | 50.27 |
Hardware
GPU Configuration: Cluster of 10x NVIDIA A6000-48GB.
GPU Usage:
- Continual Training: 30 hours.
Training Arguments
We utilize torchtune library for the latest FSDP2 training code implementation.
| Parameter | Continual Training |
|---|---|
| Epoch | 1 |
| Global batch size | 480 |
| Learning Rate | 2e-4 |
| Learning Scheduler | Cosine with warmup |
| Optimizer | AdamW fused |
| Warmup Steps | 50 |
| Weight Decay | 0.01 |
| Max Sequence Length | 512 |
Citation Information
BibTeX:
@article{Llama3-S: Sound Instruction Language Model 2024,
title={Llama3-S},
author={Homebrew Research},
year=2024,
month=August},
url={https://huggingface.co/homebrewltd/llama3.1-s-2024-08-15}
Acknowledgement
- Task
- Audio-Text-to-Text
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Created
- Sep 9, 2024
- Updated
- Nov 19, 2024