Downloads · 30 days
6
24% of all-time downloads
Danube3-500M-4chan-archive-0709
adamo1139/Danube3-500M-4chan-archive-0709
Danube3-500M-4chan-archive-0709 is a machine learning model from adamo1139. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Danube3 500M model finetuned on adamo1139/Fal7acy4chanarchiveShareGPT which is essentially 250M tokens of chat data from 4chan, organized in coherent threads, capturing various boards.
All-time downloads
25
Public
Parameters
514M
1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors1 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Sep 7, 2024
- Updated
- Sep 7, 2024
- SHA
- 986c1a78
Model card
From the Hugging Face model README
Description
Danube3 500M model finetuned on adamo1139/Fal7acy_4chan_archive_ShareGPT which is essentially 250M tokens of chat data from 4chan, organized in coherent threads, capturing various boards.
ChatML prompt format, use system prompts such as "A chat on 4chan board /3/", "A chat on 4chan board /biz/" etc, as this was trained in.
This is a very small 500M model, so it's not very smart.
Issues
Dataset doesn't have correctly formatted newspaces, so quoted content doesn't format correctly.
Instead of this:
>what did you say
I didn't say nothing
It will print
>what did you say I didn't say nothing
Training details
1 epoch, 16-bit LoRA, 8192 seq length, 256 rank, 256 alpha, rslora True, batch size 8, learning rate 0.00004, embedding learning rate 0.00001, target modules ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj","embed_tokens", "lm_head"], cosing learning scheduler.
- Type
- llama
- License
- apache-2.0
- Created
- Sep 7, 2024
- Updated
- Sep 7, 2024