Downloads · 30 days
54
0% of all-time downloads
adamo1139/LWM-7B-1M-1000000ctx-AEZAKMI-3_1-1702
LWM-7B-1M-1000000ctx-AEZAKMI-3_1-1702 is a text generation model from adamo1139. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
LargeWorldModel 7B 1000000 ctx finetuned on AEZAKMI v3.1 dataset for epochs at maxseqlen of 4000 using QLoRA with lorar 32 and cosine lr decaying from 0.00015. I will be uploading exl2 quants and base model in safeten…
Downloads · 30 days
54
0% of all-time downloads
All-time downloads
11.3K
Public
Parameters
6.7B
13.5 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors13.5 GB · 100%
From the Hugging Face model README
LargeWorldModel 7B 1000000 ctx finetuned on AEZAKMI v3.1 dataset for epochs at max_seq_len of 4000 using QLoRA with lora_r 32 and cosine lr decaying from 0.00015. I will be uploading exl2 quants and base model in safetensors format soon.
Fine-tuned with unsloth, FA2 on local RTX 3090 Ti. Training took around 6 hours. I think most of the long ctx capabilities remain.