Downloads · 30 days
96
6% of all-time downloads
RichardErkhov/adamo1139_-_LWM-7B-1M-1000000ctx-AEZAKMI-3_1-1702-gguf
adamo1139_-_LWM-7B-1M-1000000ctx-AEZAKMI-3_1-1702-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
96
6% of all-time downloads
All-time downloads
1.7K
Public
Repo size
76.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf88.7 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
LWM-7B-1M-1000000ctx-AEZAKMI-3_1-1702 - GGUF
LargeWorldModel 7B 1000000 ctx finetuned on AEZAKMI v3.1 dataset for epochs at max_seq_len of 4000 using QLoRA with lora_r 32 and cosine lr decaying from 0.00015. I will be uploading exl2 quants and base model in safetensors format soon.
Fine-tuned with unsloth, FA2 on local RTX 3090 Ti. Training took around 6 hours. I think most of the long ctx capabilities remain.