Downloads · 30 days
15
7% of all-time downloads
mesolitica/malaysian-llama-3-8b-262k
malaysian-llama-3-8b-262k is a text generation model from mesolitica. Use it when you need the model to write or continue text. It is set up for transformers.
262144 context length and 207100000 RoPE Theta.
Downloads · 30 days
15
7% of all-time downloads
All-time downloads
227
Public
Parameters
8B
16.1 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
262144 context length and 207100000 RoPE Theta.
WanDB, https://wandb.ai/huseinzol05/EasyContext-262144?nw=nwuserhuseinzol05
Source code, https://github.com/mesolitica/malaya/tree/master/session/llama3#extend-1m-context-length
Special thanks to https://github.com/jzhang38/EasyContext for wrapping https://github.com/zhuzilin/ring-flash-attention for distributed training!