Downloads · 30 days
83
2% of all-time downloads
LWM-Text-Chat-128K-iMat.GGUF
dranger003/LWM-Text-Chat-128K-iMat.GGUF
LWM-Text-Chat-128K-iMat.GGUF is a text generation model from dranger003. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as llama2.
GGUF importance matrix (imatrix) quants for https://huggingface.co/LargeWorldModel/LWM-Text-Chat-128K The importance matrix was trained for 100K tokens (200 batches of 512 tokens) using wiki.train.raw.
All-time downloads
3.7K
Public
Repo size
37.7 GB
Likes
8
Public
Repo makeup
Click a slice to open those files.
.gguf37.7 GB · 100%
At a glance
- Task
- Text Generation
- Library
- gguf
- License
- llama2
- Access
- Public
- Created
- Feb 14, 2024
- Updated
- Mar 7, 2024
- SHA
- f021a94b
Try a prompt
Model card
From the Hugging Face model README
GGUF importance matrix (imatrix) quants for https://huggingface.co/LargeWorldModel/LWM-Text-Chat-128K
The importance matrix was trained for 100K tokens (200 batches of 512 tokens) using wiki.train.raw.
- The imatrix Q4-K quant fits with 32K context on 24GB and gives me ~100 t/s inference on a 3090.
- With IQ3_XXS it seems to fit ~37K context on 24GB (and it is even faster than Q4-K).
- With either quant on a 3090 it seems to decode context at well over 2000 t/s.
- Using Q8 K-cache (instead of F16) you can fit up to 43-44K context but inference speed goes down a little bit.
- Also for some reason I need to use 1.0 penalty to avoid the response being cut-off.
| Layers | Context | Template |
|---|---|---|
| <pre>32</pre> | <pre>131072</pre> | <pre>You are a helpful assistant.<br>USER:<br>{context}<br>{question}<br>Don't give information outside the document or repeat your findings. Keep your response short and direct.<br>ASSISTANT:<br>{response}</pre> |
- Task
- Text Generation
- Library
- gguf
- License
- llama2
- Created
- Feb 14, 2024
- Updated
- Mar 7, 2024