Downloads · 30 days
6
6% of all-time downloads
falcon-40b-awq-w4g128
yujiepan/falcon-40b-awq-w4g128
falcon-40b-awq-w4g128 is a text generation model from yujiepan. Use it when you need the model to write or continue text. It is set up for transformers.
This model applies autoawq on tiiuae/falcon-40b: AutoAWQ, 4bit, groupsize=128, zeropoint=True
All-time downloads
106
Public
Parameters
41.8B
23.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors23.3 GB · 100%
Parameter types
How the weights are stored.
I3240.8B · 97%
Try a prompt
Model card
From the Hugging Face model README
yujiepan/falcon-40b-awq-w4g128
This model applies autoawq on tiiuae/falcon-40b: AutoAWQ, 4bit, group_size=128, zero_point=True
Accuracy
| task | tiiuae/falcon-40b (fp16) | this repo |
|---|---|---|
| wikitext ppl by lm_harness | 8.410 | 8.497 |
Usage
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_name_or_path = "yujiepan/falcon-40b-awq-w4g128"
# Load model
model = AutoAWQForCausalLM.from_quantized(model_name_or_path, fuse_layers=False, trust_remote_code=False)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
prompt = "Tell me about AI"
tokens = tokenizer(
prompt,
return_tensors='pt'
).input_ids.cuda()
# Generate
generation_output = model.generate(
tokens,
do_sample=True,
temperature=0.7,
top_p=0.95,
top_k=40,
max_new_tokens=10,
)
print("Output: ", tokenizer.decode(generation_output[0]))
- Task
- Text Generation
- Library
- transformers
- Type
- falcon
- Created
- Mar 4, 2024
- Updated
- Dec 11, 2024