Downloads · 30 days
42
1% of all-time downloads
Meta-Llama-3.1-70B-Instruct-gptq-4bit
ModelCloud/Meta-Llama-3.1-70B-Instruct-gptq-4bit
Meta-Llama-3.1-70B-Instruct-gptq-4bit is a text generation model from ModelCloud. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
This model has been quantized using GPTQModel.
All-time downloads
3.7K
Public
Parameters
70.6B
119 GB on disk
Likes
4
Public
Repo makeup
Click a slice to open those files.
.safetensors39.8 GB · 100%
Parameter types
How the weights are stored.
I3268.5B · 97%
Try a prompt
Model card
From the Hugging Face model README
This model has been quantized using GPTQModel.
- bits: 4
- group_size: 128
- desc_act: true
- static_groups: false
- sym: true
- lm_head: false
- damp_percent: 0.0025
- true_sequential: true
- model_name_or_path: ""
- model_file_base_name: "model"
- quant_method: "gptq"
- checkpoint_format: "gptq"
- meta：
- quantizer: "gptqmodel:0.9.9-dev0"
Example:
from transformers import AutoTokenizer
from gptqmodel import GPTQModel
model_name = "ModelCloud/Meta-Llama-3.1-70B-Instruct-gptq-4bit"
prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = GPTQModel.from_quantized(model_name)
input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)
lm-eval benchmark:
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
|arc_challenge | 1|none | 0|acc |↑ |0.6186|± |0.0142|
| | |none | 0|acc_norm |↑ |0.6297|± |0.0141|
|arc_easy | 1|none | 0|acc |↑ |0.8628|± |0.0071|
| | |none | 0|acc_norm |↑ |0.8338|± |0.0076|
|boolq | 2|none | 0|acc |↑ |0.8761|± |0.0058|
|hellaswag | 1|none | 0|acc |↑ |0.6463|± |0.0048|
| | |none | 0|acc_norm |↑ |0.8389|± |0.0037|
|lambada_openai | 1|none | 0|acc |↑ |0.7561|± |0.0060|
| | |none | 0|perplexity|↓ |3.0311|± |0.0639|
|mmlu | 1|none | |acc |↑ |0.8100|± |0.0032|
| - humanities | 1|none | |acc |↑ |0.7981|± |0.0057|
| - formal_logic | 0|none | 0|acc |↑ |0.6349|± |0.0431|
| - high_school_european_history | 0|none | 0|acc |↑ |0.8545|± |0.0275|
| - high_school_us_history | 0|none | 0|acc |↑ |0.9412|± |0.0165|
| - high_school_world_history | 0|none | 0|acc |↑ |0.9198|± |0.0177|
| - international_law | 0|none | 0|acc |↑ |0.9008|± |0.0273|
| - jurisprudence | 0|none | 0|acc |↑ |0.8796|± |0.0315|
| - logical_fallacies | 0|none | 0|acc |↑ |0.8650|± |0.0268|
| - moral_disputes | 0|none | 0|acc |↑ |0.8266|± |0.0204|
| - moral_scenarios | 0|none | 0|acc |↑ |0.8559|± |0.0117|
| - philosophy | 0|none | 0|acc |↑ |0.8360|± |0.0210|
| - prehistory | 0|none | 0|acc |↑ |0.8827|± |0.0179|
| - professional_law | 0|none | 0|acc |↑ |0.6675|± |0.0120|
| - world_religions | 0|none | 0|acc |↑ |0.9181|± |0.0210|
| - other | 1|none | |acc |↑ |0.8304|± |0.0064|
| - business_ethics | 0|none | 0|acc |↑ |0.7900|± |0.0409|
| - clinical_knowledge | 0|none | 0|acc |↑ |0.8566|± |0.0216|
| - college_medicine | 0|none | 0|acc |↑ |0.7630|± |0.0324|
| - global_facts | 0|none | 0|acc |↑ |0.5800|± |0.0496|
| - human_aging | 0|none | 0|acc |↑ |0.8206|± |0.0257|
| - management | 0|none | 0|acc |↑ |0.8835|± |0.0318|
| - marketing | 0|none | 0|acc |↑ |0.9231|± |0.0175|
| - medical_genetics | 0|none | 0|acc |↑ |0.9400|± |0.0239|
| - miscellaneous | 0|none | 0|acc |↑ |0.9144|± |0.0100|
| - nutrition | 0|none | 0|acc |↑ |0.8660|± |0.0195|
| - professional_accounting | 0|none | 0|acc |↑ |0.6454|± |0.0285|
| - professional_medicine | 0|none | 0|acc |↑ |0.8971|± |0.0185|
| - virology | 0|none | 0|acc |↑ |0.5602|± |0.0386|
| - social sciences | 1|none | |acc |↑ |0.8736|± |0.0059|
| - econometrics | 0|none | 0|acc |↑ |0.7018|± |0.0430|
| - high_school_geography | 0|none | 0|acc |↑ |0.9242|± |0.0189|
| - high_school_government_and_politics| 0|none | 0|acc |↑ |0.9741|± |0.0115|
| - high_school_macroeconomics | 0|none | 0|acc |↑ |0.8410|± |0.0185|
| - high_school_microeconomics | 0|none | 0|acc |↑ |0.8992|± |0.0196|
| - high_school_psychology | 0|none | 0|acc |↑ |0.9229|± |0.0114|
| - human_sexuality | 0|none | 0|acc |↑ |0.8779|± |0.0287|
| - professional_psychology | 0|none | 0|acc |↑ |0.8497|± |0.0145|
| - public_relations | 0|none | 0|acc |↑ |0.7273|± |0.0427|
| - security_studies | 0|none | 0|acc |↑ |0.8163|± |0.0248|
| - sociology | 0|none | 0|acc |↑ |0.9154|± |0.0197|
| - us_foreign_policy | 0|none | 0|acc |↑ |0.9300|± |0.0256|
| - stem | 1|none | |acc |↑ |0.7456|± |0.0075|
| - abstract_algebra | 0|none | 0|acc |↑ |0.6300|± |0.0485|
| - anatomy | 0|none | 0|acc |↑ |0.7926|± |0.0350|
| - astronomy | 0|none | 0|acc |↑ |0.8947|± |0.0250|
| - college_biology | 0|none | 0|acc |↑ |0.9444|± |0.0192|
| - college_chemistry | 0|none | 0|acc |↑ |0.5800|± |0.0496|
| - college_computer_science | 0|none | 0|acc |↑ |0.6700|± |0.0473|
| - college_mathematics | 0|none | 0|acc |↑ |0.5400|± |0.0501|
| - college_physics | 0|none | 0|acc |↑ |0.6275|± |0.0481|
| - computer_security | 0|none | 0|acc |↑ |0.8200|± |0.0386|
| - conceptual_physics | 0|none | 0|acc |↑ |0.7830|± |0.0269|
| - electrical_engineering | 0|none | 0|acc |↑ |0.7862|± |0.0342|
| - elementary_mathematics | 0|none | 0|acc |↑ |0.7593|± |0.0220|
| - high_school_biology | 0|none | 0|acc |↑ |0.9194|± |0.0155|
| - high_school_chemistry | 0|none | 0|acc |↑ |0.7143|± |0.0318|
| - high_school_computer_science | 0|none | 0|acc |↑ |0.9200|± |0.0273|
| - high_school_mathematics | 0|none | 0|acc |↑ |0.5185|± |0.0305|
| - high_school_physics | 0|none | 0|acc |↑ |0.6556|± |0.0388|
| - high_school_statistics | 0|none | 0|acc |↑ |0.7361|± |0.0301|
| - machine_learning | 0|none | 0|acc |↑ |0.7054|± |0.0433|
|openbookqa | 1|none | 0|acc |↑ |0.3660|± |0.0216|
| | |none | 0|acc_norm |↑ |0.4620|± |0.0223|
|piqa | 1|none | 0|acc |↑ |0.8264|± |0.0088|
| | |none | 0|acc_norm |↑ |0.8319|± |0.0087|
|rte | 1|none | 0|acc |↑ |0.7184|± |0.0271|
|truthfulqa_mc1 | 2|none | 0|acc |↑ |0.3917|± |0.0171|
|winogrande | 1|none | 0|acc |↑ |0.7924|± |0.0114|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu | 1|none | |acc |↑ |0.8100|± |0.0032|
| - humanities | 1|none | |acc |↑ |0.7981|± |0.0057|
| - other | 1|none | |acc |↑ |0.8304|± |0.0064|
| - social sciences| 1|none | |acc |↑ |0.8736|± |0.0059|
| - stem | 1|none | |acc |↑ |0.7456|± |0.0075|
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama3.1
- Created
- Jul 23, 2024
- Updated
- Jul 27, 2024