mixtral-onmt-awq-gemv

This is the OpenNMT-py converted version of Mixtral 8x7b, 4-bit AWQ quantized.

The safetensors file is 24GB hence needs 2x24GB GPUs (3090 or 4090) or 1x48GB (A6000).

To run the model on 2 GPU the config file needs to have: world_size: 2 gpu_ranks: [0, 1] parallel_mode: "tensor_parallel"

If you are lucky to have a A6000 (or V/A/H100 with more than 32GB), then use: world_size: 1 gpu_ranks: [0] #parallel_mode: "tensor_parallel"

Command line to run is:

python onmt/bin/translate.py --config /pathto/mixtral-inference-awq.yaml --src /pathto/input-vicuna.txt --output /pathto/mistral-output.txt

Where for instance, input-vicuna.txt contains:

USER:｟newline｠Show me some attractions in Boston.｟newline｠｟newline｠ASSISTANT:｟newline｠

Output will be:

Here are some attractions in Boston:｟newline｠｟newline｠1. Boston Common: This is a historic park located in the heart of Boston. It features a variety of attractions, including the Boston Common Fountain, the Boston Common Bandstand, and the Boston Common Carousel.｟newline｠｟newline｠2. Boston Public Garden: This is a historic park located in the heart of Boston. It features a variety of attractions, including the Boston Public Garden Fountain, the Boston Public Garden Bandstand, and the Boston Public Garden Carousel.｟newline｠｟newline｠3. Boston Museum of Fine Arts: This is a world-renowned art museum located in the heart of Boston. It features a variety of attractions, including the Boston Museum of Fine Arts Fountain, the Boston Museum of Fine Arts Bandstand, and the Boston Museum of Fine Arts Carousel.｟newline｠｟newline｠4. Boston Museum of Science: This is a world-renowned science museum located in the heart of Boston. It features a variety of attractions, including the Boston Museum of Science Fountain, the Boston Museum of Science Bandstand, and the Boston Museum of Science Carousel.｟newline｠｟newline｠5. Boston Museum of History: This is a world-renowned history museum located in the heart of Boston

Installation instruction:

Visit: https://github.com/OpenNMT/OpenNMT-py make sure you install flash-attn and autoawq

Enjoy

detailed MMLU scoring:

ACC-abstract_algebra: 0.3600
ACC-anatomy: 0.6444
ACC-astronomy: 0.7303
ACC-business_ethics: 0.6400
ACC-clinical_knowledge: 0.7283
ACC-college_biology: 0.8056
ACC-college_chemistry: 0.5300
ACC-college_computer_science: 0.5900
ACC-college_mathematics: 0.3700
ACC-college_medicine: 0.6936
ACC-college_physics: 0.4510
ACC-computer_security: 0.7900
ACC-conceptual_physics: 0.6468
ACC-econometrics: 0.5614
ACC-electrical_engineering: 0.6414
ACC-elementary_mathematics: 0.4630
ACC-formal_logic: 0.4524
ACC-global_facts: 0.4600
ACC-high_school_biology: 0.8000
ACC-high_school_chemistry: 0.5320
ACC-high_school_computer_science: 0.7400
ACC-high_school_european_history: 0.8121
ACC-high_school_geography: 0.8081
ACC-high_school_government_and_politics: 0.9275
ACC-high_school_macroeconomics: 0.6923
ACC-high_school_mathematics: 0.3667
ACC-high_school_microeconomics: 0.7731
ACC-high_school_physics: 0.4636
ACC-high_school_psychology: 0.8569
ACC-high_school_statistics: 0.5278
ACC-high_school_us_history: 0.8431
ACC-high_school_world_history: 0.8650
ACC-human_aging: 0.7175
ACC-human_sexuality: 0.7710
ACC-international_law: 0.8347
ACC-jurisprudence: 0.7778
ACC-logical_fallacies: 0.7791
ACC-machine_learning: 0.5357
ACC-management: 0.7767
ACC-marketing: 0.9145
ACC-medical_genetics: 0.7100
ACC-miscellaneous: 0.8404
ACC-moral_disputes: 0.7775
ACC-moral_scenarios: 0.4112
ACC-nutrition: 0.7876
ACC-philosophy: 0.7492
ACC-prehistory: 0.7963
ACC-professional_accounting: 0.5177
ACC-professional_law: 0.5111
ACC-professional_medicine: 0.7390
ACC-professional_psychology: 0.7304
ACC-public_relations: 0.6727
ACC-security_studies: 0.7061
ACC-sociology: 0.8706
ACC-us_foreign_policy: 0.9100
ACC-virology: 0.5060
ACC-world_religions: 0.8538
ACC-all: 0.6707
[2023-12-22 16:35:03,999 INFO] total run time 7156.16