mpt-7b-gsm8k-pruned80-quant-ds

mpt-7b-gsm8k-pruned80-quant

Paper: Sparse Finetuning for Inference Acceleration of Large Language Models
Code: https://github.com/neuralmagic/deepsparse/tree/main/research/mpt

This model was produced from a MPT-7B base model finetuned on the GSM8k dataset with pruning applied using SparseGPT and retrain for 4 epochs with L2 distillation. Then it was exported for optimized inference with DeepSparse.

GSM8k zero-shot accuracy with lm-evaluation-harness : 21.08% (FP32 baseline is 28.2%)

Usage

from deepsparse import TextGeneration
model_path = "hf:neuralmagic/mpt-7b-gsm8k-pruned80-quant" # or use a sparsezoo stub (zoo:mpt-7b-gsm8k_mpt_pretrain-pruned80_quantized)
model = TextGeneration(model=model_path)
model("There are twice as many boys as girls at Dr. Wertz's school. If there are 60 girls and 5 students to every teacher, how many teachers are there?", max_new_tokens=50)

All MPT model weights are available on SparseZoo and CPU speedup for generative inference can be reproduced by following the instructions at DeepSparse

For general questions on these models and sparsification methods, reach out to the engineering team on our community Slack.