mpt-7b-gsm8k-quant-ds

RedHatAI/mpt-7b-gsm8k-quant-ds

mpt-7b-gsm8k-quant-ds is a text generation model from RedHatAI. Use it when you need the model to write or continue text. It is set up for transformers.

Paper: Sparse Finetuning for Inference Acceleration of Large Language Models Code: https://github.com/neuralmagic/deepsparse/tree/main/research/mpt

Downloads · 30 days

5% of all-time downloads

All-time downloads

494

Public

Repo size

6.9 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.data6.9 GB · 100%

At a glance

Task: Text Generation
Library: transformers
Model type: mpt
Access: Public
Created: Oct 10, 2023
Updated: Oct 18, 2023
SHA: 79d2dae4

Try a prompt

Datasets

gsm8k

Model card

From the Hugging Face model README

mpt-7b-gsm8k

Paper: Sparse Finetuning for Inference Acceleration of Large Language Models
Code: https://github.com/neuralmagic/deepsparse/tree/main/research/mpt

This model was produced from a MPT-7B base model finetuned on the GSM8k dataset for 2 epochs and quantized.

GSM8k zero-shot accuracy with lm-evaluation-harness : 27.8%

All MPT model weights are available on SparseZoo and CPU speedup for generative inference can be reproduced by following the instructions at DeepSparse

Model Links	Compression
neuralmagic/mpt-7b-gsm8k-quant	Quantization (W8A8)
neuralmagic/mpt-7b-gsm8k-pruned40-quant	Quantization (W8A8) & 40% Pruning
neuralmagic/mpt-7b-gsm8k-pruned50-quant	Quantization (W8A8) & 50% Pruning
neuralmagic/mpt-7b-gsm8k-pruned60-quant	Quantization (W8A8) & 60% Pruning
neuralmagic/mpt-7b-gsm8k-pruned70-quant	Quantization (W8A8) & 70% Pruning
neuralmagic/mpt-7b-gsm8k-pruned70-quant	Quantization (W8A8) & 75% Pruning
neuralmagic/mpt-7b-gsm8k-pruned80-quant	Quantization (W8A8) & 80% Pruning

For general questions on these models and sparsification methods, reach out to the engineering team on our community Slack.

Task: Text Generation
Library: transformers
Type: mpt
Created: Oct 10, 2023
Updated: Oct 18, 2023