Downloads · 30 days
18
10% of all-time downloads
sandeshrajx/gpt-oss-20b-reap-0.4-mxfp4
gpt-oss-20b-reap-0.4-mxfp4 is a text generation model from sandeshrajx. Use it when you need the model to write or continue text.
This repository contains a quantized version of the openai/gpt-oss-20b model.
Downloads · 30 days
18
10% of all-time downloads
All-time downloads
180
Public
Parameters
13.7B
10 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.9 GB · 100%
How the weights are stored.
U812.3B · 87%
From the Hugging Face model README
This repository contains a quantized version of the openai/gpt-oss-20b model.
This model is a pruned and quantized version of the openai/gpt-oss-20b model.
openai/gpt-oss-20breap with a compression ratio of 0.4theblackcat102/evol-codealpaca-v1The quantization process specifically targeted the "expert" layers of the model, skipping self-attention and router layers, as is standard practice for Mixture-of-Experts (MoE) models to optimize performance and reduce size.
You can load this model using the transformers library:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "sandeshrajx/gpt-oss-20b-reap-0.4-mxfp4"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# Use the model for inference
# ...
Pruning Commands Used:
python ./reap/src/reap/prune.py \
--model-name "openai/gpt-oss-20b" \
--run_observer_only true \
--samples_per_category 32
python ./reap/src/reap/prune.py \
--model-name "openai/gpt-oss-20b" \
--compression-ratio 0.4 \
--prune-method reap
The model weights have been quantized to MXFP4 format using the NVIDIA Model Optimizer. This process converts the weights to a lower precision format to reduce memory footprint and improve inference speed, while aiming to maintain model accuracy.
Note: The quantization script specifically targets experts layers for quantization, skipping self_attn and router layers as configured.
(Please specify the license of the original model and any modifications)