experimental-gpt-oss-17b-mini

gpt-oss-20b-mini

Experimental MoE checkpoint obtained by expert pruning of openai/gpt-oss-20b. This is a research artefact, not a production model.

What is this?

A short summary of the technique used to produce the checkpoint.

Expert pruning is the process of removing a fraction of the experts from each Mixture-of-Experts (MoE) layer of a language model while keeping the rest of the architecture intact. The router, the attention blocks, the embeddings, the tokenizer, and the chat template are unchanged. Only the per-layer expert table is reduced, which directly shrinks the model's parameter count, memory footprint and (at inference) the amount of compute spent on the MoE block.

In this particular release the router weight matrix, the router bias and the expert tensors are sliced along the expert dimension using random-moe-pruning. The default configuration is used (--prune-level 0.25, --top-k 4, --num-examples 1000, --random-seed 42).

Provenance

The metadata fields pruned_from_num_experts and pruned_to_num_experts in config.json reflect the change for traceability.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "valendra/gpt-oss-20b-mini"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

Caveats

• This checkpoint is experimental. Quality has not been benchmarked against the base model.
• The pruning is dataset-free, so the chosen experts reflect the router's behaviour on random Gaussian probes rather than on a calibration distribution. See the project README for the mathematical justification.
• The same chat_template.jinja from the base model is reused.
• License follows the base model (Apache 2.0).