experimental-gpt-oss-90b-mini

experimental-gpt-oss-90b-mini

Experimental MoE checkpoint obtained by expert pruning of openai/gpt-oss-120b. This is a research artefact, not a production model.

What is this?

Expert pruning is the process of removing a fraction of the experts from each Mixture-of-Experts (MoE) layer while keeping the rest of the architecture intact. The router, the attention blocks, the embeddings, the tokenizer, and the chat template are unchanged. Only the per-layer expert table is reduced, which directly shrinks the model's parameter count, memory footprint and (at inference) the amount of compute spent on the MoE block.

This release keeps 75% of the original experts per layer (96 out of 128) from the 120B base model. The router weight matrix, the router bias and the expert tensors are sliced along the expert dimension using random-moe-pruning. The default configuration is used (--prune-level 0.25, --top-k 4, --num-examples 1000, --random-seed 42).

Provenance

The metadata fields pruned_from_num_experts and pruned_to_num_experts in config.json reflect the change for traceability.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "valendra/experimental-gpt-oss-90b-mini"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

Caveats

• This checkpoint is experimental. Quality has not been benchmarked against the base model.
• The pruning is dataset-free, so the chosen experts reflect the router's behaviour on random Gaussian probes rather than on a calibration distribution. See the project README for the mathematical justification.
• The same chat_template.jinja from the base model is reused.
• License follows the base model (Apache 2.0).