Downloads · 30 days
15
10% of all-time downloads
AmanPriyanshu/gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts
gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts is a text generation model from AmanPriyanshu. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
Downloads · 30 days
15
10% of all-time downloads
All-time downloads
144
Public
Parameters
14.3B
28.7 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors28.7 GB · 100%
From the Hugging Face model README
Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
<div align="center">This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 21 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for all tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~14.3B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 21 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | All |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This general-purpose model maintains broad capabilities across all domains while significantly reducing computational requirements. It preserves the essential routing patterns discovered across our comprehensive analysis of diverse evaluation benchmarks including GPQA, MMLU, SORRY-Bench, and Tulu3 datasets.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for all tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 21 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-14.3b-specialized-all-pruned-moe-only-21-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What is artificial intelligence and how does it work?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on all tasks:
Question: What is artificial intelligence and how does it work?
Response:
<|channel|>analysis<|message|>Need concise explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Artificial Intelligence (AI)** is a field of computer science that builds systems capable of performing tasks that normally require human intelligence. These tasks include:
| Task | Typical Human Skill | AI Approach |
|------|---------------------|-------------|
| Recognizing faces | Visual pattern recognition | Deep‑learning neural networks (CNNs) |
| Translating language | Understanding grammar & context | Statistical models, neural language models |
| Driving a car | Perception, decision‑making, motor control | Sensor fusion + reinforcement learning |
| Diagnosing disease | Integrating symptoms, tests, experience | Probabilistic inference, machine learning |
---
### 1. **Core Idea**
AI systems learn from data or rules, then generalize to new situations. The learning can be:
- **Rule‑based** (expert systems, logic programming)
- **Statistical** (probability models, Bayesian inference)
- **Learning‑based** (machine learning, deep learning)
---
### 2. **How It Works – A High‑Level Flow**
1. **Data Collection**
Gather examples of the problem domain (images, text, sensor readings, etc.).
2. **Representation**
Convert raw data into a format the algorithm can process (feature vectors, embeddings, etc.).
3. **Learning / Training**
- **Supervised**: model learns mapping from input → labeled output.
- **Unsupervised**: model discovers structure (clustering, auto‑encoding).
- **Re‑inforcement**: model learns actions that maximize a reward signal.
4. **Model Architecture**
- **Neural Networks**: layers of weighted connections (deep learning).
- **Decision Trees / Random Forests**: hierarchical splits.
- **Probabilistic Graphs**: Bayesian networks.
- **Symbolic Systems**: logic rules, ontologies.
5. **Optimization**
Adjust model parameters (weights, probabilities) to minimize a loss function (error, likelihood, etc.) using algorithms like gradient descent, back‑propagation, or evolutionary search.
6. **Inference / Deployment**
Apply the trained model to new data to produce predictions, classifications, or actions.
7. **Feedback & Adaptation**
Continuously update the model with new data or user feedback (online learning, fine‑tuning).
---
### 3. **Key Technologies**
| Technology | What It
Question: Explain Einstein's theory of relativity in simple terms.
Response:
<|channel|>analysis<|message|>Need simple explanation.<|end|><|start|>assistant<|channel|>final<|message|>**Einstein’s Theory of Relativity – in plain words**
---
### 1. Two main ideas
| **Special Relativity** | **General Relativity** |
|------------------------|------------------------|
| Deals with *moving objects* (like a fast train or a spaceship). | Deals with *gravity* and how it bends space. |
---
### 2. Special Relativity – “Time and space are linked”
1. **Speed of light is a limit**
Light always travels at the same speed (≈ 300 000 km/s). Nothing can go faster.
2. **Moving clocks run slower**
If you’re moving very fast, the clocks you carry tick a little slower than clocks that stay still.
*Why?* Because the distance light has to travel inside the moving clock is a bit longer.
3. **Length gets shorter**
A fast‑moving object appears a little shorter in the direction of motion.
*Why?* Light has to travel a longer path to reach the far end of the object.
4. **“Same” events look different**
Two events that happen at the same place for one observer may happen at different places for another moving observer.
*Why?* Because space and time mix together – they’re not separate.
**Take‑away:** In a world where the speed of light is fixed, moving changes how we measure time and distance. Nothing can outrun light, and the farther you go, the more “slower” your clock looks.
---
### 3. General Relativity – “Gravity is curved space”
1. **Mass bends space**
Think of space as a stretchy rubber sheet.
- A heavy ball (planet, star) pressed onto the sheet makes a dip.
- The dip is the *gravitational field*.
2. **Objects follow the dip**
A ball rolling on the sheet will slide toward the dip.
In the universe, planets, moons, and even light follow the curved paths created by mass.
3. **Light bends too**
Light rays don’t travel in straight lines near massive objects; they curve around them.
This explains why we see *gravitational lensing* (e.g., a galaxy’s light bent by a foreground galaxy).
4. **No “force” in the old sense**
Einstein replaced the idea
Question: How does democracy function in modern governments?
Response:
<|channel|>analysis<|message|>Need explain democratic functions, institutions, processes.<|end|><|start|>assistant<|channel|>final<|message|>**Democracy in the modern world is not a single, one‑style system but a set of principles and practices that together create a government in which power ultimately rests with the people.**
Below is a practical “road‑map” of how those principles are translated into everyday governance in contemporary states.
---
## 1. Core Democratic Principles
| Principle | What it means | How it shows up in practice |
|-----------|----------------|------------------------------|
| **Popular sovereignty** | The people are the ultimate source of authority. | Elections, referenda, public consultations. |
| **Rule of law** | Everyone, including leaders, is subject to the law. | Independent judiciary, constitutional supremacy. |
| **Separation of powers** | Legislative, executive, and judicial branches are distinct and check each other. | Parliament, cabinet, courts. |
| **Political pluralism** | Multiple parties and viewpoints compete for power. | Multi‑party elections, freedom of association. |
| **Civil‑rights protection** | Freedom of speech, assembly, religion, etc. | Constitutional guarantees, human‑rights courts. |
| **Transparency & accountability** | Citizens can see how decisions are made and hold leaders responsible. | Freedom of information laws, audits, whistle‑blower protections. |
---
## 2. Institutional Architecture
| Institution | Role | Typical Democratic Features |
|--------------|------|---------------------------|
| **Legislature** | Makes laws, represents citizens | Majority‑based elections, proportional representation, committee hearings, public debates. |
| **Executive** | Implements laws, runs day‑to‑day governance | Head of state elected or appointed by legislature, cabinet accountable to parliament, term limits. |
| **Judiciary** | Interprets laws, protects rights | Independent judges, appellate system, judicial review of executive actions. |
| **Electoral Commission** | Oversees elections | Non‑partisan, transparent vote‑counting, dispute resolution. |
| **Civil‑Service** | Executes policy | Merit‑based hiring, career progression, oversight by elected bodies. |
| **Media & Civil Society** | Watchdog & voice | Press freedom, NGOs, public forums, citizen‑initiated petitions. |
---
## 3. The Electoral Process
1. **Candidate nomination**
* Parties register candidates; independent candidates must meet thresholds (signatures, fees).
2. **Campaigning**
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}