Downloads · 30 days
6
2% of all-time downloads
sink_attn_swi_glu_64_16_4_8
anonym5035/sink_attn_swi_glu_64_16_4_8
sink_attn_swi_glu_64_16_4_8 is a machine learning model from anonym5035. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository contains the weights, code, and evaluation results for a custom XpertGPT model. XpertGPT implements an advanced Sparse Mixture of Experts (MoE) architecture using Parallelized Multi-Scale Information T…
All-time downloads
276
Public
Repo size
6.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin211 MB · 92%
At a glance
- Model type
- xpertgpt
- Access
- Public
- Created
- Aug 27, 2026
- Updated
- Aug 27, 2026
- SHA
- 81a6e7a0
Model card
From the Hugging Face model README
XpertGPT - Sink Attention & SwiGLU
This repository contains the weights, code, and evaluation results for a custom XpertGPT model. XpertGPT implements an advanced Sparse Mixture of Experts (MoE) architecture using Parallelized Multi-Scale Information Transmission (MoEP-MSIT), enhanced with StreamingLLM-style Sink Tokens, SwiGLU activations, and sliding window attention.
Architecture Highlights
- Sink Tokens (num_sinks=1): Force-injected bypassing Expert Choice routing to anchor attention.
- SwiGLU Activations: Used in all expert MLPs.
- Sliding Window Attention: Window sizes [64, 16, 8, 4] for hierarchical local-global context.
- Pre-norm / Post-MoE Norms: Advanced normalization logic.
Zero-Shot Evaluation Results
The model's final main checkpoint has been evaluated on the strict-small zero-shot datasets.
| Benchmark | Accuracy |
|---|---|
| BLiMP | TBD |
| BLiMP Supplement | TBD |
| EWoK | TBD |
| Entity Tracking | TBD |
| Comps | TBD |
| Reading | TBD |
| Global PIQA (Parallel) | TBD |
| Global PIQA (Non-parallel) | TBD |
Note: The official BabyLM collation script will automatically populate the final metrics into ll_full_preds_and_fast_scores_causal.json upon completion.
How to Load
``python from transformers import AutoConfig, AutoModelForCausalLM
The custom modeling_xpertgpt.py code will be executed securely
model = AutoModelForCausalLM.from_pretrained( "anonym5035/sink_attn_swi_glu_64_16_4_8", revision="main", trust_remote_code=True ) ``
- Type
- xpertgpt
- Created
- Aug 27, 2026
- Updated
- Aug 27, 2026