teutonic-II-110B-genesis

Teutonic-II 110B Genesis

Teutonic-II 110B Genesis is an open-weight, compute-efficient sparse Mixture-of-Experts (MoE) language model with approximately 110B total parameters and an estimated ~7.3B activated parameters per token. The Genesis checkpoint is initialized with random weights and has not yet undergone pretraining, providing a clean starting point for continued pretraining and decentralized model improvement through Teutonic on Bittensor Subnet 3.

The model is built on a modified MiMoV2-style architecture, combining sparse MoE layers with hybrid Global and Sliding Window Attention, Grouped Query Attention, asymmetric QK/V head dimensions, partial Rotary Position Embeddings, and attention-sink mechanisms. In addition to adopting MiMoV2 architectural concepts, the Teutonic team introduces a modification to the MoE implementation that enables expert sharing.

1. Introduction

Teutonic-II 110B Genesis is designed as a compute-efficient large-scale sparse language model architecture for continued pretraining and decentralized model improvement.

The model uses a custom MiMoV2-style architecture with several efficiency-oriented components:

• Sparse Mixture-of-Experts (MoE): 256 routed experts are available in each MoE layer, while only 8 routed experts are selected per token.
• Shared Expert: Each MoE layer additionally includes one shared expert.
• Expert Sharing: Teutonic introduces an architectural modification that enables expert sharing within the MoE implementation.
• Hybrid Attention: The network combines Global Attention with local Sliding Window Attention.
• Sliding Window Attention: SWA operates with a local window of 128 tokens.
• Grouped Query Attention: 48 query heads share 8 key/value heads.
• Asymmetric QK/V Dimensions: Query/Key heads use 192 dimensions while Value heads use 128 dimensions.
• Partial Rotary Position Embeddings: RoPE is applied to a fraction of the attention dimensions.
• Attention Sink Bias: Enabled for Sliding Window Attention layers.
• Sparse Activation: Only a small subset of the model's expert parameters is used for each token.
• BF16 Precision: The checkpoint configuration uses bfloat16.
• Long-Context Architecture: The configuration declares up to 1,048,576 positional embeddings.

The declared maximum position length is an architectural configuration value. It should not by itself be interpreted as evidence that this checkpoint has been trained or validated at the full 1M-token sequence length.

2. Model Summary