CobrIX-1.0-Coder-Flash-33B-A13B

CobrIX-1.0-Coder-Flash-MoE

This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base model (empero-ai/Qwythos-9B-v2) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit.

The model features 33B total parameters, 13B active parameters (A13B) per token, and a native 1,048,576-token context window.

Architecture

• Model Class: Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe.

• Context Window: Native support for up to 1,048,576 tokens (max_position_embeddings=1048576).

• Layer Design: Every transformer layer replaces the dense mlp with a sparse MoE block:

input_layernorm
    -> linear_attn
    -> post_attention_layernorm
    -> gate / experts[0..12] / shared_expert
    -> residual

• Routing: num_local_experts=5, num_experts_per_tok=2 (top-2 routing with softmax over the 5 experts).

• Shared Expert: An always-active shared expert based on the base MLP, gated by sigmoid(x @ w), with num_shared_experts=1.

• Precision: Router logits are computed in float32.

• Initialization: Router initialization is random. Shared-expert gates are initialized to zero.

Layer Types

layer_types = [
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention'
]

Weights

Base Model

The base model contributes:

• embed_tokens
• linear_attn.*
• Layer normalization parameters
• norm
• rotary_emb
• lm_head

Experts

The 5 expert models contribute only:

• gate_proj
• up_proven
• down_proj

for every transformer layer.

Integrity

No weight is modified, averaged, or interpolated during assembly.

The MoE model is constructed by preserving the original base-model components while routing the expert MLP projections through the sparse expert system.

Usage

The model directory contains self-contained:

• configuration_qwen35_moe.py
• modeling_qwen35_moe.py

These modules are loaded automatically through Hugging Face auto_map.

Therefore, no monkey patching is required.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CobrIX/CobrIX-1.0-Coder-Flash-33B-A13B"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    trust_remote_code=True
)

inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)

out = model.generate(
    **inputs,
    max_new_tokens=64
)

print(
    tokenizer.decode(
        out[0],
        skip_special_tokens=True
    )
)

Building & Verification

To replicate the build or verify the model structure, use the provided scripts.

Build the MoE Model

python convert.py \
    --base empero-ai/Qwythos-9B-v2 \
    --experts <e0> <e1> <e2> <e3> <e4> \
    --output Qwen35-CobrIX-MoE

Verify Architecture Integrity

python verify.py \
    --model Qwen35-CobrIX-MoE

Run Generation Tests

python test.py \
    --model Qwen35-CobrIX-MoE

Model Specifications

CobrIX Coder and CobrIX Code Models — Early Access

CobrIX Coder is the AI ​​ecosystem that runs and provides access to models such as CobrIX-1.0-Coder-Flash-MoE and CobrIX-1.0-Coder-Full-MoE.

🌐 Check out and test the AI: https://cobrix.vercel.app/coder

Visit the site to explore the AI ​​and follow the development of CobrIX Code.

Interest List

If you want to know when the AI ​​ecosystem and CobrIX Code will be available to the public, visit:

https://cobrix.vercel.app/coder

On the site, you will find a field to enter your email and join the early access waitlist.

Registered users will receive an email when the CobrIX AI ecosystem and CobrIX Code become available.

We currently have the funds to host CobrIX-1.0-Coder-Flash-33B-A13B on an RTX 6000 Ada Generation GPU (48GB VRAM), supporting between 50 and 100 concurrent users. Please help us by donating (links below); help us create AI that is usable for everyone—open-source AI forever—so we can evolve together.

🚀 Join the waitlist and get notified when CobrIX Code is released.

Donations for Infrastructure

The development of custom AI architectures such as CobrIX-1.0-Coder-Flash-MoE requires ongoing computing resources, research, experimentation, and maintenance.

If this project brings value to your work or company, you can support the development and infrastructure of the CobrIX ecosystem through the wallets below.

All donations help fund:

• AI infrastructure
• GPU computing - Nvidia RTX PRO 6000 Blackwell 96 gb vram
• Open-source model development
• New model training
• Research and experimentation
• Community support

Bitcoin (BTC)

bc1q8mu8fjak4y84qj4dlk8pu4d3zhknm92zra4r4m

Ethereum (ETH / ERC-20)

0x8D9187dEa0a77390ef668361cd5b236DE54af2BB

Solana (SOL)

GQR2jZnWuWP1c3dbuz4mC7ZnyveacBKy63q8qf9nj8bp

Thank you to all developers, researchers, enthusiasts, and partners supporting the evolution of open and accessible artificial intelligence.

CobrIX — Open AI infrastructure and custom model research.