qwen3_5_moe_35b_a3b_base

Model Overview

Model Summary

Qwen3.5 MoE is a highly efficient multimodal language model family developed by the Qwen Team (Alibaba Cloud). It extends the standard Qwen3.5 dense architecture by introducing a Sparse Mixture-of-Experts (MoE) design and a hybrid attention mechanism.

By utilizing a routed expert system along with a shared expert, the model maintains vast total capacity while keeping the activated parameter count relatively low for each token, dramatically improving inference efficiency. The architecture features hybrid attention layers that alternate between standard grouped-query attention and recurrent linear attention. Additionally, Qwen3.5 MoE models are natively multimodal, supporting interleaved image and video inputs.

Key Features:

• Sparse Mixture-of-Experts (MoE) architecture with shared and routed experts
• Hybrid attention (Grouped-Query Attention + Linear Attention)
• Native multimodal support (Image and Video understanding)
• High efficiency: 35B total parameters with only 3B activated per token

Model Details

• Model Family: Qwen3.5 MoE
• Architecture: Sparse Mixture-of-Experts (MoE) with Hybrid Attention
• Developer: Qwen Team (Alibaba Cloud)
• Source: Qwen Team Hugging Face
• Task Type: Causal Language Modeling / Vision-Language Understanding
• Multimodal: Yes (Text, Image, Video)

Architecture Details

• Total Parameters: ~35 Billion
• Activated Parameters: ~3 Billion
• Expert Routing: Top-k routed experts + 1 shared expert
• Attention Mechanism: Hybrid (Grouped-Query Attention and GatedDeltaNet Linear Attention)
• Vision Encoder: Native multimodal vision backbone
• Max Sequence Length: 32,768 (context window size varies by specific checkpoint)

More Details

• [Qwen3.5 MoE Quickstart Notebook](coming soon..!)
• Qwen3.5 MoE API Documentation
• Qwen3.5 MoE Model Card
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Preset Table

Example Usage

import keras
import keras_hub
import numpy as np

# Load pre-trained Qwen3 model
qwen3_lm = keras_hub.models.Qwen3_5MoeCausalLM.from_preset( "qwen3_5_moe_35b_a3b_base")

# Generate text from prompt
response = qwen3_lm.generate("I want to learn about", max_length=50)
print(response)

# Batch generation with multiple prompts
prompts = ["The future of AI is", "Machine learning helps us"]
responses = qwen3_lm.generate(prompts, max_length=30)
for prompt, response in zip(prompts, responses):
    print(f"Prompt: {prompt}")
    print(f"Response: {response}\n")

Custom Sampling Strategies

# Greedy sampling (default)
qwen3_lm.compile(sampler="greedy")
response = qwen3_lm.generate("Explain quantum computing", max_length=100)

# Top-k sampling
qwen3_lm.compile(sampler="top_k")
response = qwen3_lm.generate("Write a story about", max_length=80)

# Beam search
qwen3_lm.compile(sampler=keras_hub.samplers.BeamSampler(num_beams=4))
response = qwen3_lm.generate("The best way to learn programming is", max_length=60)

Fine-tuning with LoRA

# Enable LoRA for efficient fine-tuning
qwen3_lm.backbone.enable_lora(rank=8)

# Prepare training data
training_texts = [
    "The quick brown fox jumped over the lazy dog.",
    "Machine learning is a subset of artificial intelligence.",
    "Python is a popular programming language for data science.",
    "Deep learning models require large amounts of training data.",
    "Natural language processing helps computers understand human language."
]

# Compile for training
qwen3_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4),
    metrics=["accuracy"]
)

# Fine-tune the model
qwen3_lm.fit(x=training_texts, batch_size=2, epochs=3)

# Generate with fine-tuned model
response = qwen3_lm.generate("The importance of", max_length=50)
print(response)

Custom Backbone Configuration

# Create custom Qwen3_5 MoE backbone
backbone = keras_hub.models.Qwen3_5MoeBackbone(
    vocabulary_size=151936,
    num_layers=12,  # Smaller model for faster training
    num_query_heads=16,
    num_key_value_heads=8,
    head_dim=128,
    hidden_dim=1024,
    intermediate_dim=2048,
    layer_norm_epsilon=1e-6,
    dropout=0.1,
    dtype="float32"
)

# Create tokenizer first
tokenizer = keras_hub.models.Qwen3_5MoeTokenizer.from_preset("qwen3_5_moe_35b_a3b_base")

# Create preprocessor with tokenizer
preprocessor = keras_hub.models.Qwen3_5MoeCausalLMPreprocessor(
    tokenizer=tokenizer,
    sequence_length=512
)

# Create custom causal LM
custom_qwen3 = keras_hub.models.Qwen3_5MoeCausalLM(
    backbone=backbone,
    preprocessor=preprocessor
)

# Compile and train
custom_qwen3.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4)
)

# Training data
texts = ["Hello world", "How are you", "Machine learning"]
custom_qwen3.fit(x=texts, batch_size=2, epochs=1)

Example Usage with Hugging Face URI

import keras
import keras_hub
import numpy as np

# Load pre-trained Qwen3 model
qwen3_lm = keras_hub.models.Qwen3_5MoeCausalLM.from_preset( "hf://keras/qwen3_5_moe_35b_a3b_base")

# Generate text from prompt
response = qwen3_lm.generate("I want to learn about", max_length=50)
print(response)

# Batch generation with multiple prompts
prompts = ["The future of AI is", "Machine learning helps us"]
responses = qwen3_lm.generate(prompts, max_length=30)
for prompt, response in zip(prompts, responses):
    print(f"Prompt: {prompt}")
    print(f"Response: {response}\n")

Custom Sampling Strategies

# Greedy sampling (default)
qwen3_lm.compile(sampler="greedy")
response = qwen3_lm.generate("Explain quantum computing", max_length=100)

# Top-k sampling
qwen3_lm.compile(sampler="top_k")
response = qwen3_lm.generate("Write a story about", max_length=80)

# Beam search
qwen3_lm.compile(sampler=keras_hub.samplers.BeamSampler(num_beams=4))
response = qwen3_lm.generate("The best way to learn programming is", max_length=60)

Fine-tuning with LoRA

# Enable LoRA for efficient fine-tuning
qwen3_lm.backbone.enable_lora(rank=8)

# Prepare training data
training_texts = [
    "The quick brown fox jumped over the lazy dog.",
    "Machine learning is a subset of artificial intelligence.",
    "Python is a popular programming language for data science.",
    "Deep learning models require large amounts of training data.",
    "Natural language processing helps computers understand human language."
]

# Compile for training
qwen3_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4),
    metrics=["accuracy"]
)

# Fine-tune the model
qwen3_lm.fit(x=training_texts, batch_size=2, epochs=3)

# Generate with fine-tuned model
response = qwen3_lm.generate("The importance of", max_length=50)
print(response)

Custom Backbone Configuration

# Create custom Qwen3_5 MoE backbone
backbone = keras_hub.models.Qwen3_5MoeBackbone(
    vocabulary_size=151936,
    num_layers=12,  # Smaller model for faster training
    num_query_heads=16,
    num_key_value_heads=8,
    head_dim=128,
    hidden_dim=1024,
    intermediate_dim=2048,
    layer_norm_epsilon=1e-6,
    dropout=0.1,
    dtype="float32"
)

# Create tokenizer first
tokenizer = keras_hub.models.Qwen3_5MoeTokenizer.from_preset("hf://keras/qwen3_5_moe_35b_a3b_base")

# Create preprocessor with tokenizer
preprocessor = keras_hub.models.Qwen3_5MoeCausalLMPreprocessor(
    tokenizer=tokenizer,
    sequence_length=512
)

# Create custom causal LM
custom_qwen3 = keras_hub.models.Qwen3_5MoeCausalLM(
    backbone=backbone,
    preprocessor=preprocessor
)

# Compile and train
custom_qwen3.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(1e-4)
)

# Training data
texts = ["Hello world", "How are you", "Machine learning"]
custom_qwen3.fit(x=texts, batch_size=2, epochs=1)