XVERSE-Ent-A5.7B

<div align="center"> <h1> XVERSE-Ent-A5.7B </h1> </div>

Model Introduction

The XVERSE-Ent series is a family of domain-specific, pretrained models developed by XVERSE (Shenzhen Yuanxiang) for the Entertainment domain. These models are trained and optimized on XVERSE’s large-scale, high-quality entertainment-domain data, thereby filling the gap for open-source large language models in the entertainment domain.

XVERSE-Ent includes both Chinese and English models:

• XVERSE-Ent-A4.2B (Chinese)
• XVERSE-Ent-A5.7B (English)

Both models adopt a Mixture-of-Experts (MoE) architecture. Detailed technical information is provided below.

<div align="center"> </div>

Technical Overview

XVERSE-Ent leverages Sparse Upcycling to convert a dense model into a large-scale MoE model. Combined with a carefully designed multi-stage training strategy, the models substantially enhance domain-specific capabilities while preserving most general-purpose abilities.

Sparse Upcycling

Sparse Upcycling is a technique that transforms a pretrained dense model into a MoE model without training from scratch. This approach enables a significant increase in total model capacity while substantially reducing training cost and time.

The upcycling process consists of two main steps:

1. Fine-grained FFN Decomposition
   The Feed-Forward Network (FFN) layers of the dense model are decomposed into multiple smaller sub-networks. Each sub-network is treated as an independent expert in the MoE model. To better accommodate inference-time GPU memory constraints, expert sub-networks can be replicated as needed, enabling flexible adaptation to different hardware configurations.

2. Attention Reuse
   The attention layers of the original dense model are preserved and directly reused in the MoE model. This design choice maximizes the retention of the original model’s general-purpose capabilities and ensures training stability during architectural transformation.

An illustration of the fine-grained FFN decomposition is shown below. In this example, a single FFN is split into two sub-networks, each serving as a separate expert.

<p align="center"> <img width="80%" src="resources/fine-grained-upcycling.png"> </p>

In contrast, when the FFN is not decomposed, and the entire FFN is treated as a single expert, the approach we call coarse-grained decomposition is illustrated below.

<p align="center"> <img width="50%" src="resources/coarse-grained-upcycling.png"> </p> In our experiments and practical deployments, the fine-grained decomposition strategy consistently yields better overall performance. Moreover, it provides greater flexibility in configuring expert size, expert count, and memory usage, enabling the model architecture to adapt more effectively to diverse hardware environments and deployment scenarios.

Multi-Stage Training Strategy

XVERSE-Ent adopts a three-stage training pipeline to build models optimized for specific languages and domains:

• S0: Capability Reconstruction – recovering general-purpose capabilities after architectural transformation
• S1: Language Enhancement – enhancing the model’s modeling capability for the target language
• S2: Domain Enhancement – enhancing the model’s generation and understanding abilities in the entertainment domain

The first two stages use general-domain data, while the final stage uses a mixture of general-domain and entertainment-domain data. This multi-stage design maximizes retention of general capabilities while significantly improving domain-specific performance.

<p align="center"> <img width="100%" src="resources/multi-stage-train-en.png"> </p>

XVERSE-Ent-A4.2B (Chinese) is obtained by applying S2 domain-enhancement training on the general-domain backbone XVERSE-MoE-A4.2B.
XVERSE-Ent-A5.7B (English) is built by transforming a general dense backbone into a MoE model via fine-grained MoE upcycling, followed by the full multi-stage training pipeline. Both the Chinese and English models support an 8K context window and are trained on ~1T tokens.

Model Evaluation

To evaluate domain-specific performance, we constructed multiple evaluation datasets across different domains:

• fiction: novel and story-oriented texts
• conversation: dialogue-oriented texts
• webcc: general web text

The evaluation metric is Perplexity (PPL), where lower values indicate better performance.

The evaluation results demonstrate that XVERSE-Ent achieves excellent performance on entertainment tasks such as fiction writing and conversational generation, while maintaining strong general capabilities:

• Performance on general benchmarks (e.g., MMLU, mathematics, and code) shows minimal degradation
• Overall general capability retention exceeds 98%

These results confirm that XVERSE-Ent effectively enhances entertainment-domain performance without sacrificing general-purpose reasoning ability.

<div align="center"> </div> <div align="center"> </div>

Usage

Loading with Transformers

The XVERSE-Ent-A5.7B model can be loaded for inference using the following code:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("xverse/XVERSE-Ent-A5.7B")
model = AutoModelForCausalLM.from_pretrained("xverse/XVERSE-Ent-A5.7B", trust_remote_code=True, torch_dtype=torch.bfloat16, device_map='auto')
model = model.eval()
inputs = tokenizer('Time crept by, second after second. The sound of rain, the refrigerator’s occasional hum, the rush of water through unseen pipes in the walls—everything seemed amplified. Lin Yu realized he was counting his breaths, as if the moment he stopped, the room would be filled with a presence that did not belong to him.', return_tensors='pt').input_ids
inputs = inputs.cuda()
generated_ids = model.generate(inputs, max_new_tokens=70, eos_token_id=tokenizer.eos_token_id, repetition_penalty=1.1)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True))

Limitations and Disclaimer

Like all other Large Language Models (LLMs), XVERSE-Ent series models may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the model-generated content with caution and refrain from disseminating harmful content. Before deploying any application based on the XVERSE-Ent series models, developers should conduct safety tests and optimize the model for its specific application.

We strongly discourage the use of the XVERSE-Ent series models for producing or disseminating harmful information, or for conducting any activities that might harm the public, national security, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-Ent series models, including data security issues, public opinion risks, or risks arising from misunderstanding, misuse, dissemination, or non-compliance with the model.

Open Source License

The use of the source code in this repository must follow the Apache-2.0 open-source license, while the use of the model weights of XVERSE-Ent series models needs to adhere to the Model License Agreement.

The weights of XVERSE-Ent series models are fully open to academic research and support unrestricted commercial use. For other questions or collaborations, please contact [email protected].