Ultra-FineWeb-classifier

Ultra-FineWeb-Classifier

<div align="center"> <img src="assets/ultra-fineweb-logo.png" width="600"/> </div> <p align="center"> <a href="https://arxiv.org/abs/2505.05427">📜 Technical Report</a> | <a href="https://huggingface.co/datasets/openbmb/Ultra-FineWeb">🤗 Ultra-FineWeb</a> | <a href="https://huggingface.co/collections/openbmb/ultradata">📦 UltraData Collection</a> | <a href="https://ultradata.openbmb.cn/">🌐 UltraData</a> </p> <p align="center"> English | <a href="https://huggingface.co/openbmb/Ultra-FineWeb-classifier/blob/main/README_ZH.md">中文</a> </p>

📚 Introduction

Ultra-FineWeb-Classifier is a lightweight bilingual quality classifier for selecting high-quality documents from large-scale English and Chinese web corpora. It is developed through the efficient verification-based filtering pipeline proposed in the Ultra-FineWeb technical report and implemented with fastText to provide efficient, low-cost inference at web scale.

The repository provides separate English and Chinese classifier weights. Applying these classifiers to FineWeb and Chinese FineWeb produces Ultra-FineWeb, a higher-quality web pre-training dataset containing approximately 1T English tokens and 120B Chinese tokens. Ultra-FineWeb serves as a core pre-training web dataset for the MiniCPM4 Series and MiniCPM5 Series.

• Ultra-FineWeb-L1: L1 filtered data after basic cleaning, heuristic filtering, sensitive-field replacement, and deduplication.
• Ultra-FineWeb: L2 selected data containing approximately 1T English tokens and 120B Chinese tokens.
• Ultra-FineWeb-Classifier: lightweight English and Chinese quality classifiers for filtering web corpora. (Current classifier)
• Ultra-FineWeb-L3: L3 refined data built via Q&A pair generation and multi-style rewriting, containing 400B+ English tokens and 200B+ Chinese tokens.

📢 What's New

• [2026.08.20] The Ultra-FineWeb-L1 dataset is released, together with the L2 selected subset produced by the Ultra-FineWeb-Classifier. 🚀🚀🚀
• [2026.05.28] The Ultra-FineWeb-L3 dataset is released, containing 400B+ English tokens and 200B+ Chinese tokens.
• [2026.05.25] MiniCPM5-1B is released! Ultra-FineWeb serves as its core pre-training web dataset.
• [2026.02.08] The UltraData platform is now live, introducing the L0-L4 tiered data management framework.
• [2025.06.16] The Ultra-FineWeb-classifier is now available on Hugging Face: openbmb/Ultra-FineWeb-classifier. 🚀🚀🚀
• [2025.06.06] Ultra-FineWeb-en and Ultra-FineWeb-zh are now available on Hugging Face, released alongside the MiniCPM4 Series models.
• [2025.05.15] Ultra-FineWeb tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️
• [2025.05.09] The Ultra-FineWeb technical report is available on arXiv. 🔥🔥🔥

💡 Highlights

Abstract: Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it still faces two main challenges: (1) the lack of an efficient data verification strategy makes it difficult to provide timely feedback on data quality; and (2) the selection of seed data for training classifiers lacks clear criteria and relies heavily on human expertise, introducing a degree of subjectivity. To address the first challenge, we introduce an efficient verification strategy that enables rapid evaluation of the impact of data on LLM training with minimal computational cost. To tackle the second challenge, we build upon the assumption that high-quality seed data is beneficial for LLM training, and by integrating the proposed verification strategy, we optimize the selection of positive and negative samples and propose an efficient data filtering pipeline. This pipeline not only improves filtering efficiency, classifier quality, and robustness, but also significantly reduces experimental and inference costs. In addition, to efficiently filter high-quality data, we employ a lightweight classifier based on fastText, and successfully apply the filtering pipeline to two widely-used pre-training corpora, FineWeb and Chinese FineWeb datasets, resulting in the creation of the higher-quality Ultra-FineWeb dataset. Ultra-FineWeb contains approximately 1 trillion (T) English tokens and 120 billion (B) Chinese tokens. Empirical results demonstrate that the LLMs trained on Ultra-FineWeb exhibit significant performance improvements across multiple benchmark tasks, validating the effectiveness of our pipeline in enhancing both data quality and training efficiency.

<div align="center"> <img src="assets/ultra-fineweb-pipeline.png" width="600"/> </div>

• Efficient Verification Strategy: We propose a computationally efficient verification strategy that enables rapid evaluation of the impact of data on LLM training performance with minimal computational cost, significantly improving the efficiency of high-quality data filtering experiments.
• Large-Scale High-Quality Pre-training Datasets: We design and implement an efficient high-quality data filtering pipeline, applied to the FineWeb and Chinese FineWeb datasets, resulting in the creation of higher-quality datasets, which can facilitate high-quality LLM training.
• Lightweight Classifier: The Ultra-FineWeb classifier significantly reduces inference costs, achieving superior performance on extracted text from the same data source, thus validating the effectiveness of our proposed data filtering pipeline in enhancing data quality and training efficiency.

🚀 Usage of Ultra-FineWeb Classifier

Inference single content

1. Put the content you want to infer into the scripts/local_scripts/single_content.txt file.
2. Run the scripts/local_scripts/infer_single_content.py script to infer the content:

# set the language you want to infer, support: en, zh
LANGUAGE=en
# set the tokenizer path, default: local_tokenizer
# user can also directly use "deepseek-ai/DeepSeek-V2"
TOKENIZER_PATH=local_tokenizer
# set the content file path, default: scripts/local_scripts/single_content.txt
CONTENT_FILE=scripts/local_scripts/single_content.txt

python scripts/local_scripts/infer_single_content.py --language ${LANGUAGE} --tokenizer-path ${TOKENIZER_PATH} --content-file ${CONTENT_FILE}

Then you can get the result in the terminal, such as:

Content: {User's input content}

Normalized content: {Normalized content}

  - Pred label: {Pred label}
  - Pred score: {Pred score}

Folder inference

Assume the input folder is data/input, the key of the content is content, and the output folder is data/output. User can run the scripts/local_scripts/infer_folder.py script to infer the folder:

# set the language you want to infer, support: en, zh
LANGUAGE=en
# set the data path
DATA_PATH=data/input
# set the save path
SAVE_PATH=data/output
# set the content key
CONTENT_KEY=content
# bellow are optional arguments
# set the tokenizer path, default: local_tokenizer
TOKENIZER_PATH=local_tokenizer
# set the processes number, default: 64
PROCESSES_NUM=64
# set the write batch size, default: 100
WRITE_BATCH_SIZE=100

python scripts/local_scripts/infer_folder.py \
    --language ${LANGUAGE} \
    --data-path ${DATA_PATH} \
    --save-path ${SAVE_PATH} \
    --content-key ${CONTENT_KEY} \
    --tokenizer-path ${TOKENIZER_PATH} \
    --processes-num ${PROCESSES_NUM} \
    --write-batch-size ${WRITE_BATCH_SIZE} \
    [--inplace]  # optional, delete the processed data and re-process the data

For Spark inference, we also provide scripts/spark_scripts/spark_infer.py, a demo script for users to run on the Spark cluster.

NOTE:

• The numpy version should be lower than 2.0 for the fasttext package.
• The config.json file is a fake config file, the parameters are used for the fasttext training.

❤️ Acknowledgements

• The Ultra-FineWeb classifier is built based on fastText.
• The Ultra-FineWeb-en dataset is built based on FineWeb.
• The Ultra-FineWeb-zh dataset is constructed based on IndustryCorpus2, MiChao, WuDao, SkyPile, WanJuan, ChineseWebText, TeleChat, and CCI3.

Thanks for their awesome work! Open-source contributions make Ultra-FineWeb possible! 🙌

🌟 Citation

If you find our work useful, please consider citing:

@misc{wang2025ultrafineweb,
  title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu},
  year={2025},
  eprint={2505.05427},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
}

💳 License

This project is released under the Apache 2.0. Please note that since Ultra-FineWeb is built using multiple datasets, users should check the LICENSE of each dataset individually to ensure proper usage and compliance.