Downloads · 30 days
0
GneissWeb.Quality_annotator
ibm-granite/GneissWeb.Quality_annotator
GneissWeb.Quality_annotator is a machine learning model from ibm-granite. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Access
Public
Updated Feb 21, 2025
Repo size
2.4 GB
Likes
5
Public
Repo makeup
Click a slice to open those files.
.bin2.4 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Feb 12, 2025
- Updated
- Feb 21, 2025
- SHA
- 1817e4c8
Model card
From the Hugging Face model README
Model Summary
This fastText model is used as part of the ensemble filter in GneissWeb to detect and remove low-quality documents.
Please refer to the GneissWeb dataset page for more details.
- Developers: IBM Research
- Release Date: Feb 21st, 2025
- License: Apache 2.0
Training Data
The model is trained on 400k documents, equality split between positive (i.e., high-quality) and negative (i.e., low-quality) classes. Please refer to fasttext text classification tutorial for details. Training data is selected as follows.
- Positive documents: 190k synthetic documents randomly sampled from the Cosmopedia dataset, and 10k documents with high educational value selected as follows: first, 600k random documents from FineWeb-V1.1.0 are annotated asking Mixtral-8x22B-Instruct to score each document between 1 to 5 for its educational quality (with 5 being the highest quality), using a prompt similar to the one used by FineWeb-Edu. Then, 10k random documents are selected from documents with scores greater than or equal to 4.
- Negative documents: 200k random documents out of the 600k Mixtral-annotated documents with scores less than or equal to 2.
- License
- apache-2.0
- Created
- Feb 12, 2025
- Updated
- Feb 21, 2025