Downloads · 30 days
0
GneissWeb.bloom
ibm-granite/GneissWeb.bloom
GneissWeb.bloom is a machine learning model from ibm-granite. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Access
Public
Updated Feb 21, 2025
Repo size
28.8 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bloom28.8 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Feb 12, 2025
- Updated
- Feb 21, 2025
- SHA
- a2db4b89
Model card
From the Hugging Face model README
Model Summary
In order to be able to reproduce GneissWeb, we provide here a Bloom filter representing all the document ids of FineWeb 1.1.0 whose documents are part of GneissWeb. it is of size 28GB and is of the rbloom family of Bloom filters. It is to be probed with the id column of FineWeb 1.1.0 or of Common Crawl.
Please refer to the GneissWeb page for more details.
Developers: IBM Research
Release Date: Feb 21st, 2025
License: Apache 2.0.
Testing
The Bloom Filter was tested with
Positive Examples : ~10M uuids from 192 parquet files in GneissWeb. These span all 96 snapshots
Negative Examples : 10,000 uuids in CC-MAIN-2024-51 (not present in FineWeb 1.1.0 and also not in GneissWeb)
The Bloom Filter was able to return correct answers for all of them
- License
- apache-2.0
- Languages
- en
- Created
- Feb 12, 2025
- Updated
- Feb 21, 2025