Downloads · 30 days
0
suffix_array_fineweb-edu-10BT-0.1
yoheikobashi/suffix_array_fineweb-edu-10BT-0.1
suffix_array_fineweb-edu-10BT-0.1 is a machine learning model from yoheikobashi. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Querying 2,000 texts takes about 3-10 seconds.
Access
Public
Updated Jul 2, 2025
Repo size
12.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.npy12.1 GB · 100%
At a glance
- Access
- Public
- Created
- Jul 2, 2025
- Updated
- Jul 2, 2025
- SHA
- 9b25f9d1
Model card
From the Hugging Face model README
git clone [email protected]:yohei-kobashi/crypto_llm.git
cd crypto_llm
pip install -r requirements.txt
huggingface-cli download yoheikobashi/suffix_array_fineweb-edu-10BT-0.1 --local-dir ./index_10BT
python script/preprocess/suffix_array_search.py --index-dir ./index_10BT --input [dirname including txt or jsonl files] --window 35
Querying 2,000 texts takes about 3-10 seconds.
- Created
- Jul 2, 2025
- Updated
- Jul 2, 2025