Downloads · 30 days
0
KhadijaMir/P1_financial_news_headlines_CharBPETokenizer
P1_financial_news_headlines_CharBPETokenizer is a machine learning model from KhadijaMir. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Hugging Face CharBPETokenizer trained on 410,849 raw train-period headlines (2019-08-02 to 2020-09-30) only, following the course L3 recipe: minfrequency=5, special tokens <s <pad </s <unk <mask, case and punctuation…
Downloads · 30 days
0
Access
Public
Updated Sep 24, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json3.7 MB · 100%
From the Hugging Face model README
Hugging Face CharBPETokenizer trained on 410,849 raw train-period headlines (2019-08-02 to 2020-09-30) only, following the
course L3 recipe: min_frequency=5, special tokens <s> <pad> </s> <unk> <mask>, case and punctuation kept.
Vocabulary target 50,000 (chosen by a 5% compression rule over 5k/10k/20k/50k); 34,541 tokens learned.
8.90 spaCy words vs 17.54 BPE tokens per headline; unknown-token rate on later headlines 0.004%.
Project details: https://huggingface.co/datasets/KhadijaMir/SP_DOW_NASDAQ_stocks__News_Headlines_labeled/tree/project1_triple_barrier