Downloads · 30 days
0
hpt-v2-tokenizer
nfeliperealbr/hpt-v2-tokenizer
hpt-v2-tokenizer is a machine learning model from nfeliperealbr. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Brazilian AI Tokenizer — SentencePiece BPE 32K
Access
Public
Updated Apr 29, 2026
Repo size
547 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model547 KB · 50%
At a glance
- Access
- Public
- Created
- Apr 29, 2026
- Updated
- Apr 29, 2026
- SHA
- 72dbe067
Model card
From the Hugging Face model README
HPT v2 — Tokenizer
Brazilian AI Tokenizer — SentencePiece BPE 32K
About
Bilingual (Portuguese + English) + Code tokenizer for HPT v2 (118M params).
Trained on:
- 143 MB Portuguese Wikipedia (20K articles)
- 191 MB English Wikipedia (35K articles)
- Code samples (Python, JavaScript, SQL, Bash)
Special Tokens
| Token | ID | Usage |
|---|---|---|
<pad> | 0 | Padding |
<s> | 1 | BOS |
</s> | 2 | EOS |
<unk> | 3 | Unknown |
| `< | user | >` |
| `< | assistant | >` |
| `< | system | >` |
| `< | thinking | >` |
| `< | /thinking | >` |
Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("tokenizer/hpt_tokenizer.model")
# Encode
ids = sp.EncodeAsIds("Oi, tudo bem?")
# Decode
text = sp.DecodeIds(ids)
Stats
- Vocab size: 32,000
- Training corpus: 334 MB (PT + EN + Code)
- Character coverage: 99.95%
- Byte fallback: enabled (no UNK ever)
NeuralSpark Labs 🇧🇷
- Created
- Apr 29, 2026
- Updated
- Apr 29, 2026