Downloads · 30 days
0
whiterabbitneo-33b-tokfix
mcarey42/whiterabbitneo-33b-tokfix
whiterabbitneo-33b-tokfix is a machine learning model from mcarey42. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Tokenizer-only repo. Identical tokenizer.json / specialtokensmap.json to TheBloke/WhiteRabbitNeo-33B-v1-AWQ, but tokenizerconfig.json declares PreTrainedTokenizerFast instead of LlamaTokenizer. Under transformers=5 th…
Access
Public
Updated Jul 3, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.4 MB · 100%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Jul 3, 2026
- Updated
- Jul 3, 2026
- SHA
- f196a13a
Model card
From the Hugging Face model README
WhiteRabbitNeo-33B tokenizer fix
Tokenizer-only repo. Identical tokenizer.json / special_tokens_map.json to TheBloke/WhiteRabbitNeo-33B-v1-AWQ, but tokenizer_config.json declares PreTrainedTokenizerFast instead of LlamaTokenizer. Under transformers>=5 the Llama class mishandles the ByteLevel decoder and leaks Ġ/Ċ glyphs into vLLM output; the fast class decodes cleanly. Set worker-vllm TOKENIZER= to this repo while keeping the AWQ weights as MODEL_NAME.
- Library
- transformers
- Created
- Jul 3, 2026
- Updated
- Jul 3, 2026