gpt2-numfix

cyrilzhang/gpt2-numfix

gpt2-numfix is a machine learning model from cyrilzhang. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.

A fork of the GPT-2 tokenizer, which removes multi-digit tokens:

Downloads · 30 days

6% of all-time downloads

All-time downloads

245

Public

Repo size

—

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.json2.8 MB · 86%

At a glance

Library: transformers
License: mit
Access: Public
Created: Dec 24, 2022
Updated: Dec 25, 2022
SHA: f50db44f

Model card

From the Hugging Face model README

GPT-2 Tokenizer with unmerged digits

A fork of the GPT-2 tokenizer, which removes multi-digit tokens:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('cyrilzhang/gpt2-numfix')

tokenizer('123.45')  # [16, 17, 18, 13, 19, 20]
gpt2_tokenizer('123.45')  # [10163, 13, 2231]

Backward-compatible:

tokenizer.decode([10163, 46387])  # '<unused123> pigeon'
gpt2_tokenizer.decode([10163, 46387])  # '123 pigeon'

This is for my investigations into the arithmetic capabilities of large language models. There is no model here, only a tokenizer.
PaLM does this. I think it's very reasonable.
Many models (illustriously, GPT-3) don't do this, because they use the GPT-2 tokenizer.

Library: transformers
License: mit
Created: Dec 24, 2022
Updated: Dec 25, 2022