Downloads · 30 days
0
hf-slow-tokenizer-v21b1
llm-jp/hf-slow-tokenizer-v21b1
hf-slow-tokenizer-v21b1 is a machine learning model from llm-jp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Usage: (you need to specify usefast=False, trustremotecode=True as the args of AutoTokenizer.frompretrained())
Access
Public
Updated Sep 20, 2023
Repo size
832 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model832 KB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Sep 19, 2023
- Updated
- Sep 20, 2023
- SHA
- c0972984
Model card
From the Hugging Face model README
Usage: (you need to specify use_fast=False, trust_remote_code=True as the args of AutoTokenizer.from_pretrained())
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"llm-jp/hf-slow-tokenizer-v21b1",
legacy=True,
use_fast=False,
trust_remote_code=True,
)
When you create the tokenizer instance from the downloaded model with trust_remote_code=True, the model directory must be placed directly under the current directory, and the pretrained_model_name_or_path should not start with "./" but should end with "/".
tokenizer = AutoTokenizer.from_pretrained(
"hf-slow-tokenizer-v21b1/",
legacy=True,
use_fast=False,
trust_remote_code=True,
)
- License
- apache-2.0
- Created
- Sep 19, 2023
- Updated
- Sep 20, 2023