kotoba-whisper-bilingual-ct2

Converted https://huggingface.co/kotoba-tech/kotoba-whisper-bilingual-v1.0 to a CT2 model. Tested compatibility with https://github.com/m-bain/whisperX.

1. Generate tokenizer.json

Make sure transformers and ctranslate2 is installed. In the interactive python shell:

# Import the transformers module
import transformers
# Import the AutoTokenizer submodule (so you won't run into NameError: name 'AutoTokenizer' is not defined)
from transformers import AutoTokenizer
# Load tokenizer from existing files
tokenizer = AutoTokenizer.from_pretrained(<source_model_dir>)

# Save the tokenizer data as tokenizer.json
tokenizer.save_pretrained(<output_dir>, legacy_format=False)

1. Copy all generated files (tokenizer_config.json, tokenizer.json) into source_model_dir

2. Generate CT2 Model

ct2-transformers-converter --model <source_model_dir> --copy_files tokenizer.json preprocessor_config.json --output_dir <output_dir> --quantization float32

The model can do translation to English (using --task translate --language=en), and transcription (using --task transcribe --language=ja).