Downloads · 30 days
139
16% of all-time downloads
tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer
Japanese_to_english_gpt2CasualLM_GemmaTokenizer is a machine learning model from tirthadagr8. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Made using Gpt-Small from scratch for learning purpose. Tokenizer used is from Gemma 2-2B-JPN-IT which is trained on japanese dataset from JESC.
Downloads · 30 days
139
16% of all-time downloads
All-time downloads
874
Public
Parameters
282M
10.3 GB on disk
Likes
2
Public
Click a slice to open those files.
.pt2.3 GB · 66%
From the Hugging Face model README
Made using Gpt-Small from scratch for learning purpose. Tokenizer used is from Gemma 2-2B-JPN-IT which is trained on japanese dataset from JESC.
Model usage:-
from transformers import AutoTokenizer,AutoModelForCausalLM
tokenizer=AutoTokenizer.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
model=AutoModelForCausalLM.from_pretrained('tirthadagr8/Japanese_to_english_gpt2CasualLM_GemmaTokenizer')
model.cuda()
src_text='あなたとは遊びたくない'
print(tokenizer.batch_decode(model.generate(tokenizer.encode(f"Translate the following Japanese sentence to English:\n\nJapanese:{src_text}\nEnglish:",return_tensors='pt')[:,:-1].cuda(),max_length=128))[0])
OUTPUT:
<bos>Translate the following Japanese sentence to English:
Japanese:あなたとは遊びたくない
English:i don't want to play with you.<eos>
@ARTICLE{pryzant_jesc_2018,
author = {{Pryzant}, R. and {Chung}, Y. and {Jurafsky}, D. and {Britz}, D.},
title = "{JESC: Japanese-English Subtitle Corpus}",
journal = {Language Resources and Evaluation Conference (LREC)},
keywords = {Computer Science - Computation and Language},
year = 2018
}