Downloads · 30 days
80
0% of all-time downloads
rinna/gemma-2-baku-2b
gemma-2-baku-2b is a text generation model from rinna. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
Downloads · 30 days
80
0% of all-time downloads
All-time downloads
16.9K
Public
Parameters
2.6B
10.5 GB on disk
Likes
7
Public
Click a slice to open those files.
.safetensors10.5 GB · 100%
From the Hugging Face model README
Gemma 2 Baku 2B (rinna/gemma-2-baku-2b)
We conduct continual pre-training of google/gemma-2-2b on 80B tokens from a mixture of Japanese and English datasets. The continual pre-training improves the model's performance on Japanese tasks.
The name baku comes from the Japanese word 獏/ばく/Baku, which is a kind of Japanese mythical creature (妖怪/ようかい/Youkai).
Library
The model was trained using code based on Lightning-AI/litgpt.
Model architecture
A 26-layer, 2304-hidden-size transformer-based language model. Please refer to the Gemma 2 Model Card for detailed information on the model's architecture.
Training
The model was initialized with the google/gemma-2-2b model and continually trained on around 80B tokens from a mixture of the following corpora
Contributors
Release date
October 3, 2024
Please refer to rinna's LM benchmark page (Sheet 20241003).
import transformers
import torch
model_id = "rinna/gemma-2-baku-2b"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "eager"},
device_map="auto"
)
output = pipeline(
"西田幾多郎は、",
max_new_tokens=256,
do_sample=True
)
print(output[0]["generated_text"])
It is recommended to use eager attention when conducting batch inference under bfloat16 precision. Currently, Gemma 2 yields NaN values for input sequences with padding when the default attention mechanism (torch.scaled_dot_product_attention) is employed in conjunction with bfloat16.
The model uses the original google/gemma-2-2b tokenizer.
@misc{rinna-gemma-2-baku-2b,
title = {rinna/gemma-2-baku-2b},
author = {Wakatsuki, Toshiaki and Chen, Xinqi and Sawada, Kei},
url = {https://huggingface.co/rinna/gemma-2-baku-2b}
}
@inproceedings{sawada2024release,
title = {Release of Pre-Trained Models for the {J}apanese Language},
author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
month = {5},
year = {2024},
pages = {13898--13905},
url = {https://aclanthology.org/2024.lrec-main.1213},
note = {\url{https://arxiv.org/abs/2404.01657}}
}
@article{gemma-2-2024,
title = {Gemma 2},
url = {https://www.kaggle.com/models/google/gemma-2},
publisher = {Kaggle},
author = {Gemma Team},
year = {2024}
}
@misc{litgpt-2023,
author = {Lightning AI},
title = {LitGPT},
howpublished = {\url{https://github.com/Lightning-AI/litgpt}},
year = {2023}
}