sarashina2.1-1b

Sarashina2.1-1B

This repository provides large language models trained by SB Intuitions.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
 
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2.1-1b", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2.1-1b")
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
set_seed(123)
 
text = generator(
    "おはようございます、今日の天気は",
    max_length=30,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
    num_return_sequences=3,
)

for t in text:
    print(t)

Model Description

We constructed this Sarashina2.1-1B model, which consists of 1 billion parameters, using a two-phase training process. First, we trained the model on 10 trillion tokens, including Japanese and English data extracted from web corpora. Then, we trained the model using 1 trillion tokens, predominantly consisting of Japanese data, to enhance its performance in Japanese. The following tables show the model's performance on Japanese and English tasks. We also show the performance of other public LLMs for reference.

Evaluation in Japanese tasks

Evaluation in English tasks

Ethical Considerations and Limitations

Sarashina2.1 has not been tuned to follow an instruction yet. Therefore, sarashina2.1 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs. Before using sarashina2.1, we would like developers to tune models based on human preferences and safety considerations.

License

Sarashina Model NonCommercial License Agreement