russian-jokes-generator

Russian Jokes Generator

This repository contains three versions of a Transformer-based language model fine-tuned on a dataset of Russian jokes (Anecdotes). The models are designed to generate humorous and coherent Russian text. There are three branch available: "main" (with nano model), "mini", "small". Also in this repository remain the pretrained on the dataset Byte-level BPE Tokenizer. The most coherent and powerful model is small.

Model Details

Model Architecture

These models is a Transformer with ALiBi positional embeddings (or RoPE, Rotary Positional Embedding), Grouped-Query Attention (GQA), and SwiGLU activation. Two of three models were trained with Multi-Head Latent Attention.

There are three versions:

• Nano: 3 layers, 4 heads, 96 hidden dimensions.
• Mini: 6 layers, 6 heads, 384 hidden dimensions. Trained with RoPE and MHLA.
• Small: 12 layers, 12 heads, 768 hidden dimensions.Trained with RoPE and MHLA.
• Tokenizer: Byte-level BPE tokenizer trained on the Russian jokes dataset.

Training Details

1. Training Epochs are calculated from the number of full iterations of all dataset and were set from the n_step parameter in the initialization of Trainer. Finally, there are 1 for nano model, 1 for mini model, 6 for small model.

2. Batch Size: 32 - for nano and mini. 64 - for small.

3. Learning Rate: 5e-4 with cosine decay for small, 3e-4 for nano and small.

4. As a Loss were used Cross-entropy loss.

5. For Hardware were used NVIDIA A100 GPU in the Google Colab.

Performance

Nano Plots

Epoch:

Loss:

---

Mini Plots

Epoch:

Loss:

Small Plots

Epoch:

Loss:

Usage

You can load the models and tokenizer:

In the main branch - nano, in the mini - mini, in the small - small:

# Small model
model_small = TransformerForCausalLM.from_pretrained("estnafinema0/russian-jokes-generator ", revision="small")
tokenizer = ByteLevelBPETokenizer.from_pretrained("estnafinema0/russian-jokes-generator ")

To generate the examples with the initial prompt:

text = "Штирлиц пришел домой"
input_ids = torch.tensor(tokenizer.encode(text), device=device)
model_output = check_model.generate(
    input_ids[None, :], max_new_tokens=200, eos_token_id=tokenizer.eos_token_id, do_sample=True, top_k=10
)
tokenizer.decode(model_output[0].tolist())

Example Generations

Here are some examples of generated jokes by the small model:

1. Input: "Пришел Петя в баню и говорит"

   Output: "Пришел Петя в баню и говорит - Василий Иванович, вы знаете, кто я - Петя, или Петя? - Ахааха, и я - Ахаилая, я - Ахаил! - А какая Петя? - Я - Ахаилая! - Ну и я, когда я банкрот, банкротство, конечно..."

2. Input: "Вышел как-то на крыльцо"

   Output: "Вышел как-то на крыльцо, а там плачет. Стукнулся: упал, выпал. Плачет – упал."

3. Input: "Священник задает ребёнку вопрос" Output: "Священник задает ребёнку вопрос ему на ухо:- Что, братан, опять несёл?- Братан, ты что, братан, охуел?"

License

This model is licensed under the Apache 2.0 License.

This model has been pushed to the Hub using the PytorchModelHubMixin integration:

• Library: [More Information Needed]