llm-course-hw1

LLM Course HW1 Transformer

Model Description

This model is a Transformer-based causal language model developed as part of an educational assignment. It employs a custom Byte-Level BPE tokenizer and incorporates modern techniques such as:

• ALiBi for positional embeddings
• Grouped-Query Attention (GQA)
• SwiGLU in the feed-forward layers

Additional optional components include Rotary Positional Embedding and Multi-Head Latent Attention.

Model Architectures

Multiple configurations are available:

• Nano: 3 layers, 4 heads, 96 hidden dimensions
• Mini: 6 layers, 6 heads, 384 hidden dimensions
• Small: 12 layers, 12 heads, 768 hidden dimensions

Training Details

• Dataset: Trained on IgorVolochay/russian_jokes
• Sequence Length: 128 tokens per sequence
• Tokenizer: Custom Byte-Level BPE tokenizer with a vocabulary size of 1024
• Objective: Causal language modeling with cross-entropy loss

Intended Use

This model is intended for educational and experimental purposes in the realm of language modeling. It is suitable for generating informal and creative text (e.g., jokes) and can serve as a baseline for research on efficient Transformer architectures.

Limitations

• Trained on a niche dataset (Russian jokes), which may limit its generalizability.
• Its small-scale architecture might not perform well on large or diverse text corpora.

How to Use

• Tokenization: Utilize the provided Byte-Level BPE tokenizer for encoding and decoding text.
• Generation: Leverage the built-in generation function with adjustable parameters (max tokens, temperature, top-k sampling).
• Integration: Load the model using Hugging Face’s from_pretrained method.