llm-course-hw1

1 · Project Summary

This project started as a course assignment whose goal was to build a complete small-scale language-model pipeline from scratch:

1. Train a byte-level BPE tokenizer.
2. Implement a Transformer-based causal LM.
3. Train the LM on a corpus of Russian anecdotes.

All tasks are now complete; the repository history contains several model generations and ablation experiments.

---

2 · Model Variants

Baseline architecture – ALiBi positional bias · Grouped-Query Attention · SwiGLU feed-forward.

---

Enhanced architecture – Rotary Positional Embedding + Multi-Head Latent Attention on top of the baseline.

---

3 · Training & Data

• Dataset: a cleaned collection of Russian jokes (IgorVolochay/russian_jokes).
• Optimizer: AdamW, linear warm-up, gradient-clipping 1.0.
• Loss: masked next-token cross-entropy, padding-aware.

(The full training scripts and notebooks will be published right after grading is finished.)

---

4 · Evaluation

Lower is better. The rotary + latent tweak gives a consistent (though modest) improvement on both sizes.

---

5 · Repository Notes

• Each major experiment lives in its own commit tag.
• After course review, the complete training source will be pushed to the public branch.

---

6 · Quick peek at the model’s sense of humour

(exact raw samples taken from my own training run)

(top_k=10, max_new_tokens=200, eos_token_id=tokenizer.eos_token_id, do_sample=True)

---

7 · How to run the model

pip install transformers>=4.40 torch

import torch

REPO_NAME = "amocualg/llm-course-hw1"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = ByteLevelBPETokenizer.from_pretrained(REPO_NAME)
check_model = TransformerForCausalLM.from_pretrained(REPO_NAME)
check_model = check_model.to(device)
check_model = check_model.eval()

text = "Штирлиц пришел домой" # change if needed
input_ids = torch.tensor(tokenizer.encode(text), device=device)
model_output = check_model.generate(
    input_ids[None, :], max_new_tokens=200, eos_token_id=tokenizer.eos_token_id, do_sample=True, top_k=10
)
print(tokenizer.decode(model_output[0].tolist()))

ByteLevelBPETokenizer and TransformerForCausalLM will be available along with the source code

This model has been pushed to the Hub using the PytorchModelHubMixin