poetry-dual-gpt

<div align="center">

🎭 PoetryDuel-GPT

A 31.5M-parameter GPT-style Transformer for Vietnamese Poetry Dueling (Đối Thơ)

</div>

PoetryDuel-GPT is a GPT-style autoregressive Transformer built from scratch in raw PyTorch — zero HuggingFace wrappers. It performs đối thơ (Vietnamese poetry dueling): given one or two lines of Lục Bát poetry, the model responds with a matching couplet that follows rhyme, tone, and syllable constraints.

🚀 Try it live at doitho.net

---

✨ Features

• Pure PyTorch — No transformers, no trl, just torch.nn
• 5 Lục Bát rules enforced via control tokens: rhyme (vần lưng), tone (bằng-trắc), syllable count, trầm-bổng (diacritic pairs), rhythm
• Generation pipeline — Soft rhyme biasing, repetition penalty, top-k/top-p, generate-and-rerank
• FastAPI + React chat UI included
• Colab-ready — Single-click training from scratch

---

📊 Results

Evaluated on 116 held-out Lục Bát couplet prompts (v4.2.3):

Sample Output

Input:  Công cha như núi thái sơn
        Nghĩa mẹ như nước trong nguồn chảy ra

Output: Dù cho vật đổi sao dời
        Nhưng dân vẫn Việt bốn phương rạng ngời

Input:  Thân em như chẽn lúa đòng
        Phất phơ dưới ngọn nắng hồng ban mai

Output: Bao la trời đất cao vời
        Tình em tha thiết muôn lời nhớ thương

---

🏗️ Model Architecture

Training Data Format

<|start|> [LUC_BAT] [RHYME:X] [TONE:BBBBBB] [TRAMBONG:NH]
  câu lục <|linebreak|> câu bát <|reply|>
  câu lục <|linebreak|> câu bát <|end|>

Control tokens condition the model on rhyme target, tone pattern, and tonal contrast — enabling rule-compliant generation without external rule engines.

---

🚀 Quick Start

Installation

git clone https://github.com/weseegod/poetry-dual-gpt.git
cd poetry-dual-gpt
pip install -r requirements.txt

Download Checkpoint

# Option 1: From HuggingFace
pip install huggingface_hub
python -c "
from huggingface_hub import hf_hub_download
hf_hub_download('grindytech/poetry-dual-gpt', 'doi_tho_best.pt', local_dir='checkpoints')
hf_hub_download('grindytech/poetry-dual-gpt', 'poetry_bpe.model', local_dir='tokenizer')
"

# Option 2: Train from scratch (see Training section)

Generate Poetry

# Single prompt
python src/sample.py --checkpoint checkpoints/doi_tho_best.pt

# Interactive mode
python src/sample.py --interactive

Chat UI

cd client
python start.py
# Opens at http://localhost:8000
# Frontend at http://localhost:3000

---

🔬 How It Works

Control Token Conditioning

The model is trained with an explicit format that encodes poetic rules as special tokens:

<|start|> [LUC_BAT] [RHYME:ong] [TONE:BBBTTB] [TRAMBONG:NH]

During inference, tags are extracted from the user's input couplet:

• [RHYME:X] — Target rhyme group from position 8 of the input's second line
• [TONE:XXXXXX] — Bằng/Trắc tone pattern from the input's first line
• [TRAMBONG:NH/HN] — Trầm-Bổng (Ngang→Huyền or Huyền→Ngang)

The model attends to these tags via standard self-attention, learning to generate output whose rhyme, tone, and diacritic patterns match the constraints.

Lục Bát Rules Enforced

Generation Features

• Soft rhyme constraint — Rhyme candidates get +2.0 logit boost instead of hard masking
• Repetition penalty — Recent 16 tokens penalized (-1.2) for lexical diversity
• Top-k + Top-p — Dual filtering for quality/diversity balance
• Generate-and-rerank — N candidates scored by lexical diversity, BPE validity, repetition

---

📁 Project Structure

poetry-dual-gpt/
├── src/                    # Model, training, generation, preprocessing
│   ├── model.py            # PoetryDuelGPT — decoder-only Transformer
│   ├── train.py            # Training loop with cosine LR, mixed precision
│   ├── generation.py       # Canonical generation (CLI + API + eval)
│   ├── sample.py           # CLI demo with interactive mode
│   ├── dataset.py          # Example-aligned batching, curriculum datasets
│   ├── tones.py            # Tone classification, rhyme extraction
│   ├── preprocess.py       # Data preprocessing pipeline
│   └── train_bpe.py        # BPE tokenizer training
├── client/                 # FastAPI backend + React frontend
│   ├── server.py           # /chat endpoint with model inference
│   └── frontend/           # React chat UI
├── evaluate/               # Evaluation suite
│   ├── eval_rules.py       # 5-rule structural evaluation
│   └── eval_quality.py     # Semantic quality metrics
├── tests/                  # Unit tests
├── scripts/                # Deployment scripts
└── checkpoints/            # Model weights (download from HF)

---

🔧 Training

From Scratch

# 1. Download & preprocess dataset
python src/preprocess.py

# 2. Train BPE tokenizer (12K vocab)
python src/train_bpe.py --corpus data/poetry_corpus.txt

# 3. Train the model (~10K steps, ~3h on T4)
python src/train.py --mode train --name doi_tho_ --corpus data/poetry_corpus.txt

Google Colab

Use colab/colab_train.ipynb for one-click training on free Colab T4 GPU.

Training Config

---

📚 Dataset

Training data derived from phamson02/vietnamese-poetry-corpus on HuggingFace.

• 84,000 Lục Bát poems preprocessed into 540,000 window-1 couplet-to-couplet pairs
• Rhyme (from position 8), tone (BTB pattern), and Trầm-Bổng tags extracted per example
• Example-aligned batching — zero cross-poem noise

---

📄 License

MIT License — see LICENSE for details.

---

🎓 Citation

If you use this project in your research, please cite:

@software{poetry_dual_gpt,
  author = {Thanh},
  title = {PoetryDuel-GPT: A Transformer for Vietnamese Poetry Dueling},
  year = {2026},
  url = {https://github.com/weseegod/poetry-dual-gpt},
}

---

🙏 Acknowledgments

• phamson02/vietnamese-poetry-corpus for the training dataset
• Andrej Karpathy's nanoGPT for architectural inspiration
• The Vietnamese poetry community for preserving Lục Bát tradition