Monostich-2-base

<div align="center">

Monostich-2-base

Pretrained base model for the Monostich-2 family (~150M)

Second-generation Monostich-class model with a renewed hybrid architecture

</div>

---

What this is

Monostich-2-base is the pretrained (base) checkpoint for Monostich-2, my second Monostich-class language model.

• Successor to kerzgrr/Monostich (Monostich-1, ~100M LLaMA-style)
• Renewed architecture: hybrid Gated DeltaNet-2 recurrent layers + gated GQA full-attention layers (not a plain LLaMA stack)
• This repo is pretrain-only raw text continuation
• Instruction-tuned / chat (SFT): kerzgrr/Monostich-2

If you want chat behaviour, use kerzgrr/Monostich-2 — this base model is for continuation / research and will not follow instructions reliably.

---

Model Architecture

Pipeline: Text Prompt → BPE-49K Tokenizer → TinyGDN Hybrid Decoder (32L) → Next-token Prediction

Hybrid block schedule (×32)

Every 4th layer is full attention; the rest are Gated DeltaNet-2:

GDN2, GDN2, GDN2, GQA, … (3:1 recurrent-to-attention)

Technical specifications

---

Training (pretrain)

---

Install

1) System requirements

• Python 3.10+
• CUDA GPU strongly recommended (CPU works but is slow; GDN-2 kernels expect GPU for best path)
• PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example (CUDA 12.4 wheels):

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

Manual install is optional; see windows_fla_patches/ if you want to vendor FLA yourself.

5) Download the inference script

# wget / curl
curl -L -o inference.py https://huggingface.co/kerzgrr/Monostich-2-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Monostich-2-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo (cached after the first run). It also auto-installs flash-linear-attention if missing.

---

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Example:

python inference.py \
  --prompt "In a distant galaxy," \
  --temperature 0.9 \
  --top-p 0.95 \
  --max-new-tokens 200 \
  --seed 1234

---

Pretrain vs chat

---

Files

kerzgrr/Monostich-2-base/
  README.md                 # This model card
  inference.py              # Terminal inference (streaming)
  requirements.txt          # Pip helper list
  model.safetensors         # EMA weights (bfloat16)
  config.json               # TinyGDN architecture config
  tokenizer.json            # BPE tokenizer
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  tiny_gdn/                 # Minimal model package for inference
    __init__.py
    config.py
    model.py

---

Limitations

• Base model: not instruction-tuned; may ramble, ignore “user” framing, or fail at Q&A format
• Scale: ~150M parameters — research / edge prototype, not a frontier model
• Dependency: requires flash-linear-attention (GDN-2); not GGUF / llama.cpp compatible today
• Context: trained at 2,048; longer windows are experimental

---

Model family

---

Citation

@misc{monostich2base2026,
  title={Monostich-2-base: A Hybrid GDN-2 + GQA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Monostich-2-base}
}

---

Acknowledgments

• flash-linear-attention (Gated DeltaNet-2)
• FineWeb-Edu
• Chat / SFT release: kerzgrr/Monostich-2
• Predecessor: kerzgrr/Monostich
• PyTorch SDPA / Hugging Face Hub + tokenizers

---

<div align="center">

A monostich is a poem of a single line — small, but complete.
Monostich-2 renews the form.

</div>