NEXORA

Nexora-1

Nexora-1 is a compact GPT-style generative language model built from scratch using PyTorch.

It is a Transformer-based autoregressive language model trained on English Wikipedia text using next-token prediction.

The project was developed to understand the fundamental components behind GPT-style language models, including tokenization, embeddings, positional embeddings, self-attention, multi-head attention, Transformer blocks, autoregressive training, checkpointing, and text generation.

Model Architecture

• Architecture: GPT-style Transformer
• Transformer Layers: 6
• Attention Heads: 8
• Embedding Dimension: 384
• Context Length: 256 tokens
• Vocabulary Size: 8,000
• Dropout: 0.1
• Tokenization: Byte-Level BPE
• Framework: PyTorch

Training Dataset

Nexora-1 was pretrained on English Wikipedia text.

The training pipeline was configured to collect Wikipedia articles and create a text corpus for autoregressive language-model training.

Articles were separated using the special token:

<|endofarticle|>