fast_stupidity

---

---

tags: [text-generation, causal-lm, rotary-positional-embedding, gradio-app] language: en license: apache-2.0 metrics: [perplexity]

Model Card — fast_stupidity

One-line summary:
A 34-layer causal Transformer language model with rotary embeddings, trained from scratch with a BERT tokenizer and served via Gradio for interactive text generation.

---

Model details

• Architecture: Decoder-only Transformer (GPT-style)
• Layers: 34
• Embedding dimension: 1280
• Attention heads: 20 (64 dim/head)
• FFN hidden dim: 5120 (4 × embedding dim)
• Context length: 1216 tokens
• Tokenizer: BERT tokenizer
• Parameters: 748,089,280
• Positional encoding: Rotary embeddings (RoPE)
• Activation: SwiLU/SiLU
• Norm: LayerNorm
• Checkpoint format: PyTorch final_checkpoint.pt

---

Intended uses

• General text generation (English, tokenizer-dependent)
• Interactive prompting with sampling controls (temperature, top-k) via Gradio UI

---

Limitations & Risks

• May hallucinate facts or generate incoherent outputs
• Limited context length (1216 tokens)

---

Training data & procedure

• Tokenizer: local BERT tokenizer (vocab size 30,522)
• Objective: Causal language modeling
• Training framework: PyTorch + custom FSDP/AMP setup
• Checkpoints: Final checkpoint (final_checkpoint.pt) available

---