FaridLLM-Tiny

FaridLLM

A modular large language model research project by Farid Ahmed.

This is a tiny experimental model built to validate the full training pipeline (dataset -> tokenizer -> dataloader -> forward -> loss -> backward -> optimizer -> scheduler -> checkpoint -> resume -> evaluation -> generation). It does not represent frontier-level performance.

Model Description

FaridLLM is a modular transformer with:

• Grouped Query Attention (GQA)
• Rotary Position Embeddings (RoPE)
• SwiGLU feed-forward layers
• RMSNorm
• Optional Sparse Mixture-of-Experts (off in this model)

This exported model is dense, trained on synthetic numeric sequences ('toy arith numeric sequences, pattern=arith, numeric vocab 32'), trained for 300 steps.

Architecture (from the actual checkpoint)

Tokenizer

ByteLevel BPE (Hugging Face tokenizers backend). Vocabulary size and contents match the checkpoint used during training. Special tokens: <pad>, <unk>, <bos>, <eos>, <mask>.

Intended Use

• Research and experimentation
• Validating training/inference pipelines
• Educational demonstrations of a small transformer

Limitations

• Tiny model; results do not represent frontier-level performance.
• Experimental: unvalidated outside the toy benchmark below.
• Benchmark coverage is limited (single tiny perplexity estimate, initially random, then trained for a few hundred steps on synthetic data).
• Outputs may be incorrect or nonsensical.
• Not a production system. Do not treat as such without further validation.

Training

Evaluation (measured, from the tiny experiment)

• Validation perplexity on 64 held-out sequences: 3.0788 (baseline at init: 275.6866)
• Train perplexity: 1.5325
• Greedy generation exact-match on 6 held-out prompts (16/16 tokens): 0.8333
• Overfit sanity passed: final loss 0.0207 on the 8-sequence fixed corpus; exact match 100%
• Checkpoint/resume equivalence passed (max loss diff 0.0 vs continuous run)

License

Apache-2.0