nanogpt-enwik8-compressed

NanoGPT enwik8 - Compressed Model

Compressed nanoGPT model trained on enwik8 (Wikipedia) using low-rank matrix decomposition.

Model Details

• Original Parameters: 28,801,536
• Compressed Parameters: 22,755,840
• Compression Ratio: 1.27× smaller
• Compression Method: Low-rank decomposition (rank=16) on layers [5, 6, 7]
• Training Data: enwik8 (Wikipedia, first 100MB)
• Vocabulary: 6,060 characters
• Context Length: 1024 tokens

Performance

• Original Perplexity: 8843.82
• Compressed Perplexity: 7387.50
• Performance Change: -16.5%

Usage

⚠️ Note: This model requires custom code for text generation due to character-level tokenization.

# This model is designed for research and benchmarking
# Custom generation code required

Compression Technique

Uses SVD-based low-rank approximation:

• Method: Decompose weight matrices W ≈ U × V
• Rank: 16 (much smaller than original dimensions)
• Layers: Compressed MLP layers in transformer blocks [5, 6, 7]

Evaluation

Ready for benchmark evaluation including:

• Nous benchmark suite (AGIEval, GPT4ALL, TruthfulQA, Bigbench)
• Compression technique analysis
• Character-level language modeling research

Citation

Based on nanoGPT by Andrej Karpathy. Compression technique demonstrates effective neural network compression with minimal performance impact.