5m-terminal-lm-saturated

5M Terminal & Multilingual Language Model (Full Saturation - 250M Tokens)

This repository contains a 5.0 Million Parameter Causal Language Model trained from scratch on an over-trained budget of 250 Million Tokens (50x parameter count) for maximum model saturation and peak accuracy.

The model is specialized in Linux Terminal Commands, Shell Automation, and Multilingual Text Generation (English, Indonesian, Spanish, French, German).

---

📊 Dataset Breakdown & Raw Corpus Links

The model was trained on 250,000,000 Tokens mixed across three distinct sources:

🔗 Direct Dataset Downloads:

• 💻 Pure CLI Commands Dataset: huggingface.co/datasets/kipasyangin5/terminal-cli-commands-dataset
• 📦 Full 50M Token Mixed Corpus: huggingface.co/datasets/kipasyangin5/5m-terminal-lm-dataset

---

📂 Complete Training Source Code

The full, self-contained training codebase is included directly in this repository inside the /training_code folder:

• training_code/main_kaggle.py: Complete Kaggle GPU standalone training script.
• training_code/train.py: Main PyTorch model architecture, loss functions, and HF auto-upload.
• training_code/terminal_dataset.py: CLI command generator and shell interaction builder.
• training_code/tokenizer_builder.py: Custom 4,096 BPE Tokenizer trainer.

---

⚙️ Model Architecture

• Total Trainable Parameters: 4,984,064 (~4.98M / 5.0M)
• Architecture: Decoder-only Transformer (LLaMA style with SwiGLU & RMSNorm)
• Vocabulary Size: 4,096 (Byte-Pair Encoding Tokenizer)
• Hidden Dimension (d_model): 256
• Number of Layers (n_layer): 6
• Number of Attention Heads (n_head): 8 (Head dimension = 32)
• MLP Hidden Dimension (inter_dim): 512 (SwiGLU activation)
• Context Window (max_seq_len): 256 tokens
• Weight Tying: Tied Embedding and LM Head weights

---

💻 Usage & Inference Example (PyTorch)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "kipasyangin5/5m-terminal-lm-saturated"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

inputs = tokenizer("User: How do I navigate up one directory?\nAssistant:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))

Citation & License

MIT License. Developed by kipasyangin5.