Downloads · 30 days
13
13% of all-time downloads
kipasyangin5/5m-terminal-lm-chinchilla
5m-terminal-lm-chinchilla is a text generation model from kipasyangin5. Use it when you need the model to write or continue text. The card lists the license as mit.
This repository contains a 5.0 Million Parameter Causal Language Model trained from scratch on a compute-optimal token budget of 100 Million Tokens (20x parameter count) following Chinchilla scaling laws.
Downloads · 30 days
13
13% of all-time downloads
All-time downloads
103
Public
Parameters
4.1M
33.1 MB on disk
Likes
0
Public
Click a slice to open those files.
.pt16.7 MB · 50%
From the Hugging Face model README
This repository contains a 5.0 Million Parameter Causal Language Model trained from scratch on a compute-optimal token budget of 100 Million Tokens (20x parameter count) following Chinchilla scaling laws.
The model is specialized in Linux Terminal Commands, Shell Automation, and Multilingual Text Generation (English, Indonesian, Spanish, French, German).
The model was trained on 100,000,000 Tokens mixed across three distinct sources:
| Dataset Name | Source / Hugging Face ID | Token Count | Share (%) | Utilization Details |
|---|---|---|---|---|
| Linux Terminal & CLI Engine | kipasyangin5/terminal-cli-commands-dataset | ~20.0 Million | 20% | 100% Fully Utilized: Pure synthesized Linux CLI commands (cd .., ls -la, mkdir, grep, git, chmod, curl, Q&A pairs). |
| English General Text | wikitext (wikitext-2-raw-v1) | ~24.0 Million | 24% | Streamed Sample: Streamed line-by-line via Hugging Face load_dataset("wikitext", streaming=True). |
| Multilingual Wikipedia | wikimedia/wikipedia | ~56.0 Million | 56% | Streamed Multilingual Sample: Streamed across 4 language sub-splits (Indonesian id, Spanish es, French fr, German de). |
| TOTAL | Mixed Corpus | 100.0 Million | 100% | Chinchilla Compute-Optimal (20x params) |
huggingface.co/datasets/kipasyangin5/terminal-cli-commands-datasethuggingface.co/datasets/kipasyangin5/5m-terminal-lm-datasetThe full, self-contained training codebase is included directly in this repository inside the /training_code folder:
training_code/main_kaggle.py: Complete Kaggle GPU standalone training script.training_code/train.py: Main PyTorch model architecture, loss functions, and HF auto-upload.training_code/terminal_dataset.py: CLI command generator and shell interaction builder.training_code/tokenizer_builder.py: Custom 4,096 BPE Tokenizer trainer.d_model): 256n_layer): 6n_head): 8 (Head dimension = 32)inter_dim): 512 (SwiGLU activation)max_seq_len): 256 tokensfrom transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "kipasyangin5/5m-terminal-lm-chinchilla"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer("User: How do I navigate up one directory?\nAssistant:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))
MIT License. Developed by kipasyangin5.