AlpineLLM-Tiny-10M-Base

<div align="center"> <h1>AlpineLLM-Tiny-10M-Base</h1> <p> A domain-specific language model for alpine storytelling. <br> Try asking about mountain adventures! 🏔️ <br> <strong>Author:</strong> <a href="https://borzyszkowski.github.io/">Bartek Borzyszkowski</a> </p> </div> <div align="center"> <img src="https://github.com/Borzyszkowski/AlpineLLM/blob/main/assets/background_round.png?raw=true" width="60%" alt="AlpineLLM" /> </div> <p align="center"> <a href="https://github.com/Borzyszkowski/AlpineLLM"><b>⛏️ AlpineLLM GitHub</b></a> | <a href="https://huggingface.co/spaces/Borzyszkowski/AlpineLLM-App"><b>🤗 AlpineLLM Live Demo</b></a> </p>

About AlpineLLM

AlpineLLM-Tiny-10M-Base is a lightweight base language model with ~10.8 million trainable parameters. It was pre-trained from scratch on raw text corpora drawn primarily from public-domain literature on alpinism, including expedition narratives and climbing essays. Please note that AlpineLLM is a base model, and it has not been fine-tuned for downstream tasks such as summarization or dialogue. Its outputs reflect patterns learned directly from the training texts.

Model Details

• Model name and version: AlpineLLM-Tiny-10M-Base
• Model type: Language Model (Base Model)
• Architecture: TransformerDecoder
• Parameters: ~10M
• Context window size: 256 tokens
• Tokenization: CharacterLevelTokenizer
• Vocabulary size: 96 tokens
• Training language: English
• Training set size: ~1M tokens
• Training iterations: ~15K iters

Training Setup

• Hardware: 1X NVIDIA RTX A4000
• Software: Ubuntu 22.04 with CUDA 12.2 and Python3.10
• Pre-training duration: ~1h

For a complete list of dependencies, refer to the source training repository.

Usage

Check out the <a href="https://huggingface.co/spaces/Borzyszkowski/AlpineLLM-App"> AlpineLLM Live Demo</a> repository on HuggingFace Spaces for a practical implementation.

Citation

@misc{alpinellm2025,
  title={AlpineLLM: A Domain-Specific Language Model for Alpine Storytelling.},
  author={Borzyszkowski, Bartek},
  year={2025},
  publisher={Hugging Face},
  howpublished={https://github.com/Borzyszkowski/AlpineLLM}
}