Stalk-mini

Stalk-Mini

A 43.66M parameter Llama-architecture causal language model trained from scratch on TinyStories.

Model Details

Model Description

Stalk-Mini is a compact causal language model built on the Llama architecture. Trained from scratch as an experimental model for low-latency inference on custom architectures.

• Developed by: Ventie
• Model type: LlamaForCausalLM
• Language(s) (NLP): English
• License: MIT
• Architecture: 43.66M Parameters
• Vocabulary Size: 16,000 (Custom BPE Tokenizer)

Uses

Direct Use

Intended for lightweight text generation benchmarks, local inference speed tests, and experimental LLM architecture evaluation.

Out-of-Scope Use

Not suitable for complex reasoning, instruction following, or production tasks requiring factual accuracy.

Training Details

Training Data

Trained on 50,000 samples from the TinyStories dataset.

Training Procedure

• Epochs: 2.0
• Batch Size / Steps: 3,126 steps
• Final Training Loss: ~1.54
• Optimizer: AdamW with cosine schedule

Technical Specifications

Compute Infrastructure

• Software Stack: PyTorch (CUDA 13.0), Hugging Face Transformers