StellarAI-1-beta-0.05b

StellarAI-Tiny

A lightweight multimodal language model trained from scratch — ~50M parameters (0.05B), runs on CPU with 4GB RAM.

---

Overview

StellarAI-Tiny is a from-scratch, bilingual (Chinese + English) causal language model with multimodal vision support. Designed for educational and prototyping purposes, it requires minimal hardware and ships with a built-in plugin system for tool calling.

---

Architecture

StellarAI-Tiny (~50M Parameters)
├── Embedding          vocab(32000) x d_model(384)
├── Text Transformer (4 layers)
│   ├── Multi-Head Self-Attention (6 heads, RoPE)
│   └── FFN (GELU, intermediate=1536)
├── Vision Encoder (CNN + ViT)
│   ├── CNN Feature Extractor (4 layers: 24→48→96→192 channels)
│   └── ViT Transformer (2 layers, 6 heads)
├── Fusion Block (1 layer)
│   ├── Self-Attention + Cross-Attention
│   └── FFN (1536)
└── LM Head (tied weights)

---

Quick Start

Requirements

pip install torch transformers safetensors

Text Generation

from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
import torch

model_name = "amtstudio/stellarai-tiny"  # or your local path

config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    trust_remote_code=True,
    torch_dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

# Generate
prompt = "Artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt")

output_ids = model.generate(
    **inputs,
    max_new_tokens=64,
    temperature=0.7,
    top_k=40,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Using generate_text Convenience Method

result = model.generate_text(
    prompt="Artificial intelligence is",
    tokenizer=tokenizer,
    max_new_tokens=100,
    temperature=0.8,
)
print(result)

---

Training Details

The training corpus was built from a mix of hand-crafted bilingual data, synthetic instruction-tuning data, and Chinese NLP datasets across 10+ domains. The model was trained with a next-token-prediction objective using the custom SimpleTokenizer (BPE).

---

File Structure

├── config.json                    # HF model configuration
├── configuration_stellarai.py     # Custom PretrainedConfig class
├── modeling_stellarai.py          # Custom PreTrainedModel class
├── tokenization_stellarai.py      # Custom PreTrainedTokenizer class
├── model.safetensors              # Safetensors weights (93.6 MB)
├── pytorch_model.bin              # PyTorch weights (93.6 MB)
├── tokenizer_config.json          # Tokenizer configuration
├── special_tokens_map.json        # Special token mappings
├── tokenizer.json                 # HF tokenizer definition (BPE)
├── backend_tokenizer.json         # Original backend tokenizer
├── vocab.json                     # BPE vocabulary (11,030 tokens)
├── merges.txt                     # BPE merge rules
├── README.md                      # This file
└── LICENSE                        # MIT License

---

Limitations

Important: This is a lightweight educational / prototyping model.

1. Limited knowledge: Trained on ~19K lines of curated data. Knowledge coverage is narrow.
2. Factual accuracy: May produce inaccurate, nonsensical, or hallucinated content.
3. Generation quality: Suitable for demonstrating basic language modeling — not production-level dialogue.
4. Vision capability: The vision encoder is pre-trained on text-only data. VQA requires additional fine-tuning with image-text pairs.
5. Plugin calling: The model learned the [TOOL:xxx] format but calling accuracy needs improvement.
6. Not suitable for: Production environments, medical/legal/financial domains.

Suggested Improvements

• Expand corpus to 50K+ lines or use public datasets (WikiText, C4, Oscar)
• Increase training to 50K+ steps
• Add real dialogue data (ShareGPT, Alpaca format) for SFT
• Collect image-text pairs (e.g., COCO captions) to fine-tune multimodal capability
• Try larger config: 6 layers / 512d / 8 heads (~0.1B)

---

License

MIT License — fully permissive for personal and commercial use.

---

Acknowledgements

• Architecture inspired by GPT-2, LLaMA, ViT, and BLIP-2
• Built with Hugging Face transformers
• RoPE: RoFormer: Enhanced Transformer with Rotary Position Embedding

---

StellarAI — Exploring AI, one star at a time.