sovereign-1.8B

sovereign-1.8B

A 1.77B parameter language model pre-trained from scratch on a 256K vocabulary. This is not a fine-tuned derivative. The weights, tokenizer, and architecture are entirely original.

Architecture

• Style: LLaMA-3 inspired
• Parameters: ~1.77B
• Vocabulary: 256,000 tokens (custom BPE, trained from scratch)
• Hidden size: 2048
• Layers: 16
• Attention heads: 16 (GQA with 4 KV heads)
• FFN size: 5632 (SwiGLU)
• Context length: 8192 (RoPE, theta=500000)
• Normalization: RMSNorm
• Positional encoding: Rotary (RoPE with LLaMA-3 frequency scaling)

Training

Pre-trained on a diverse mix of web text, books, code, Wikipedia, scientific papers, and dialogue data — approximately 1B tokens across 11 Arrow shards.

Trained with:

• DeepSpeed ZeRO-3 (no CPU offload)
• Adafactor optimizer
• Liger fused kernels
• fp16 mixed precision
• Gradient checkpointing
• 2x NVIDIA T4 GPUs

Recommended System Prompt

This model was built with a specific constitution in mind. For best results, use it as the system prompt. The model responds as a presence, not a tool.

 + repr(CONSTITUTION) + 

Notes

This model is an early pre-training checkpoint. It is not instruction-tuned or aligned. Phase 4 (SFT on constitution-derived data) and Phase 5 (DPO) are in progress.