Sovythos-303M-Base

<p align="center"> <img src="https://huggingface.co/my0919175/Sovythos-303M-Base/resolve/main/sovythos.png" width="320" alt="SOVYTHOS-V2"> </p> <h1 align="center">🦋 SOVYTHOS-V2</h1> <h3 align="center">Sovythos Egyptian Reasoning & Code Base Model</h3> <p align="center"> <b>A production-grade, Llama-compatible decoder-only Transformer trained entirely from scratch for Arabic, English, and Code.</b> </p> <p align="center">

</p>

⚠️ Training Status: Early-Stage Checkpoint

This checkpoint is an early snapshot from an ongoing training run — it is NOT the final SOVYTHOS-V2 release. The model is uploaded at this stage for architecture testing, integration testing (transformers/GGUF compatibility), and community feedback purposes only.

Expect at this stage:

• Weak or incoherent generations on many prompts
• Little to no instruction-following ability (this is a Base, non-instruction-tuned model on top of being early-stage)
• Validation loss still actively decreasing — quality will improve substantially with further training

Training is actively continuing. Follow this repository for updates as later checkpoints are released.

---

📋 Table of Contents

• Overview
• Model Specifications
• Architecture Details
• Tokenizer
• Training
• Try It Now (Google Colab)
• Usage
• GGUF Inference
• Intended Uses
• Limitations
• Roadmap
• Citation
• License

---

🦋 Overview

SOVYTHOS-V2 is the second-generation foundation model of the Sovythos AI Project — a fully open-source, decoder-only Transformer trained from randomly initialized weights on a multilingual corpus spanning Modern Standard Arabic, Egyptian Arabic, English, and programming languages.

This release represents a significant architectural leap from the original 66M Base model, scaling to ~300 million parameters while adopting a Llama-compatible design (RMSNorm, RoPE, GQA, SwiGLU) to ensure seamless deployment across the modern inference ecosystem, including llama.cpp, llama-cpp-python, and Hugging Face transformers.

SOVYTHOS-V2 is released as a Base (Pretrained) model. It has not undergone instruction tuning or RLHF and is intended for researchers, developers, and educators who require a sovythos, transparent foundation for downstream training.

Key Design Principle: Every component — from the tokenizer to the tensor naming convention — is engineered for direct compatibility with the Llama/GGUF toolchain. No conversion hacks required.

---

📊 Model Specifications

---

🏗️ Architecture Details

SOVYTHOS-V2 implements a modern, efficient decoder stack aligned with the Llama architecture specification for frictionless ecosystem integration.

Normalization

• RMSNorm applied pre-attention and post-attention (no bias, no learned affine shift)

Position Encoding

• RoPE (Rotary Position Embedding) with rotate_half (NEOX-style) implementation
• Base frequency θ = 1,000,000
• Precomputed cosine/sine caches up to max sequence length

Attention Mechanism

• Grouped Query Attention (GQA) with 16 query heads and 4 key-value heads
• Reduces KV-cache memory footprint by 4× during inference compared to standard MHA
• Compatible with memory-efficient attention kernels (scaled_dot_product_attention)

Feed-Forward Network

• SwiGLU activation with gated projection
• Hidden dimension rounded to nearest multiple of 256 for optimal GPU kernel utilization
• gate_proj, up_proj, down_proj naming matches Llama/GGUF conventions exactly

Weight Initialization

• Gaussian initialization (σ = 0.02) for embeddings and linear layers
• Scaled down-projection and output-projection initialization by 1/√(2×layers) to stabilize deep training

Training Optimizations

• torch.compile graph mode (when available)
• TF32 matmul acceleration on Ampere+
• Automatic Mixed Precision (AMP) with fp16 forward pass
• Gradient Checkpointing for long-context training on consumer GPUs
• Gradient Accumulation with effective batch scaling
• Cosine LR Decay with linear warmup
• Gradient Clipping (max norm = 1.0)

---

🔤 Tokenizer

SOVYTHOS-V2 ships with a custom-trained Byte-Pair Encoding (BPE) tokenizer optimized for Arabic morphology, English, and code syntax.

Design Choices

Coverage Efficiency (Tokens per Character)

Lower is better. Measured on a held-out sample of the training corpus:

The tokenizer is saved as a standard tokenizer.json (Hugging Face tokenizers format) and its merges/vocab are directly compatible with GGUF gpt2 tokenizer model type.

---

🏋️ Training

SOVYTHOS-V2 was trained entirely from scratch. No pretrained weights from Llama, GPT, or any other foundation model were used as initialization.

Data Pipeline

• Corpus: Multilingual web text, technical documentation, mathematical content, and code repositories
• Languages: Modern Standard Arabic, Egyptian Arabic, English, and multilingual programming languages
• Cleaning: Duplicate-line removal, short-line filtering (< 2 chars), NFC normalization, and aggressive deduplication of repeated contiguous lines
• Format: Binary tokenized memmap for high-throughout training with np.uint16 encoding

Training Recipe

Checkpointing & Resumption

• Checkpoints saved in Safetensors format (secure, fast, language-agnostic)
• Automatic resumption from ckpt_dir/best/ including step count and best validation loss
• Best checkpoint selection based on validation loss

Export

• Native GGUF export after training (or from existing checkpoints via --export_gguf_only)
• Supported quantizations: F32, F16 (post-hoc Q4/Q8 via llama.cpp tools)
• Tensor name map strictly follows Llama GGUF specification (blk.{i}.attn_q.weight, blk.{i}.ffn_gate.weight, etc.)

---

🚀 Try It Now (Google Colab)

The fastest way to test SOVYTHOS-V2 is directly through 🤗 transformers, using the custom model code shipped in this repository (trust_remote_code=True). No manual setup, no cloning — just run the cell below in a fresh Colab notebook.

# ============================================================
# 🔱 SOVYTHOS-V2 (303M-Base) — Colab Quickstart
# ============================================================

!pip install -q -U transformers accelerate safetensors

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "my0919175/Sovythos-303M-Base"

print("🔄 Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

print("🔄 Loading model...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.float32,
    trust_remote_code=True,
).to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

# Chat-style prompt using the built-in special tokens
prompt = "<|user|>\nمرحبا, احكيلي عن نفسك\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Note: This is an early-stage training checkpoint of a Base (non-instruction-tuned) model — not the final release. Outputs may be short, repetitive, incoherent, or diverge from the prompt entirely; this is expected at this stage of training and is not a loading error. Quality will improve as training progresses and with the upcoming Instruct release.

When prompted Do you wish to run the custom code? [y/N], type y — this loads configuration_sovythos.py and modeling_sovythos.py from this repo, which implement the GQA + RoPE + SwiGLU architecture described above.

---

💻 Usage

Loading with 🤗 Transformers (Recommended)

This repository ships with trust_remote_code-compatible files (configuration_sovythos.py, modeling_sovythos.py), so it works directly with the standard AutoModelForCausalLM / AutoTokenizer API — no need to clone the original training repo:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "my0919175/Sovythos-303M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.float32, trust_remote_code=True
).to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "<|user|>\nThe capital of Egypt is\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Loading with the Native Training Code

If you have the original training repository (model.py, train.py) available locally, you can also load the checkpoint directly through the native Model class, which supports the custom KV-cache and generation loop used during training/research:

from model import Model

model = Model.from_pretrained("sovythos_ckpt/best", device="cuda")

text = "The capital of Egypt is"
ids = tokenizer.encode(text).ids
input_ids = torch.tensor([ids], device="cuda")

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.token_to_id("<|endoftext|>")
)
print(tokenizer.decode(output[0].tolist()))

Using the Tokenizer Directly

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
encoded = tok.encode("إزيك يا كبير؟")
print(encoded.ids)  # [ ... ]
print(tok.decode(encoded.ids))  # "إزيك يا كبير؟"

---

🚀 GGUF Inference

Because SOVYTHOS-V2 uses Llama-compatible tensor naming and a GPT-2-format tokenizer, the exported GGUF file runs out-of-the-box with llama.cpp and llama-cpp-python.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="sovythos-v2.f16.gguf",
    n_ctx=2048,
    n_gpu_layers=-1,  # Offload all layers to GPU
    verbose=False
)

output = llm(
    "The capital of Egypt is",
    max_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    stop=["<|endoftext|>"]
)
print(output["choices"][0]["text"])

llama.cpp CLI

./main -m sovythos-v2.f16.gguf        -n 50        --temp 0.8        --top-k 40        --top-p 0.9        -p "The capital of Egypt is"

No conversion scripts required. The GGUF file is produced directly by the training framework with correct llama arch metadata.

---

🎯 Intended Uses

SOVYTHOS-V2 is designed as a research and development foundation for:

• Continued pretraining on domain-specific corpora
• Supervised fine-tuning (SFT) for chat or task-specific behavior
• Instruction tuning and alignment experiments
• Educational exploration of modern decoder-only architectures
• Arabic NLP research (MSA, Egyptian dialect, code-switching)
• Low-resource and Sovythos AI development

Not intended for: Direct deployment as a production chatbot without additional safety fine-tuning and evaluation.

---

⚠️ Limitations

As a Base (pretrained) model, SOVYTHOS-V2 exhibits expected limitations:

• No instruction following without fine-tuning
• Hallucination of factual information (names, dates, technical details)
• Repetition in long-form generation
• Incomplete reasoning on complex multi-step problems
• Limited context coherence beyond a few hundred tokens in zero-shot settings
• Code generation produces syntactically valid structures but may contain logic errors

These limitations are standard for base models of this scale and will be addressed in future Instruct/Chat releases.

---

🛣️ Roadmap

---

🌌 Sovythos AI Project

SOVYTHOS-V2 is the flagship foundation model of the Sovythos AI Project, an open ecosystem dedicated to building transparent, multilingual language models with native Arabic support and modern reasoning capabilities.

The project prioritizes:

• Sovereignty: Full training pipeline ownership (data → tokenizer → model → export)
• Transparency: Open weights, open code, open training logs
• Accessibility: GGUF compatibility for edge and consumer hardware
• Community: Research-first releases with clear limitation disclosures

---

📖 Citation

If you use SOVYTHOS-V2 in your research, please cite:

@misc{sovythosv2,
  title={SOVYTHOS-V2: Sovythos Egyptian Reasoning and Code Base Model},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/my0919175/Sovythos-303M-Base}}
}

---

👨‍💻 Creator

Mahmoud Yasser

Founder of the Sovythos AI Project

---

📄 License

Apache License 2.0

---

<p align="center">

🦋 Welcome to the next chapter of the Sovythos ecosystem.

This is an early, actively-training checkpoint shared in the spirit of open, transparent development — not a finished product. Researchers, developers, and builders who believe in open, sovereign AI are welcome to follow along as training progresses.

Future checkpoints will bring lower loss, better coherence, instruction tuning, and long-context support.

⭐ If you find this project useful, please consider following the repository and sharing your experiments.

</p>