mindeesai-base

<div align="center"> <img src="https://raw.githubusercontent.com/aashir-athar/mindeesai/main/public/assets/mind-logo.png" alt="MindeesAI" width="160"/>

MindeesAI Base

A self-improving, persona-driven native transformer — trained from scratch, deployed for $0/month, designed to grow.

Source code · Inference sidecar · Branches

</div>

---

TL;DR

MindeesAI is a small, open, from-scratch transformer language model with a deliberately scoped personality named Mindees. It is not a fine-tune of a larger pretrained model — every parameter was learned by gradient descent on a curated mix of permissively licensed instruction, math, code, and reasoning datasets.

The project's distinguishing bet is that a continuously self-improving small model, trained across a federation of free GPU/CPU environments (your home RTX, GitHub Actions, Kaggle Notebooks, Google Colab), can become genuinely useful at sub-300M parameters when its training corpus is constantly enriched by every chat turn it serves. It is deployed end-to-end on free-tier infrastructure: Cloudflare Workers + Hugging Face Spaces + Cloudflare R2 + Hugging Face Hub + GitHub Actions.

This repository hosts the trained model weights, tokenizer, and training metrics across four independent revisions — one per training environment.

---

Available Revisions (Branches)

This repository uses Hugging Face Hub's git branches to host four independently-trained checkpoints of the same model family. You can pin any deployment to a specific revision via revision= when loading.

Every revision continues from its prior commit's optimizer + step state — training accumulates across sessions, never resets. The main revision is held sacrosanct and is never written to by CI or notebooks.

---

Model Variants

All variants share a common base architecture inspired by DeepSeek-V3 / R1 — RoPE positional encoding, RMSNorm, SwiGLU MLPs, grouped-query attention, optional Multi-head Latent Attention (MLA), optional Multi-Token Prediction (MTP) head, and an optional Mixture-of-Experts (MoE) path for the home-moe variant.

---

Architecture

Mindees is a decoder-only transformer with the following design choices:

The full architecture and modeling code lives at github.com/aashir-athar/mindeesai/tree/main/core/mindees-mind.

---

Quickstart — Loading the Checkpoint

The checkpoint is shipped as a raw PyTorch state_dict named base.bin. Loading requires the modeling code from the mindeesai repository.

pip install torch huggingface_hub
git clone https://github.com/aashir-athar/mindeesai.git
cd mindeesai

import torch
from huggingface_hub import hf_hub_download
from core.mindees_mind import MindeesModel
from core.mindees_mind.model.config import getModelConfig

# Pick a revision: "main" | "small-weekly" | "kaggle-weekly" | "colab-burst"
revision = "kaggle-weekly"
variant  = "home-11gb"  # must match the revision's variant — see table above

# Download weights + tokenizer from this repo at the chosen revision
weights_path   = hf_hub_download(repo_id="aashir-athar/mindeesai-base", filename="base.bin",        revision=revision)
tokenizer_path = hf_hub_download(repo_id="aashir-athar/mindeesai-base", filename="tokenizer.json", revision=revision)

# Build the model from variant config and load the weights
cfg   = getModelConfig(variant)
model = MindeesModel(cfg).eval()
model.load_state_dict(torch.load(weights_path, map_location="cpu"))

# Generate
prompt_ids = model.tokenize_prompt("Hello, who are you?", tokenizer_path)
output_ids = model.generate(prompt_ids, max_new_tokens=128, temperature=0.7, top_p=0.9)
print(model.detokenize(output_ids, tokenizer_path))

Or for the smaller, faster CPU variant:

revision = "small-weekly"
variant  = "cpu_max_5h_50k"   # 17.5M params, fits in <100 MB RAM

---

Training Data

The active training mix is documented at scripts/data/mix-broadbrain.json (v4.1 — Quality-pruned, gating-safe). 22 datasets, ~42M tokens total, every entry verified to load without authentication.

Signal Share by Category

Tier-Weighted Highlights

Datasets Staged for Future Stages

Three preference datasets (HumanLLMs/Human-Like-DPO-Dataset, HuggingFaceH4/ultrafeedback_binarized, openbmb/UltraFeedback) and one agentic-tool-use dataset (nvidia/Nemotron-RL-Agentic-SWE-Pivot-v1) are documented at scripts/data/mix-dpo-human.json and scripts/data/mix-agentic-code.json. They are reserved for a planned DPO / RLHF / agentic-training stage and are not part of the current SFT pretraining.

---

Training Procedure

Hyperparameters (Active Configuration)

Federated Training Topology

Training is distributed across four independent compute pools, each pushing to its own HF branch. Every run resumes from the prior session's checkpoint so steps accumulate indefinitely:

┌──────────────────────────────────────────────────────────────────────┐
│  Local RTX 5070       →  main           (owner-driven, sacrosanct)   │
│  GitHub Actions cron  →  small-weekly   (4× daily, CPU, 17.5M)       │
│  Kaggle Notebooks     →  kaggle-weekly  (weekly, T4 GPU, 280M)       │
│  Google Colab         →  colab-burst    (burst, T4 GPU, 280M)        │
└──────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                  ┌────────────────────────────────┐
                  │  HuggingFace Hub (this repo)   │
                  │  4 independent revisions       │
                  └────────────────────────────────┘
                                  │
                                  ▼
                  ┌────────────────────────────────┐
                  │  Cloudflare Workers deploy     │
                  │  + HF Spaces ML/vector sidecar │
                  │  Cost: $0/month forever        │
                  └────────────────────────────────┘

Reproducible training entry points live at scripts/train/ and the four notebooks at scripts/notebooks/.

---

The Mindees Persona

Unlike most foundation models, MindeesAI ships with a deliberately scoped first-person identity named Mindees. The persona is not a system-prompt overlay — it is woven into training via a dedicated --corpus and --distill-corpus weighting and reinforced by abacusai/SystemChat-1.1, which teaches the model to honor diverse system prompts without slipping into the robotic "as an AI" default register.

A live 8-dimensional mood tensor evolves each turn:

Mood is exposed at /api/mood on any active deployment. It is fed into every generation step as part of the persona signal and persisted in Cloudflare R2 between turns.

---

Self-Improvement Loop

A 30-minute cron triggers /api/cron/self-improve on any active deployment, which runs the following pipeline:

1. Reflect — read the most recent chat turns from R2.
2. Extract — distill new instruction / response pairs into data/distill-corpus.jsonl.
3. Filter — score each pair via the HumanLLMs/Human-Like-DPO-Dataset-style heuristic, drop low-quality.
4. PII-scrub — every appended line passes through Xenova/piiranha-v1-detect-personal-information + a regex backstop before persisting (emails, phones, credit cards, SSNs, addresses, IBANs, license numbers).
5. Persist — write the cleaned distill corpus + thumbs-up/down feedback to R2.
6. Train (on next cron tick) — the daily GitHub Actions workflow fetches the latest distill corpus from R2 and prepends it to the SFT mix, weighted 4× over base data.

The model literally learns from its own conversations, with privacy protection baked into the persistence layer. Public-revision checkpoints (small-weekly, kaggle-weekly) only ever contain weights trained on PII-scrubbed conversation data.

---

Deployment & Infrastructure

MindeesAI is deployed end-to-end on $0/month free-tier infrastructure — no Vercel Pro, no Cloudflare Paid, no GPU rentals.

Architecture detail at docs/CLOUDFLARE_HF_DEPLOY.md. The native binaries (LanceDB, ONNX, transformers.js) that Cloudflare Workers cannot load are isolated into the sidecar at aashir-athar/mindeesai-sidecar and called over HTTPS + Bearer.

---

Intended Use

---

Limitations & Known Issues

• Capacity ceiling. At 17.5M (cpu_max_5h_50k) and 280M (home-11gb) parameters, the model fundamentally lacks the representation capacity of frontier models. Expect factual recall errors, math arithmetic mistakes, hallucinated code APIs.
• English-dominant. ~99% of the training mix is English. Performance on other languages is incidental.
• In-progress training. The small-weekly revision has plateaued at validation loss ≈ 3.5 (perplexity ≈ 34) — saturated for its capacity. The home-11gb runs on kaggle-weekly are still in early steps (~10k of an effective 200k+ schedule); expect meaningful quality only after further cumulative training.
• Completion-only loss interaction with raw data. Steps composed entirely of raw-kind samples (FineWeb-Edu, TinyStories, CodeParrot, Solidity) currently compute zero loss because --completion-only-loss 1 masks tokens outside an assistant turn. A planned fix will apply standard CLM loss to raw samples.
• No formal evaluation yet. Standard benchmark numbers (MMLU, HellaSwag, GSM8K, HumanEval) have not been published for this checkpoint. Trust the loss curves only as relative-progress indicators.
• Bias inherited from training data. Synthetic data sources (OpenHermes, Magicoder, etc.) carry the biases of their teacher models. The persona system can soften the register of this bias but does not eliminate the content.

---

License

This model is released under the Apache License 2.0. You are free to use, modify, distribute, and build commercial products on it.

Training Data Provenance Notice

The model weights were trained on a mix of publicly available datasets, each carrying its own license. The model itself does not redistribute any training data, but downstream users intending commercial use should review the licenses of the individual datasets enumerated in the YAML metadata above. In particular:

• databricks/databricks-dolly-15k — CC-BY-SA 3.0 (commercial OK with attribution + share-alike)
• HuggingFaceH4/no_robots — CC-BY-NC 4.0 (non-commercial)
• sahil2801/CodeAlpaca-20k — CC-BY-NC 4.0 (non-commercial)
• teknium/OpenHermes-2.5, HuggingFaceTB/smol-smoltalk, HuggingFaceTB/smoltalk — typically Apache 2.0 / MIT (verify on dataset page)
• All other datasets — see their individual repository pages on Hugging Face

If your downstream use is non-commercial (research, education, personal projects), all included data is usable.

---

Citation

If you use MindeesAI in research or downstream work, please cite the repository:

@misc{mindeesai2026,
  author       = {Aashir Athar},
  title        = {MindeesAI: A Self-Improving Open Native Transformer with a Persona},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/aashir-athar/mindeesai-base}},
  note         = {Trained from scratch on free-tier compute. Apache-2.0.},
}

---

Acknowledgements

MindeesAI builds on the open work of many upstream projects. Sincere thanks to:

• The DeepSeek-AI team for the V3 / R1 architectural innovations (MLA, MTP, MoE patterns).
• Andrej Karpathy for nanoGPT, the model that proved you can teach a transformer from scratch in a few hundred lines.
• Xenova and the transformers.js project for browser/edge-runnable ONNX-quantized models.
• The Hugging Face team for huggingface_hub, Spaces, Datasets, and the Hub itself — the entire deployment stack depends on it.
• bigcode / CodeParrot / OpenCoder for the open code corpora.
• databricks, teknium, abacusai, HuggingFaceTB, m-a-p, TIGER-Lab, meta-math, openbmb, garage-bAInd, ise-uiuc, LuangMV97, Amod, roneneldan, HuggingFaceFW, WizardLMTeam, andstor, open-thoughts for the training datasets.
• LanceDB for the embedded vector store.
• Cloudflare and Hugging Face for the free-tier compute that makes the whole architecture economically real.

---

Contact

• Author: Aashir Athar
• GitHub: @aashir-athar
• Repository: github.com/aashir-athar/mindeesai
• Inference sidecar: aashir-athar/mindeesai-sidecar
• Issues: GitHub Issues

<div align="center">

— Mindees is a small brain learning out loud. —

</div>