ax1-pip5

AX1-Pip5 (ax1-pip5)

<img src="https://ax0rz0.xyz/pip.gif" alt="Pip" height="45"> **A ~187M-parameter conversational language model built entirely from scratch** — custom architecture (**AX1**), custom tokenizer (**AX0Tok**), and a hand-written autodiff lineage. **No PyTorch/TensorFlow, no pretrained weights, nothing pulled from another model, safe to say it's relatively sovereign.** It was pretrained on public-domain books and fine-tuned into a small, friendly chat persona named **Pip**.

---

TL;DR

• Params: 187.3M (187,272,960)
• Architecture: AX1 "phasor retention" (a proprietary attention-free sequence mixer)
• Tokenizer: AX0Tok — byte-level BPE, 8,192 vocab, custom chat control tokens
• Context: 256 tokens
• Trained: from scratch on an Apple M5 GPU via MLX; runs inference on pure Python + NumPy, CPU-only (Windows / Linux / macOS, no GPU required)
• Good at: greetings, small talk, its own identity, simple jokes, light emotional support, very simple Python, honestly saying "I don't know"
• Not for: facts, math, long reasoning, or anything safety-critical — it's tiny

---

Model details

The AX1 architecture (in brief)

AX1 replaces softmax attention with a per-head complex decay ("phasor") retention: each head weights past tokens by (q·k/√d) · r^(t−s) · cos(θ·(t−s)). Heads with θ=0 act as smooth memory channels; heads with θ>0 are resonators that track periodic structure. Combined with a learnable per-channel token-shift (blending each token with the previous one) and a swish-gated output, this gives an attention-free mixer with an O(1)-state recurrent form for streaming inference and unbounded context at generation time. There are no positional embeddings.

Tokenizer (AX0Tok)

A custom byte-level BPE tokenizer (8,192 merges) with a small private chat protocol built on three control tokens: <user>, <pip>, <end>. A turn is encoded as <user> {message} <pip> {reply} <end>.

---

Hardware

1x M5 Pro, 48GB Unified Memory (15 Core CPU & 16 Core GPU) 1x Samsung T7 Shield (1TB) External SSD

Intended uses & limitations

Intended: a research/educational demonstration of a genuinely from-scratch small language model, and a lightweight local chatbot for casual conversation on CPU.

Out of scope: factual question answering, arithmetic, code beyond trivial snippets, multi-step reasoning, and any production or safety-critical use.

Limitations:

• At 187M params it hallucinates facts and is weak at math and reasoning.
• Short 256-token context and limited multi-turn coherence.
• English only.
• Pretraining used pre-1929 public-domain texts, so it can produce archaic or dated/biased phrasing.
• Best sampled at low temperature (~0.3–0.4).

---

How to use

This is not a 🤗 Transformers checkpoint — it uses a custom NumPy runtime and the AX0Tok tokenizer, so it will not load via AutoModelForCausalLM. Inference is done with the included (pure-Python + NumPy) runtime.

pip install -r requirements.txt      # just numpy
python run.py                        # terminal chat with pip5.npz
python run.py --temp 0.3             # steadier replies

For a browser UI (live model-switcher, token streaming), see https://github.com/ax0rz0/pip-webui.

Chat prompt format (handled automatically by the runtime):

<user> hello <pip>

Files: pip5.npz (weights + tokenizer), plus the minimal NumPy inference runtime (run.py, infer.py, recurrent.py, ax0.py, engine.py, tokenizer.py, optim.py).

---

Training

From scratch, two phases:

1. Pretrain (language): ~7,949 public-domain Project Gutenberg books (~975M tokens). Held-out cross-entropy 9.5 → 2.844.
2. Fine-tune (persona): a chat + book-replay mix (~38% synthetic chat dialogues, including targeted capability/follow-up packs; rest is book replay to avoid catastrophic forgetting). Final chat-corpus val ≈ 0.073.

Setup: custom AX1 model in Apple MLX (M5 GPU); AdamW, cosine LR schedule, block 256, batch 32. The MLX weights are converted to a portable NumPy format for CPU-only inference (logits match to ~5e-4).

Evaluation

Evaluated by held-out cross-entropy (no standard NLP benchmarks — the model is deliberately tiny and uses a custom tokenizer, so public leaderboards aren't meaningful). Qualitatively: reliable on greetings, identity, jokes, light emotional support, and simple Python; unreliable on facts, math, and long reasoning. Relative to Pip4.2 (139M), Pip5 is noticeably more coherent on multi-turn, contextual follow-ups.

Bias, risks, and limitations

Trained on old public-domain literature plus synthetic dialogue; outputs may be factually wrong, archaic, or reflect biases in 19th/early-20th-century texts. Do not rely on its outputs for factual, medical, legal, financial, or safety-critical decisions.

Author's Note

The name 'AX0' & 'AX1' were purely named after my username, ax0rz0, because it sounded cool 😇

License

Released under the GNU General Public License v3.0. Derivative works must remain GPL-3.0.

Support

If Pip is useful to you, you can help cover training compute for future versions on the cloud:

<a href="https://www.buymeacoffee.com/ax0rz0" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="45"></a>

Citation

@misc{ax1pip2026,
  title  = {AX1-Pip: a from-scratch small language model with phasor-retention},
  author = {Pip project},
  year   = {2026},
  note   = {Custom AX1 architecture and AX0Tok tokenizer; trained from scratch, no pretrained weights.}
}