Acid-GSN-Architecture

GSN — Gated Sparse Network

Acid Research (ACRS)

GSN is a language model architecture designed around one principle: do not spend compute you do not need.

Unlike transformers — which run the full model on every input regardless of complexity — GSN gates compute dynamically. Simple inputs take a shallow path. Complex inputs go deeper. The network decides, not the configuration.

---

Architecture

GSN is built from three core ideas stacked together:

1. Gated Depth A lightweight gate network evaluates each input and decides how many layers to activate. Layers that are not needed do not run. Their compute cost is exactly zero.

2. Sparse Activation Within each active layer, only the top-k neurons fire. The rest are masked to zero. A 4-layer GSN running at 25% sparsity uses a fraction of the compute a dense model would.

3. Recurrent Encoder Input tokens are processed sequentially through a GRU encoder before the gate sees anything. This replaces mean pooling — token order matters, context accumulates, and the gate receives a hidden state that actually encodes sequence structure.

4. Compute Penalty in Training The loss function penalizes wasted compute. The model is trained to be cheap, not just accurate. Over time the gate learns: if I can answer correctly with one layer, using two is a mistake.

No attention. No transformers. O(T) encoding. O(1) per sparse layer.

---

Why GSN

GSN was developed and trained entirely on a consumer CPU with 2 physical cores. No GPU. No cloud compute. That is not a limitation — it is the point.

---

Usage

Requirements

pip install numpy

No PyTorch. No CUDA. No framework dependency. Pure NumPy.

Training

python train.py your_corpus.txt 20000

Point it at any plain text file. The tokenizer trains from scratch on your corpus. Checkpoints save every 500 steps. Resume is automatic.

Inference

python infer.py "your prompt here" --max_new 100

The inference report shows complexity score, gate depth decision, and compute saved per generation.

Configuration

All hyperparameters live in config.py. Key settings:

GSNConfig(
    vocab_size      = 1024,   # BPE vocabulary size
    dim             = 256,    # embedding and hidden dimension
    enc_dim         = 256,    # GRU encoder hidden dimension
    n_layers        = 4,      # total sparse layers available
    compute_penalty = 0.001,  # λ — weight of compute cost in loss
    lr              = 3e-4,   # learning rate
    seq_len         = 128,    # context length
)

---

Training Details

Corpus: Shakespeare complete works (~1.1M characters)
Vocabulary: 1024 BPE tokens
Steps: 20,000
Hardware: 2 physical CPU cores (Debian Linux)
Training time: ~25 minutes
Final loss: ~3.5
Average gate depth: 1.44 / 4 layers
Average compute saved: ~88%

The gate learned that Shakespeare — structured, repetitive, consistent vocabulary — is mostly shallow complexity. On a more diverse corpus the gate is expected to show greater depth variation.

---

Repository Structure

config.py       — all hyperparameters
tokenizer.py    — BPE tokenizer, trains from scratch
model.py        — GSN model, forward pass, analytical backprop
train.py        — training loop, Adam optimizer, checkpointing
infer.py        — inference CLI with compute report

---

Limitations

• Undertrained on small corpus. 20k steps on Shakespeare is a proof of concept. Coherent generation requires significantly more training on a larger and more diverse corpus.
• Small vocabulary. 1024 tokens is minimal. Real deployments should use 8k-32k.
• No pretrained weights included. This release is an architecture and training framework, not a ready-to-use model. Train your own.
• Gate behavior is corpus-dependent. The gate learns complexity relative to the training distribution. A model trained on Shakespeare will gate differently than one trained on code or web text.
• Single-sequence inference only. Batched inference is not yet implemented.

---

Known Issues and Community Contributions Welcome

• Batched inference
• Larger vocabulary and longer context experiments
• Perplexity benchmarking against RSM and small transformers
• Gate visualization tooling
• Training on FineWeb, OpenWebText, or code corpora

This is an incomplete release by design. The architecture is sound. The community is invited to take it further.

---

License

Licensed under the Acid Research Protected Interests License (APRIL) v1.0.

• Free for personal, academic, and non-commercial use.
• Derivatives must be open sourced under APRIL.
• Commercial use requires written permission from ACRS.
• Attribution to Acid Research (ACRS) is required in all derivatives.

See LICENSE for full terms.

---

Citation

If you use GSN in research or build on this architecture, please cite:

@misc{gsn2025,
  title  = {GSN: Gated Sparse Network},
  author = {Acid Research (ACRS)},
  year   = {2025},
  url    = {https://huggingface.co/AcidAI/Acid-GSN-Architecture}
}

---

About Acid Research

Acid Research (ACRS) is an independent AI research organization building CPU-native, economically viable alternatives to transformer-based architectures.

Current architecture portfolio:

• HAM — Hebbian Architecture Model
• RSM — Recurrent State Machine
• RDM — Recurrent Depth Machine
• IMA — Intent Machine Architecture
• GSN — Gated Sparse Network

"Make it linear, or else your cost ain't going to be."