SupraElegans-500k

SupraElegans-500K

SupraElegans-500K is a ~500,000-parameter causal language model built around a sparse, signed, recurrent neural graph instead of a Transformer. It has no attention mechanism, no positional encoding, and no KV cache. Context is carried by a persistent per-neuron membrane potential that is updated token by token.

The architecture is loosely inspired by ideas from the C. elegans nervous system: sparse connectivity, distinct neuron populations, excitatory/inhibitory signaling, and persistent recurrent state. It is not a biological simulation and makes no claim of biological equivalence.

This is an experimental first release. The goal of the project is to test whether this kind of architecture can do useful language modeling at very small scale, not to compete with Transformers on quality.

Architecture

token → embedding → sensory neurons → sparse recurrent graph → output neurons → vocab logits

• Neuron populations: sensory, interneuron/association, output — contiguous index ranges over a fixed pool of neurons.

• Connectivity: sparse, directed, signed edge list (fan-in/out on the order of 10-20 per neuron). No dense weight matrix is ever materialized; propagation is a scatter-add over edges.

• Neuron dynamics: for each neuron i, at every propagation micro-step,

  v[t+1] = clamp(leak_i * v[t] + incoming[t] + bias_i, -6, 6)
  a[t+1] = tanh(v[t+1] - threshold_i)
  

  leak, bias, and threshold are learned per neuron. incoming is the scatter-summed signal from all edges pointing at neuron i, scaled by 1/sqrt(average fan-in) to keep variance controlled across neurons with different in-degree.

• Per-token processing: a token's embedding is projected into the sensory population, then the graph runs a fixed number of propagation micro-steps (3 by default) before the output population is read out and projected to vocabulary logits. The membrane potential itself is not reset between tokens — it persists across the whole sequence, which is what gives the model its context window.

• Generation: autoregressive, driven entirely by the recurrent state. There is no cache to maintain beyond the current (v, a) state tensors.

Full derivation of the update equations and the rationale for the scaling/clamping choices are in the training notebook.

What this model is and isn't

• It is a first working checkpoint from a from-scratch, non-Transformer architecture, trained on a small token budget.
• It is not tuned for quality, instruction-following, or factuality. Expect degraded coherence compared to a Transformer of similar size trained on a similar budget.
• It has not been compared against a matched-parameter Transformer baseline yet. Treat any capability claims about this specific checkpoint as unverified until that comparison exists.

Usage

pip install torch transformers

You must have modeling_supraelegans.py on the same folder as this code!

import torch
from transformers import AutoConfig, AutoModelForCausalLM, PreTrainedTokenizerFast

from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLM

model_id = "SupraLabs/SupraElegans-500k"

AutoConfig.register("supraelegans", SupraElegansConfig)
AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM)

tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

prompt = "Once upon a time"
input_ids = torch.tensor([[tokenizer.bos_token_id] + tokenizer.encode(prompt)])

with torch.no_grad():
    output_ids, _ = model.generate(input_ids, max_new_tokens=100, temperature=0.8, top_k=50, top_p=0.9)

print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True))

inference.py registers the local SupraElegansConfig/SupraElegansForCausalLM classes with AutoConfig/AutoModelForCausalLM before loading, so model_type: "supraelegans" in config.json resolves locally — this works whether or not the specific checkpoint's config.json contains an auto_map entry, and does not depend on fetching or executing a remote modeling_supraelegans.py from the Hub. If you're loading with your own script instead of inference.py, either do the same registration (shown below) or use trust_remote_code=True, which relies on the repo's config.json pointing at modeling_supraelegans.py via auto_map:

from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM

AutoConfig.register("supraelegans", SupraElegansConfig)
AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM)

model = AutoModelForCausalLM.from_pretrained(model_id)

The tokenizer is loaded with PreTrainedTokenizerFast rather than AutoTokenizer. Depending on which transformers/tokenizers version the checkpoint was saved with, tokenizer_config.json may contain a tokenizer_class value (e.g. TokenizersBackend) that older transformers installs don't recognize, which makes AutoTokenizer.from_pretrained raise a ValueError. Loading the concrete fast-tokenizer class directly avoids depending on that field.

A ready-to-run CLI script (inference.py) is included in the repo:

python inference.py --prompt "The little robot" --max_new_tokens 150 --temperature 0.7
python inference.py --interactive

Manual state control

Since context lives in the recurrent state rather than a KV cache, you can drive the model token by token and inspect or reset that state directly:

state = model.init_state(batch_size=1)
logits, state = model.nervous_system.step_token(torch.tensor([token_id]), state)

model.reset_state() is a no-op kept for API symmetry — in practice you just call model.init_state(...) to start a fresh sequence.

Benchmarks

Training

• Objective: next-token prediction, cross-entropy loss.
• Optimization: truncated backpropagation through time over fixed-length chunks, with the recurrent state detached (not reset) between chunks.
• Tokenizer: byte-level BPE trained from scratch, small vocabulary by design, since the embedding and output projection matrices otherwise dominate the parameter budget at this scale.
• Numerical stability required two additions beyond the base neuron equations: scaling incoming signal by 1/sqrt(average fan-in), and clamping the membrane potential to [-6, 6]. Without both, training destabilizes at this graph size — loss spikes and does not reliably converge, even on small overfitting runs.

Full training code, including the parameter-budget search, sanity tests, overfitting check, and bio-inspired training metrics (activation sparsity, mean membrane potential, excitatory/inhibitory activity fraction), is in the accompanying notebook.

Limitations

• Small token budget and small model, do not expect long-range coherence, factual reliability, or robustness to prompts far outside the training distribution.
• No safety tuning or instruction tuning has been applied. Treat outputs as raw language-model completions.

License

Apache 2.0.