nanofly-decoder-ru

nanofly-decoder-ru

_{43,993 neurons at their measured MaleCNS v1.0 coordinates, coloured by this checkpoint's state at one tick while continuing the prompt «Сегодня утром». Orange excited, blue inhibited, grey at rest. Frontal view; the optic lobes flank the central brain.}

A Russian language model whose recurrent layer is the measured wiring of a fruit fly. The connectome is a frozen echo state network reservoir — no synapse is trained. Only the input projection, per-neuron gain/bias/leak, one global scale and the readout learn.

Non-commercial. The training data (DaruLM) permits scientific, non-commercial use only. That restriction travels with these weights.

Unfiltered. No toxicity or profanity filtering at any stage. It emits Russian obscenity unprompted. Do not put it in front of users without a filter.

• Code: github.com/igorktech/nanoFLY
• English sibling: igorktech/nanofly-decoder-en

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "igorktech/nanofly-decoder-ru"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()
model = model.to("cuda" if torch.cuda.is_available() else "cpu")

ids = tok("Сегодня утром", return_tensors="pt").input_ids
ids = torch.cat([torch.tensor([[model.config.bos_token_id]]), ids], dim=1).to(model.device)
out = model.generate(ids, max_new_tokens=80, do_sample=True, top_k=50, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

• Sample, do not decode greedily — greedy falls into repetition loops within a sentence or two.
• Prepend BOS: every training example started with it.
• Beam search and assisted generation are unsupported (stateful model).
• ~960 forward passes/s on an RTX 5080, ~10/s on a laptop CPU.

Architecture

Training

Evaluation

Perplexities across different tokenizers are not comparable — this model's vocabulary is 4× larger and 3.04 characters per token. Bits per character is the fair axis, and there the gap is under 2×, not 6×. The corpora also differ in difficulty: open-domain web Russian against a deliberately closed and repetitive TinyStories. Validation fell 98.1 → 42.0 over 23 evaluations and was still improving at the end; the checkpoint is undertrained.

No shuffled-wiring control has been run for this model (the English one has: 1.933 real vs 1.979 degree-matched shuffle).

Samples, top-k 50, temperature 0.7, prompt in bold:

По данным синоптиков, в городе Мой биологи в регионе было обнаружено в одном городе и блинском городе Уфе. Об этом сообщает пресс-служба столичных регионах страны.

Вчера вечером я решил подробно настроить на сайтах: — Чувак, которые я вам не сижу на пикабу сижу, что я хочу поделиться с =)

Morphology, short-range agreement and register are learned — the first is recognisably newswire down to the "Об этом сообщает пресс-служба" formula, the second recognisably a Pikabu post. Meaning is not.

Limitations

• 17.8M trainable parameters over 163M token-steps of web Russian. Fluent-looking Russian that does not mean anything.
• Greedy decoding degenerates into loops. Sampling is required.
• 8-token delay line plus a short leaky recurrent memory; it cannot hold a subject across a sentence.
• Unfiltered Pikabu, Lenta and Gazeta: obscenity, the biases of that data, and a mid-2010s news skew. DaruLM flags itself not-for-all-audiences and notes its domain splits are noisy.
• A tanh rate neuron is not a spiking model: no spikes, no synaptic delays, no neuromodulation — modulatory edges are removed outright.
• Central brain only; the optic lobes and ventral nerve cord of the 166,700-neuron CNS are absent.
• Synapse count is a proxy for strength, and rows are normalised. Neither is measured physiology.

Credits

• Connectome: MaleCNS v1.0 — FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0. The published buffers derive from that release; keep the attribution when redistributing.
• Data: DaruLM by dichspace, from corpora collected by Ilya Gusev. Scientific, non-commercial use only — the same restriction applies to these weights.
• Transmitter signs: Shiu et al., Nature 2024.
• Connectome as reservoir: Costi, Hadjiivanov, Dold, Hale, Izzo, 2025.
• Prior art: ngxson/fly-llm-hf, whose graph subset this reproduces.

The licence tag is other, not cc-by-4.0: the connectome would allow CC BY, the training data does not permit commercial use, and the stricter term governs. Modeling code Apache-2.0.