FlyBrain-Pollard-VL-CNSv1

FlyBrain-Pollard-VL-CNSv1

A fruit-fly connectome used as the memory of a vision-language model. The artifact is the brain -- 117 MB of connectome, synapse signs, trained adapters and gate. It attaches to a frozen stock Qwen2-VL-2B-Instruct and is never merged into it.

This is the same brain design as FlyBrain-Pollard-CNSv1, trained against a different backbone. Not a variant, not a second architecture -- the same trainer, the same recipe, no vision-specific code anywhere. That is the point of publishing it: the method is backbone-agnostic, and this is the evidence.

Measured

A six-letter string is stated once, buried under filler, and asked about far beyond the attention window. The model never sees more than 128 tokens at a time, so it cannot see the fact and the question together. Words are generated fresh for every sample and never reused.

64 samples through pollard-brainverify, which carries the verified measurement construction. Cold start reached 100% at step 44 and held it flat across eight consecutive evaluations.

Honest scope -- read this before assuming what it does

This brain recalls TEXT exactly, on a model that can see. It does not recall images.

Exact recall works by storing token ids as bipolar signs, and a sign only has to stay on the correct side of zero to survive interference. Qwen2-VL's vision encoder is continuous -- image patches have no ids -- so there is nothing discrete to store. The brain sits on the LANGUAGE side of the model.

Because a VL model injects visual embeddings into the token stream, the brain does see image content flow past and can address on it. What it cannot do is reproduce it. Byte-exact image recall needs a backbone whose image tokenizer is discrete (VQ-style), where image tokens are real ids and the existing bit path applies unchanged. That is a backbone choice, not a brain feature.

Which file should I download?

This repo ships the brain and the backbone it attaches to, quantized by Pollard. The .pt is the memory; the .gguf is the model. You need both.

None of these is a flat preset. Pollard measures each layer's sensitivity against the model's own calibration data and spends bits where they change the output -- which is why IQ4_XS holds one layer at 5 bits and IQ3_S holds two at 4.

Read the KL column as a warning, not a score. It is the divergence from f16 when the WHOLE model is cast to that width -- the measured noise curve the allocator plans against. A mixed build does better than its own headline number, because the layers that would have cost most were never cast that low. What it tells you is where the cliff is: this model holds up well to 3-bit and collapses at 2-bit (KL ~14).

This model is notably more robust than the small text backbone in the sibling repos -- KL 0.61 at 3-bit against 2.40 for Qwen2.5-0.5B. More parameters, more redundancy to spend.

Recommended: IQ4_XS -- 910 MB for KL 0.062 is the sweet spot here. Q6_K if you want the ceiling, IQ3_S if you are memory-bound; it is still usable on this model, which is not true of the 0.5B.

This sensitivity profile is the first Pollard has measured on a vision-language model -- the toolchain could not load one until the fix that shipped alongside this release.

Use it

pip install 'pollard-weights[flybrain]'

from pollard_flybrain import FlyBrain, load_backbone
from transformers import AutoTokenizer

M = "Qwen/Qwen2-VL-2B-Instruct"
tok = AutoTokenizer.from_pretrained(M)
model = load_backbone(M, device="cuda")          # frozen; the vision tower is untouched

brain = FlyBrain.load("FlyBrain-Pollard-VL-CNSv1.pt").bind(model, tok)
brain.feed(open("long_document.txt").read())
print(brain.recall(" Question: what is the secret word? Answer: The secret word is"))

The prompt is part of the experiment. A token is filed under the words immediately before it, so a query has to reproduce that context. Ask "what is the secret word?" alone and a brain measuring 100% measures 46% -- the memory is intact, the question arrives at the wrong address. Verify any brain with pollard-brainverify, which has the correct construction built in.

Train one for YOUR model

A brain is fitted to one backbone: the address matrix has that model's hidden size (1536 here) and the token codes come from its output embedding. bind() refuses a mismatch rather than returning confident nonsense. For any other model, train your own -- it is quick:

pollard-flybrain --train 900 --model <your-hf-id> \
    --connectome graph.feather --signs signs.npy \
    --probes corpus.txt --brain MyModel-FlyBrain.pt
pollard-brainverify --brain MyModel-FlyBrain.pt --model <your-hf-id> --filler corpus.txt

pollard-brainlanes reports which runtimes can host a brain on your machine.

Also true

• It remembers; it does not reason. Language, reasoning and vision come from the backbone.
• Single-fact recall is solved; multi-fact selection is not -- several facts in one document is ~72-80% and is the open problem.
• PyTorch path. Verified lanes: transformers, MLX, GGUF/llama.cpp, EXL3. Not every runtime can host a brain, and pollard-brainlanes tells you which can on your hardware.

Connectome

MaleCNS v1.0 (FlyEM/Janelia), CC-BY. The full volume is 188,778 neurons and 26,028,386 synapses; this uses the associative-memory core -- mushroom body and central complex -- pruned to 8,552 neurons and 300,880 synapses.