mixture-of-tiny-sliding-experts

Mixture of Tiny Sliding Experts

One frozen public checkpoint is loaded at a time on a 24 GB GPU. A hash router picks it. Optional chains pass text from one expert to the next. A bag of those chains votes on the final string. A separate 128-dimensional cache, shared only by the routers, can slide along the chain.

This repository is an assembly of published methods. It does not introduce a new routing, memory, or decoding algorithm. The bibliography at the bottom is the full set of ideas used in the code. Upstream weight files keep their own licenses and are not redistributed here.

What the system does

A request is hashed into a bag of word ids. A small classifier scores 138 domain cards. Those cards are not 138 models. They point at 17 checkpoint ids (eight code checkpoints plus the modality rows). Confidence for escalation sums every card that shares a checkpoint and a backend, so splitting one model into many cards does not by itself look like low confidence. Under that aggregated confidence the route selects the int4 14B coder.

Creating an image is a separate decision from editing image-processing code, explaining a Docker image, or captioning pixels. Only a create-intent loads diffusion. Captioning requires image pixels through a vision adapter. StarCoder2 is prompted as a base fill-in-the-middle model. SQLCoder receives an explicit schema block. Instruction models use the tokenizer chat template when it exists. Generation is one forward under inference mode. A stored past_key_values prefix is not reused, because the old path prefilled the prefix and then generated from the full prompt anyway.

chain keeps the original request in later prompts. The previous model's text is a candidate, not a replacement task. Hop routers still see that candidate when they choose the next card. A verifier or a FINISH line can stop the chain before the hop cap. Named use cases fix the kinds: diagram, label-then-explain, draft-then-illustrate, and match-then-redraw. A bag can draw distinct experts and can prefer candidates that pass an external check. The vote share is not a calibrated probability.

The public checkpoints stay frozen. Training covers the router and the external cache. The cache's learned shift is implemented and tested. On a synthetic history task it did not beat plain text history, so a cache loaded for inference leaves the shift off unless MOTE_SHIFT=1.

Why this exists

A 24 GB laptop can run one 7B checkpoint. It cannot keep dozens of specialists resident at once. The useful split is which model to load, not a new layer inside one model. The router makes that choice from the request text. Chains exist because some requests need two kinds of model in order: draw, then read the drawing; label, then explain the label. Each hop writes text (or a PNG path) into a shared note the next expert can prefill. That note is ordinary text. Heterogeneous checkpoints do not share a native key-value cache, so the slide cache is a separate 128-wide ring used only by the routers.

The slide stores where a memory layer sits on the chain. That position is a workflow slot (model A, then B, then C), not an expert id and not a slot inside one ring. Training and inference call the same read: layer vectors are soft-assigned onto those workflow positions, a position code is added, the positions are mixed, and a position-index residual keeps the mix from erasing the move. The slot policy is a query-key dot product, q(x) · (key_i + embed_i) / sqrt(dim), which is one dim×dim map plus a dot product per slot. Writes decay by 0.5 so a long session stays bounded. PPO uses the same label reward as REINFORCE, with a learned value and a clipped ratio (Schulman et al., 2017). A fixed slot is the policy baseline. None of this updates the frozen expert.

Labels for hop pairs come from catalog interests that appear as whole words. The hash is only the router's feature. Two strings that collide in that feature keep one label. A missing router or cache file is an error. --demo is the explicit untrained path.

python -m pytest tests -q
python -m mote.train_router
python -m mote.train_chain_routers
python -m mote.train_exkv
python -m mote.benchmark
python -m mote --map
python -m mote "fix the pytest" --root .
python -m mote "fix the pytest" --root . --demo

pytest is the CPU suite. It does not need checkpoints/router.pt. train_router writes that file. train_exkv fits the cache, freezes it, fits router injection layers, then saves checkpoints/exkv.pt and checkpoints/bundle.pt together. benchmark writes artifacts/benchmark.json. It does not download specialists. GPU generation is python -m mote "..." --root . --generate after the router exists. vLLM, diffusers, and transformers are optional extras in pyproject.toml (hf, vllm). Set PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 if a local plugin crashes collection.

Download and generate

Install torch, transformers, and huggingface_hub. Then:

python -m mote download --variant tiny --output ./models/tiny
python -m mote generate --variant ./models/tiny --prompt "Write a Python function that validates an email address" --max-new-tokens 512 --offline
python -m mote chat --variant ./models/tiny --offline

from mote import MoTEPipeline

pipeline = MoTEPipeline.from_pretrained("./models/tiny", device="auto", local_files_only=True)
print(pipeline.generate("Explain and fix this SQL query: select * users", max_new_tokens=512).text)

tiny is the small local variant. gpu-8gb and laptop-24gb stay manifests until --weights. A finished download stores each expert's upstream model card (README.md) next to its weights when that file exists, and generation records the Hugging Face model-card URL. The full list is below and in docs/CHECKPOINTS.md. Install, reset, and offline errors are in docs/DOWNLOAD.md. The explanation of the system is PAPER.md.

Checkpoints and their model cards

Each row is a checkpoint decoding can load. The link is that checkpoint's model card on Hugging Face. This project does not rehost those weights or those cards.

Four pools

The downloadable specialists are nested: 6, then 18, then 24, then 36. Each larger pool contains the smaller ones. Every pool has text, an image generator, and a classifier. From 18 upward, each pool also has speech and OCR. The list and the card text for each cut are in mote/variants.py. A card for the 6, 18, or 24 cut still names the checkpoints that appear only in the full 36, because the router and the stacked cache are the same system.

Where each idea is used

Two papers also use the name Chain-of-Experts and are not this system. Xiao et al. (2024) assign roles to separate LLM agents for operations-research problems. Wang et al. (2025) pass a token through experts inside one MoE layer. This repo chains separate checkpoints by text, outside the layer.

Finetuned weights

Training in this project produced six checkpoints. The public specialist models were not finetuned here, and their weight files are not in this repo. At decode time the loader fetches the chosen specialist from its original Hub id.

Decoder

Decoding is one greedy generation from the resident expert, or a seeded diffusion call when the intent is to create an image. The shared slide cache is added to the router. It is not the expert's past_key_values, and those private tensors are not copied across checkpoints. Text memory is rendered into the prompt. Recent notes are kept when the character budget is tight. See docs/DECODER.md and docs/REPORT.md.

Clone this repo so checkpoints/ sits next to the mote package. Route only, with no expert download:

python -m mote "fix the pytest" --root .

Route, load the one selected checkpoint, and decode:

python -m mote "fix the pytest" --root . --generate

FileAgent loads checkpoints/router.pt, checkpoints/exkv.pt, and checkpoints/routers/after-*.pt on startup. Step-by-step notes are in docs/DECODER.md.

Tests

python -m pytest tests -q

tests/test_exkv.py checks residual cache writes, router injection, REINFORCE, stack gradients, and the slide from model A onto B or C. tests/test_iteration.py checks train/inference memory equivalence, query-dependent slots, image intent, symlink containment, adapters, and checkpoint errors. tests/test_chain.py, tests/test_bag.py, tests/test_cascade.py, and tests/test_usecases.py check the text handoff, the vote, the image cascade, and the named use cases. tests/test_experts.py checks cue routing when checkpoints/router.pt is present and skips otherwise. These tests do not download the 7B checkpoints.

The CPU history ablation in artifacts/benchmark.json is the measurement of sliding memory. Text history scored 1.0 on both seeds. Learned shifts scored about 0.96 and did not win. That is why inference loads the cache with the shift off. It is not a code-generation result. RouteLLM-style win-rate routing was not fit here; the bibliography names it as related work.

Bibliography

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. arXiv:1701.06538.

Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. arXiv:2101.03961.

Weinberger, K., Dasgupta, A., Langford, J., Smola, A., and Attenberg, J. Feature hashing for large scale multitask learning. In International Conference on Machine Learning, 2009.

Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00598, 2022.

Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In Advances in Neural Information Processing Systems, 2023. arXiv:2303.17580.

Chen, L., Zaharia, M., and Zou, J. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024. arXiv:2305.05176.

Breiman, L. Bagging predictors. Machine Learning, 24(2):123–140, 1996.

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. arXiv:2203.11171.

Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-Agents enhances large language model capabilities. arXiv:2406.04692, 2024.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Symposium on Operating Systems Principles, 2023.

Weston, J., Chopra, S., and Bordes, A. Memory networks. In International Conference on Learning Representations, 2015. arXiv:1410.3916.

Graves, A., Wayne, G., and Danihelka, I. Neural Turing machines. arXiv:1410.5401, 2014.

Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. End-to-end memory networks. In Advances in Neural Information Processing Systems, 2015. arXiv:1503.08895.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.

Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256, 1992.

Zaremba, W. and Sutskever, I. Reinforcement learning neural Turing machines. arXiv:1505.00521, 2015.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.

Liu, F. and Perez, J. Gated end-to-end memory networks. In European Chapter of the Association for Computational Linguistics, 2017.

Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. Deformable convolutional networks. In International Conference on Computer Vision, 2017.

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin Transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021.

Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. In International Conference on Learning Representations, 2024. arXiv:2308.00951.

Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. RouteLLM: Learning to route LLMs with preference data. arXiv:2406.18665, 2024.

Xiao, Z., Zhang, D., Wu, Y., Xu, L., Wang, Y., Han, X., Fu, X., Zhong, T., Zeng, J., Song, M., and Chen, G. Chain-of-Experts: When LLMs meet complex operations research problems. In International Conference on Learning Representations, 2024.

Wang, Z., Pan, R., Yao, J., Csordás, R., Li, L., Yin, L., Wu, J., Zhang, T., Li, M., and Liu, S. Chain-of-Experts: Unlocking the communication power of mixture-of-experts models. arXiv:2506.18945, 2025.

BibTeX for the same list is in CITATIONS.bib.