dual-flow-transformer

Dual-Flow Transformer

Allocating additional computation to autoregressive decoding while preserving prompt-side computation and a single persistent KV cache.

📄 Paper PDF

Abstract

Abstract

As large language models serve an increasing number of requests, cumulative inference cost is becoming more important relative to the one-time cost of training. Prompt prefill and autoregressive decoding impose different hardware demands: prefill is highly parallel and typically compute-bound, whereas decoding is sequential and often constrained by memory bandwidth. Conventional width, depth, and Mixture-of-Experts scaling increase computation in both phases together, even when additional capacity is primarily needed for continuation prediction.

We introduce the Dual-Flow Transformer, an architecture that allocates learned computation asymmetrically between prefill and decoding. Its primary flow is a complete causal language model that processes the full prompt and alone maintains the persistent key-value cache. An auxiliary flow can be omitted over the prompt and activated only from its final position onward, adding continuation-prediction computation without introducing a second persistent KV cache or altering the primary flow’s cached states. When combined with Mixture-of-Experts layers, the two flows route through the expert pool separately, allowing more expert computation to be activated during decoding while retaining the lower prompt-side expert budget of the primary flow.

We train the two flows jointly using a confidence-weighted likelihood objective, where the primary distribution determines how their token-level losses are combined. Experiments show that Dual-Flow models improve continuation modeling relative to compute-matched conventional scaling, while preserving the primary model’s prefill path and KV-cache footprint. These results suggest that model capacity need not be allocated uniformly across inference phases, and that phase-aware computation provides a practical new scaling dimension for language models.

Resources

• Paper: DualFlow.pdf
• arXiv: Pending public release
• Code: To be released
• Models: To be released

Repository status

This repository provides an early public version of the manuscript while the arXiv submission is being processed. The PDF may be updated as the work evolves.

Citation

A BibTeX entry will be added once the arXiv identifier is available. For now, please cite the manuscript using its title and authors:

@misc{dualflowtransformer2026,
  title        = {Dual-Flow Transformer},
  author       = {Liming Liu, Mingze Wang, Tuo Zhao},
  year         = {2026},
  howpublished = {Manuscript},
  note         = {Available from this Hugging Face repository}
}

License

The manuscript is released under the Creative Commons Attribution 4.0 International License.