TokenRouter paper describes serving system for token-level LLM routing

Researchers behind TokenRouter say they designed a serving system for token-level LLM routing, a setup that sends inference work across different models at the token level rather than at the session or query level. In the paper, they describe token-level routing as a way to improve the cost-quality balance of LLM serving, while noting that existing systems are built around single-LLM assumptions and struggle with desynchronization, batch admission delays, and implementation complexity. P1

To address those problems, the researchers propose a request-centric programming model with model-centric execution. Developers describe routing from the standpoint of one request, while the runtime launches a subserver for each model and dispatches requests asynchronously. They also say each subserver uses a delayed-batching scheduler whose hyperparameters come from a mathematical throughput model. P2

In their reported tests, the researchers say TokenRouter delivered 2.01-64.15 times higher decoding throughput than existing systems across different routing algorithms, workloads, and model pairs. They also say the system is intended to expose a single external interface and serve as a drop-in replacement for standard single-LLM servers. P1 P6 P7

The paper also notes a limitation: its mathematical model assumes the number of tokens between two consecutive sends follows a geometric distribution, and the authors say handling cases that break that assumption is left for future work. They add that code for the system is available online. P3 P7

Sources

• TokenRouter: Efficient Serving System for Token-Level LLM Routing

• TokenRouter: Efficient Serving System for Token-Level LLM Routing