Near-Optimal Online Deployment and Routing for Streaming LLMs

putilov_denis, https://stock.adobe.com/uk/335969458, stock.adobe.com
Background
The rapid evolution of large language models (LLMs) creates a highly dynamic environment where new models are continuously released and older ones become obsolete, requiring service providers to manage a fluctuating pool of models with diverse accuracy, latency, and cost profiles. Operating these models in production introduces significant logistical hurdles, particularly the need to balance performance optimization against strict real-world constraints such as hard concurrency caps on simultaneous deployments, long-term cost budgets, and per-model throughput limits.
Technology
A hierarchical algorithm:
(i) optimistically selects up to Mmax models for the next stage using reward upper-confidence and cost lower-confidence bounds.
(ii) routes each incoming query by solving a budget- and throughput-constrained bandit subproblem over the deployed set.
StageRoute tracks a strong oracle under tight budgets across diverse workloads, and proves a regret of O ̃(T2/3) with a matching lower bound, establishing near-optimality, and validates theory empirically.
Advantages
- Hierarchical framework to stream a continually evolving pool of LLMs
- Dynamic deployment:
- (1) at the deployment level, it periodically selects a subset of models by solving an optimization problem that balances upper confidence bounds for rewards and lower confidence bounds for costs against strict concurrency caps, cost budgets, and throughput limits.
- (2) at the routing level, it assigns incoming queries to these deployed models by solving a linear program to calculate optimal routing probabilities based on continuously updated performance statistics.
Application
- Enterprise AI API Gateway Optimization: AI API gateways aggregate multiple LLMs to provide a unified endpoint for enterprise applications.
- High-Volume Automated Content Moderation: Social media platforms process millions of user-generated posts requiring real-time moderation under strict operational budgets.
Inventors
Jian Li, Assistant Professor, Department of Applied Mathematics and Statistics
Shaoang Li, Postdoc, AMS
Licensing Potential
Development partner - Commercial partner - Licensing
Licensing Status
Available
Licensing Contact
Jillian True, Licensing Specialist, Intellectual Property Partners, Jillian.True@stonybrook.edu,
Patent Status
Patent Application Filed
Stage of Development
Proof of Concept
Tech ID
050-9570
