Skip Navigation
Search

Back to Search Innovations

Near-Optimal Online Deployment and Routing for Streaming LLMs


Framework for streaming LLMs and optimizing performance by choice of model

Tech Image

putilov_denis, https://stock.adobe.com/uk/335969458, stock.adobe.com

Background


The rapid evolution of large language models (LLMs) creates a highly dynamic environment where new models are continuously released and older ones become obsolete, requiring service providers to manage a fluctuating pool of models with diverse accuracy, latency, and cost profiles. Operating these models in production introduces significant logistical hurdles, particularly the need to balance performance optimization against strict real-world constraints such as hard concurrency caps on simultaneous deployments, long-term cost budgets, and per-model throughput limits.

Technology


A hierarchical algorithm:

(i) optimistically selects up to Mmax models for the next stage using reward upper-confidence and cost lower-confidence bounds. 

(ii) routes each incoming query by solving a budget- and throughput-constrained bandit subproblem over the deployed set. 

StageRoute tracks a strong oracle under tight budgets across diverse workloads, and proves a regret of O ̃(T2/3) with a matching lower bound, establishing near-optimality, and validates theory empirically.

Advantages

  • Hierarchical framework to stream a continually evolving pool of LLMs
  • Dynamic deployment: 
  • (1) at the deployment level, it periodically selects a subset of models by solving an optimization problem that balances upper confidence bounds for rewards and lower confidence bounds for costs against strict concurrency caps, cost budgets, and throughput limits.
  • (2) at the routing level, it assigns incoming queries to these deployed models by solving a linear program to calculate optimal routing probabilities based on continuously updated performance statistics.

Application

  • Enterprise AI API Gateway Optimization: AI API gateways aggregate multiple LLMs to provide a unified endpoint for enterprise applications. 
  • High-Volume Automated Content Moderation: Social media platforms process millions of user-generated posts requiring real-time moderation under strict operational budgets.

Inventors

Jian Li, Assistant Professor, Department of Applied Mathematics and Statistics
Shaoang Li, Postdoc, AMS

Licensing Potential


Development partner - Commercial partner - Licensing

Licensing Status


Available 

Licensing Contact

Jillian True, Licensing Specialist, Intellectual Property Partners, Jillian.True@stonybrook.edu,

Patent Status


Patent Application Filed

Stage of Development


Proof of Concept

Tech ID

050-9570