ray.serve.request_router.RequestRouter.max_num_routing_tasks_cap

RequestRouter.max_num_routing_tasks_cap = 50

Hard limit on the maximum number of routing tasks to run. Having too many of these tasks can cause stability issue due to too much load on the local process and many too requests in flight to fetch replicas’ queue lengths.