ray.serve.batch
ray.serve.batch#
- ray.serve.batch(func: ray.serve.batching.F) ray.serve.batching.G [source]#
- ray.serve.batch(max_batch_size: Optional[int] = 10, batch_wait_timeout_s: Optional[float] = 0.0) Callable[[ray.serve.batching.F], ray.serve.batching.G]
Converts a function to asynchronously handle batches.
The function can be a standalone function or a class method. In both cases, the function must be
async def
and take a list of objects as its sole argument and return a list of the same length as a result.When invoked, the caller passes a single object. These will be batched and executed asynchronously once there is a batch of
max_batch_size
orbatch_wait_timeout_s
has elapsed, whichever occurs first.Example: >>> from ray import serve >>> @serve.batch(max_batch_size=50, batch_wait_timeout_s=0.5) # doctest: +SKIP … async def handle_batch(batch: List[str]): # doctest: +SKIP … return [s.lower() for s in batch] # doctest: +SKIP
>>> async def handle_single(s: str): ... # Returns s.lower(). ... return await handle_batch(s)
- Parameters
max_batch_size – the maximum batch size that will be executed in one call to the underlying function.
batch_wait_timeout_s – the maximum duration to wait for
max_batch_size
elements before running the underlying function.
PublicAPI (beta): This API is in beta and may change before becoming stable.