Serving ML Models¶

This section should help you:

  • batch requests to optimize performance

  • serve multiple models by composing deployments

  • serve multiple models by making ensemble deployments

Request Batching¶

You can also have Ray Serve batch requests for performance, which is especially important for some ML models that run on GPUs. In order to use this feature, you need to do the following two things:

  1. Use async def for your request handling logic to process queries concurrently.

  2. Use the @serve.batch decorator to batch individual queries that come into the replica. The method/function that’s decorated should handle a list of requests and return a list of the same length.

class BatchingExample:
    def __init__(self):
        self.count = 0

    async def handle_batch(self, requests):
        responses = []
        for request in requests:

        return responses

    async def __call__(self, request):
        return await self.handle_batch(request)


Please take a look at Batching Tutorial for a deep dive.

Model Composition¶


Serve recently added an experimental API for building deployment graphs of multiple models. Please take a look at the Deployment Graph API and try it out!

Ray Serve supports composing individually scalable models into a single model out of the box. For instance, you can combine multiple models to perform stacking or ensembles.

To define a higher-level composed model you need to do three things:

  1. Define your underlying models (the ones that you will compose together) as Ray Serve deployments.

  2. Define your composed model, using the handles of the underlying models (see the example below).

  3. Define a deployment representing this composed model and query it!

In order to avoid synchronous execution in the composed model (e.g., it’s very slow to make calls to the composed model), you’ll need to make the function asynchronous by using an async def. You’ll see this in the example below.

That’s it. Let’s take a look at an example:

from random import random
import requests
import ray
from ray import serve

# Our pipeline will be structured as follows:
# - Input comes in, the composed model sends it to model_one
# - model_one outputs a random number between 0 and 1, if the value is
#   greater than 0.5, then the data is sent to model_two
# - otherwise, the data is returned to the user.

# Let's define two models that just print out the data they received.

def model_one(data):
    print(f"Model 1 called with data:{data}")
    return random()

def model_two(data):
    print(f"Model 2 called with data:{data}")
    # Use this data sent from model_one.
    return data

# max_concurrent_queries is optional. By default, if you pass in an async
# function, Ray Serve sets the limit to a high number.
@serve.deployment(max_concurrent_queries=10, route_prefix="/composed")
class ComposedModel:
    def __init__(self):
        # Use the Python ServeHandle APIs.
        # Set sync=False to override default, which is use this in a
        # synchronous mode.  We want these deployments to be run within an
        # asynchronous event loop for concurrency. See documentation for Sync
        # and Async ServeHandle APIs for details:
        self.model_one = model_one.get_handle(sync=False)
        self.model_two = model_two.get_handle(sync=False)

    # This method can be called concurrently.
    async def __call__(self, starlette_request):
        # At this point you are yielding to the event loop to take in another
        # request.
        data = await starlette_request.body()

        # Use await twice here for two reasons:
        # 1. Since we are running within a async def callable function and we
        # want to use this model_one deployment to run in an asynchronous
        # fashion, this is standard async-await pattern. This await call will
        # return an ObjectRef.
        # 2. The second await waits on the ObjectRef to do an implicit
        # ray.get(Object) to fetch the actual value returned.
        # Hence two awaits.
        score = await (await self.model_one.remote(data=data))
        if score > 0.5:
            await (await self.model_two.remote(data=data))
            result = {"model_used: 1 & 2;  score": score}
            result = {"model_used: 1 ; score": score}

        return result

if __name__ == "__main__":

    # Start ray with 8 processes.
    if ray.is_initialized():
    # Start deployment instances.

    # Now send requests.
    for _ in range(8):
        resp = requests.get("", data="Hey!")


# Output
# {'model_used: 1 ; score': 0.20814435670233788}
# {'model_used: 1 ; score': 0.02964993348224776}
# {'model_used: 1 & 2;  score': 0.7570845154147877}
# {'model_used: 1 & 2;  score': 0.8166808845518793}
# {'model_used: 1 ; score': 0.28354556740137904}
# {'model_used: 1 & 2;  score': 0.5826064390148368}
# {'model_used: 1 ; score': 0.4460146836937825}
# {'model_used: 1 ; score': 0.37099434069129833}

Model Ensemble¶

Ray Serve supports creating different ensemble models

To define an ensemble of different models you need to do three things:

  1. Define your underlying sub models (the ones that make up the ensemble) as Ray Serve deployments.

  2. Define your ensemble model, using the handles of the underlying models (see the example below).

  3. Define a deployment representing this ensemble model and query it!

In order to avoid synchronous execution in the ensemble model, you’ll need to make the function asynchronous by using an async def. In contrast to a composition model, within an ensemble model, you want to call all sub models in parallel. This will be achieved by sending all prediction calls to the sub models via async by using asyncio.wait(). Each serve deployment used in an ensemble use case is independently scalable via changing num_replicas.

That’s it. Let’s take a look at an example:

import ray
import time
import asyncio

from ray import serve

def model_one(input_data):
    print("Model 1 predict")
    return 1

def model_two(input_data):
    print("Model 2 predict")
    return 2

@serve.deployment(max_concurrent_queries=10, route_prefix="/composed")
class EnsembleModel:
    def __init__(self):
        self.model_one = model_one.get_handle()
        self.model_two = model_two.get_handle()

    async def __call__(self, input_data):
        print("Call models concurrently, wait for both to finish")
        tasks = [self.model_one.remote(input_data), self.model_two.remote(input_data)]
        print("collect models predictions (non-blocking)")
        predictions = await asyncio.gather(*tasks)
        return predictions

def send_concurrent_model_requests(num_single_model_replicas=2):
    ensemble_model = EnsembleModel.get_handle()
    all_data = [
        for input_data in range(num_single_model_replicas)
    all_predictions = ray.get(all_data)

if __name__ == "__main__":
    # start local cluster and ray serve processes
    # Start ray with 8 processes.
    if ray.is_initialized():

    # deploy all actors / models

    # Send 2 concurrent requests to the Ensemble Model for predictions.
    # This runs 4 seconds in total calling 2 times the ensemble model
    # concurrently,
    # which calls 2 models in parallel for each call. In total 4 models run
    # parallel.
    st = time.time()
    print("duration", time.time() - st)

    # Output
    # [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]]
    # duration 4.015406847000122

Integration with Model Registries¶

Ray Serve is flexible. If you can load your model as a Python function or class, then you can scale it up and serve it with Ray Serve.

For example, if you are using the MLflow Model Registry to manage your models, the following wrapper class will allow you to load a model using its MLflow Model URI:

import pandas as pd
import mlflow.pyfunc

class MLflowDeployment:
    def __init__(self, model_uri):
        self.model = mlflow.pyfunc.load_model(model_uri=model_uri)

    async def __call__(self, request):
        csv_text = await request.body() # The body contains just raw csv text.
        df = pd.read_csv(csv_text)
        return self.model.predict(df)

model_uri = "model:/my_registered_model/Production"

To serve multiple different MLflow models in the same program, use the name option:



The above approach will work for any model registry, not just MLflow. Namely, load the model from the registry in __init__, and forward the request to the model in __call__.

For a complete hands-on and seamless integration with MLflow, try this self-contained example on your laptop. But first install mlflow.

pip install mlflow
# This brief example shows how to deploy models saved in a model registry such as
# MLflow to Ray Serve, using the simple Ray Serve deployment APIs. You can peruse
# the saved models' metrics and parameters in MLflow ui.
import json
import numpy as np
import pandas as pd
import requests
import os
import tempfile

from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from mlflow.tracking import MlflowClient

from ray import serve
import mlflow

def create_and_save_model():
    # load Iris data
    iris_data = load_iris()
    data, target, target_names = (iris_data['data'],

    # Instantiate a model
    model = GradientBoostingClassifier()

    # Training and validation split
    np.random.shuffle(data), np.random.shuffle(target)
    train_x, train_y = data[:100], target[:100]
    val_x, val_y = data[100:], target[100:]

    # Create labels list as file
    LABEL_PATH = os.path.join(tempfile.gettempdir(), "iris_labels.json")
    with open(LABEL_PATH, "w") as f:
        json.dump(target_names.tolist(), f)

    # Train the model and save our label list as an MLflow artifact
    # mlflow.sklearn.autolog automatically logs all parameters and metrics during
    # the training.
    with mlflow.start_run() as run:, train_y)
        # Log label list as a artifact
        mlflow.log_artifact(LABEL_PATH, artifact_path="labels")

# Create our Ray Serve deployment class

class BoostingModel:
    def __init__(self, uri):
        # Load the model and label artifact from the local
        # Mlflow model registry as a PyFunc Model
        self.model = mlflow.pyfunc.load_model(model_uri=uri)

        # Download the artifact list of labels
        local_dir = "/tmp/artifact_downloads"
        if not os.path.exists(local_dir):
        client = MlflowClient()
        local_path = f"{client.download_artifacts(run_id, 'labels', local_dir)}/iris_labels.json"
        with open(local_path, "r") as f:
            self.label_list = json.load(f)

    async def __call__(self, starlette_request):
        payload = await starlette_request.json()
        print(f"Worker: received Starlette request with data: {payload}")

        # Get the input vector from the payload
        input_vector = [
            payload["sepal length"],
            payload["sepal width"],
            payload["petal length"],
            payload["petal width"],

        # Convert the input vector in a Pandas DataFrame for prediction since
        # an MLflow PythonFunc model, model.predict(...), takes pandas DataFrame
        prediction = self.model.predict(pd.DataFrame([input_vector]))[0]
        human_name = self.label_list[prediction]
        return {"result": human_name}

if __name__ == '__main__':

    # Train and save the model artifacts in MLflow.
    # Here our MLflow model registry is local file
    # directory ./mlruns
    run_id = create_and_save_model()

    # Start the Ray Serve instance
    # Construct model uri to load the model from our model registry
    uri = f"runs:/{run_id}/model"
    # Deploy our model.

    # Send in a request for labels types virginica, setosa, versicolor
    sample_request_inputs = [{
        "sepal length": 6.3,
        "sepal width": 3.3,
        "petal length": 6.0,
        "petal width": 2.5},
        "sepal length": 5.1,
        "sepal width": 3.5,
        "petal length": 1.4,
        "petal width": 0.2},
        "sepal length": 6.4,
        "sepal width": 3.2,
        "petal length": 4.5,
        "petal width": 1.5},
    for input_request in sample_request_inputs:
        response = requests.get("http://localhost:8000/regressor",

    print("Launch MLflow ui to see the model parameters, metrics, and artifacts: `mlflow ui` from current directory.")

    #   "result": "versicolor"
    #    "result": "virginica"
    #    "result": "setosa"
    # Launch MLflow ui to see the model parameters, metrics, and artifacts: `mlflow ui` from current directory.

For an even more hands-off and seamless integration with MLflow, check out the Ray Serve MLflow deployment plugin. A full tutorial is available here.

Framework-Specific Tutorials¶

Ray Serve seamlessly integrates with popular Python ML libraries. Below are tutorials with some of these frameworks to help get you started.