Ray Serve: Scalable and Programmable Serving

Warning

Ray Serve is changing fast! You’re probably running the latest pip release and not the nightly build, so please ensure you’re viewing the correct version of this documentation. Here’s the documentation for the latest pip release of Ray Serve.

Warning

As of Ray 1.4, Serve has a new API centered around the concept of “Deployments.” Deployments offer a more streamlined API and can be declaratively updated, which should improve both development and production workflows. The existing APIs have not changed from Ray 1.4 and will continue to work until Ray 1.5, at which point they will be removed (see the package reference if you’re not sure about a specific API). Please see the migration guide for details on how to update your existing Serve application to use this new API and as always we welcome feedback on Slack, GitHub, or the Ray forum!

../_images/logo.svg

Ray Serve is an easy-to-use scalable model serving library built on Ray. Ray Serve is:

  • Framework-agnostic: Use a single toolkit to serve everything from deep learning models built with frameworks like PyTorch, Tensorflow, and Keras, to Scikit-Learn models, to arbitrary Python business logic.

  • Python-first: Configure your model serving with pure Python code—no more YAML or JSON configs.

Since Ray Serve is built on Ray, it allows you to easily scale to many machines, both in your datacenter and in the cloud.

Ray Serve can be used in two primary ways to deploy your models at scale:

  1. Have Python functions and classes automatically placed behind HTTP endpoints.

  2. Alternatively, call them from within your existing Python web server using the Python-native ServeHandle API.

Tip

Chat with Ray Serve users and developers on our forum!

Ray Serve Quickstart

Ray Serve supports Python versions 3.6 through 3.8. To install Ray Serve, run the following command:

pip install "ray[serve]"

Now you can serve a function…

import requests

from ray import serve

serve.start()


@serve.deployment
def hello(request):
    name = request.query_params["name"]
    return f"Hello {name}!"


hello.deploy()

# Query our endpoint over HTTP.
response = requests.get("http://127.0.0.1:8000/hello?name=serve").text
assert response == "Hello serve!"

…or serve a stateful class.

import requests

import ray
from ray import serve

serve.start()


@serve.deployment
class Counter:
    def __init__(self):
        self.count = 0

    def __call__(self, *args):
        self.count += 1
        return {"count": self.count}


# Deploy our class.
Counter.deploy()

# Query our endpoint in two different ways: from HTTP and from Python.
assert requests.get("http://127.0.0.1:8000/Counter").json() == {"count": 1}
assert ray.get(Counter.get_handle().remote()) == {"count": 2}

See Core API: Deployments for more exhaustive coverage about Ray Serve and its core concept of a Deployment. For a high-level view of the architecture underlying Ray Serve, see Serve Architecture.

Why Ray Serve?

There are generally two ways of serving machine learning applications, both with serious limitations: you can use a traditional web server—your own Flask app—or you can use a cloud-hosted solution.

The first approach is easy to get started with, but it’s hard to scale each component. The second approach requires vendor lock-in (SageMaker), framework-specific tooling (TFServing), and a general lack of flexibility.

Ray Serve solves these problems by giving you a simple web server (and the ability to use your own) while still handling the complex routing, scaling, and testing logic necessary for production deployments.

Beyond scaling up your backends with multiple replicas, Ray Serve also enables:

  • Model Composition—ability to flexibly compose multiple models and independently scale and update each.

  • Request Batching—built in request batching to help you meet your performance objectives.

  • Resource Management (CPUs, GPUs)—specify fractional resource requirements to fully saturate each of your GPUs with several models.

For more on the motivation behind Ray Serve, check out these meetup slides and this blog post.

When should I use Ray Serve?

Ray Serve is a flexible tool that’s easy to use for deploying, operating, and monitoring Python-based machine learning applications. Ray Serve excels when you want to mix business logic with ML models and scaling out in production is a necessity. This might be because of large-scale batch processing requirements or because you want to scale up a model pipeline consisting of many individual models with different performance properties.

If you plan on running on multiple machines, Ray Serve will serve you well!

What’s next?

Check out the End-to-End Tutorial and Core API: Deployments, look at the Ray Serve FAQ, or head over to the Advanced Tutorials to get started building your Ray Serve applications.

For more, see the following blog posts about Ray Serve: