Keras and Tensorflow Tutorial¶

In this guide, we will train and deploy a simple Tensorflow neural net. In particular, we show:

How to load the model from file system in your Ray Serve definition
How to parse the JSON request and evaluated in Tensorflow

Please see the Core API: Deployments to learn more general information about Ray Serve.

Ray Serve is framework agnostic – you can use any version of Tensorflow. However, for this tutorial, we use Tensorflow 2 and Keras. Please make sure you have Tensorflow 2 installed.

pip install "tensorflow>=2.0"

Let’s import Ray Serve and some other helpers.

from ray import serve

import os
import tempfile
import numpy as np
import requests

We will train a simple MNIST model using Keras.

TRAINED_MODEL_PATH = os.path.join(tempfile.gettempdir(), "mnist_model.h5")


def train_and_save_model():
    import tensorflow as tf
    # Load mnist dataset
    mnist = tf.keras.datasets.mnist
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0

    # Train a simple neural net model
    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),
        tf.keras.layers.Dense(128, activation="relu"),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(10)
    ])
    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])
    model.fit(x_train, y_train, epochs=1)

    model.evaluate(x_test, y_test, verbose=2)
    model.summary()

    # Save the model in h5 format in local file system
    model.save(TRAINED_MODEL_PATH)


if not os.path.exists(TRAINED_MODEL_PATH):
    train_and_save_model()

Services are just defined as normal classes with __init__ and __call__ methods. The __call__ method will be invoked per request.

@serve.deployment(route_prefix="/mnist")
class TFMnistModel:
    def __init__(self, model_path):
        import tensorflow as tf
        self.model_path = model_path
        self.model = tf.keras.models.load_model(model_path)

    async def __call__(self, starlette_request):
        # Step 1: transform HTTP request -> tensorflow input
        # Here we define the request schema to be a json array.
        input_array = np.array((await starlette_request.json())["array"])
        reshaped_array = input_array.reshape((1, 28, 28))

        # Step 2: tensorflow input -> tensorflow output
        prediction = self.model(reshaped_array)

        # Step 3: tensorflow output -> web output
        return {
            "prediction": prediction.numpy().tolist(),
            "file": self.model_path
        }

Now that we’ve defined our services, let’s deploy the model to Ray Serve. We will define a Serve deployment that will be exposed over an HTTP route.

serve.start()
TFMnistModel.deploy(TRAINED_MODEL_PATH)

Let’s query it!

resp = requests.get(
    "http://localhost:8000/mnist",
    json={"array": np.random.randn(28 * 28).tolist()})
print(resp.json())
# {
#  "prediction": [[-1.504277229309082, ..., -6.793371200561523]],
#  "file": "/tmp/mnist_model.h5"
# }

Ray v2.0.0.dev0

Keras and Tensorflow Tutorial¶