Developing Ray Serve Python scripts on a RayCluster#

In this tutorial, you will learn how to effectively debug your Ray Serve scripts against a RayCluster, enabling enhanced observability and faster iteration speed compared to developing the script directly with a RayService. Many RayService issues are related to the Ray Serve Python scripts, so it is important to ensure the correctness of the scripts before deploying them to a RayService. This tutorial will show you how to develop a Ray Serve Python script for a MobileNet image classifier on a RayCluster. You can deploy and serve the classifier on your local Kind cluster without requiring a GPU. Refer to ray-service.mobilenet.yaml and mobilenet-rayservice.md for more details.

Step 1: Install a KubeRay cluster#

Follow this document to install the latest stable KubeRay operator via Helm repository.

Step 2: Create a RayCluster CR#

helm install raycluster kuberay/ray-cluster --version 1.0.0

Step 3: Log in to the head Pod#

export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl exec -it $HEAD_POD -- bash

Step 4: Prepare your Ray Serve Python scripts and run the Ray Serve application#

# Execute the following command in the head Pod
git clone https://github.com/ray-project/serve_config_examples.git
cd serve_config_examples

# Try to launch the Ray Serve application
serve run mobilenet.mobilenet:app
# [Error message]
#     from tensorflow.keras.preprocessing import image
# ModuleNotFoundError: No module named 'tensorflow'

serve run mobilenet.mobilenet:app: The first mobilenet is the name of the directory in the serve_config_examples/, the second mobilenet is the name of the Python file in the directory mobilenet/, and app is the name of the variable representing Ray Serve application within the Python file. See the section “import_path” in rayservice-troubleshooting.md for more details.

Step 5: Change the Ray image from `rayproject/ray:${RAY_VERSION}` to `rayproject/ray-ml:${RAY_VERSION}`#

# Uninstall RayCluster
helm uninstall raycluster

# Install the RayCluster CR with the Ray image `rayproject/ray-ml:${RAY_VERSION}`
helm install raycluster kuberay/ray-cluster --version 1.0.0 --set image.repository=rayproject/ray-ml

The error message in Step 4 indicates that the Ray image rayproject/ray:${RAY_VERSION} does not have the TensorFlow package. Due to the significant size of TensorFlow, we have opted to use an image with TensorFlow as the base instead of installing it within Runtime Environments. In this Step, we will change the Ray image from rayproject/ray:${RAY_VERSION} to rayproject/ray-ml:${RAY_VERSION}.

Step 6: Repeat Step 3 and Step 4#

# Repeat Step 3 and Step 4 to log in to the new head Pod and run the Ray Serve application.
# You should successfully launch the Ray Serve application this time.
serve run mobilenet.mobilenet:app

# [Example output]
# (ServeReplica:default_ImageClassifier pid=139, ip=10.244.0.8) Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224.h5
#     8192/14536120 [..............................] - ETA: 0s)
#  4202496/14536120 [=======>......................] - ETA: 0s)
# 12902400/14536120 [=========================>....] - ETA: 0s)
# 14536120/14536120 [==============================] - 0s 0us/step
# 2023-07-17 14:04:43,737 SUCC scripts.py:424 -- Deployed Serve app successfully.

Step 7: Submit a request to the Ray Serve application#

# (On your local machine) Forward the serve port of the head Pod
kubectl port-forward --address 0.0.0.0 $HEAD_POD 8000

# Clone the repository on your local machine
git clone https://github.com/ray-project/serve_config_examples.git
cd serve_config_examples/mobilenet

# Prepare a sample image file. `stable_diffusion_example.png` is a cat image generated by the Stable Diffusion model.
curl -O https://raw.githubusercontent.com/ray-project/kuberay/master/docs/images/stable_diffusion_example.png

# Update `image_path` in `mobilenet_req.py` to the path of `stable_diffusion_example.png`
# Send a request to the Ray Serve application.
python3 mobilenet_req.py

# [Error message]
# Unexpected error, traceback: ray::ServeReplica:default_ImageClassifier.handle_request() (pid=139, ip=10.244.0.8)
#   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serve/_private/utils.py", line 254, in wrap_to_ray_error
#     raise exception
#   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/serve/_private/replica.py", line 550, in invoke_single
#     result = await method_to_call(*args, **kwargs)
#   File "./mobilenet/mobilenet.py", line 24, in __call__
#   File "/home/ray/anaconda3/lib/python3.7/site-packages/starlette/requests.py", line 256, in _get_form
#     ), "The `python-multipart` library must be installed to use form parsing."
# AssertionError: The `python-multipart` library must be installed to use form parsing..

python-multipart is required for the request parsing function starlette.requests.form(), so the error message is reported when we send a request to the Ray Serve application.

Step 8: Restart the Ray Serve application with runtime environment.#

# In the head Pod, stop the Ray Serve application
serve shutdown

# Check the Ray Serve application status
serve status
# [Example output]
# There are no applications running on this cluster.

# Launch the Ray Serve application with runtime environment.
serve run mobilenet.mobilenet:app --runtime-env-json='{"pip": ["python-multipart==0.0.6"]}'

# (On your local machine) Submit a request to the Ray Serve application again, and you should get the correct prediction.
python3 mobilenet_req.py
# [Example output]
# {"prediction": ["n02123159", "tiger_cat", 0.2994779646396637]}

Step 9: Create a RayService YAML file#

In the previous steps, we found that the Ray Serve application can be successfully launched using the Ray image rayproject/ray-ml:${RAY_VERSION} and the runtime environments python-multipart==0.0.6. Therefore, we can create a RayService YAML file with the same Ray image and runtime environment. For more details, please refer to ray-service.mobilenet.yaml and mobilenet-rayservice.md.