GPU Usage with Kubernetes¶
This document provides some notes on GPU usage with Kubernetes.
To use GPUs on Kubernetes, you will need to configure both your Kubernetes setup and add additional values to your Ray cluster configuration.
The Ray Docker Hub hosts CUDA-based images packaged with Ray for use in Kubernetes pods.
For example, the image
rayproject/ray-ml:nightly-gpu is ideal for running GPU-based ML workloads with the most recent nightly build of Ray.
Read here for further details on Ray images.
Using Nvidia GPUs requires specifying the relevant resource limits in the container fields of your Kubernetes configurations. (Kubernetes sets the GPU request equal to the limit.) The configuration for a pod running a Ray GPU image and using one Nvidia GPU looks like this:
apiVersion: v1 kind: Pod metadata: generateName: example-cluster-ray-worker spec: ... containers: - name: ray-node image: rayproject/ray:nightly-gpu ... resources: cpu: 1000m memory: 512Mi limits: memory: 512Mi nvidia.com/gpu: 1
GPU taints and tolerations¶
Users using a managed Kubernetes service probably don’t need to worry about this section.
The Nvidia gpu plugin for Kubernetes applies taints to GPU nodes; these taints prevent non-GPU pods from being scheduled on GPU nodes. Managed Kubernetes services like GKE, EKS, and AKS automatically apply matching tolerations to pods requesting GPU resources. Tolerations are applied by means of Kubernetes’s ExtendedResourceToleration admission controller. If this admission controller is not enabled for your Kubernetes cluster, you may need to manually add a GPU toleration each of to your GPU pod configurations. For example,
apiVersion: v1 kind: Pod metadata: generateName: example-cluster-ray-worker spec: ... tolerations: - effect: NoSchedule key: nvidia.com/gpu operator: Exists ... containers: - name: ray-node image: rayproject/ray:nightly-gpu ...
Further reference and discussion¶
If you run into problems setting up GPUs for your Ray cluster on Kubernetes, please reach out to us at https://discuss.ray.io.
Questions or Issues?¶
You can post questions or issues or feedback through the following channels: