Examples

Below is a list of tutorials that you can use to learn more about the different pieces of Ray Serve functionality and how to integrate different modeling frameworks.

Serve Examples

• Serving ML Models (Tensorflow, PyTorch, Scikit-Learn, others)
• Serving a Stable Diffusion Model
• Serving a Distilbert Model
• Serving an Object Detection Model
• Serving an inference model on AWS NeuronCores using FastAPI (Experimental)
• Serving RLlib Models
• Scaling your Gradio app with Ray Serve
  • Dependencies
  • Quickstart: Deploy your Gradio app with Ray Serve
    • Deploying Gradio Server
  • Parallelizing models with Ray Serve
    • Original Approach
    • Parallelize using Ray Serve
• Batching Tutorial
  • Define the Deployment
  • Deploy the Deployment
  • Deploy the Deployment using Python API
  • Troubleshooting
• Streaming Tutorial
• Create a Streaming Deployment
• Stream inputs and outputs using WebSockets
• Batch requests and stream the output for each
• Java Tutorial
  • Example Model
  • Converting to a Ray Serve Deployment
  • Testing the Ray Serve Deployment
  • Calling Ray Serve Deployment with HTTP