HTTP Adapters¶

HTTP adapters are functions that convert raw HTTP request to Python types that you know and recognize. Its input arguments should be type annotated. At minimal, it should accept a starlette.requests.Request type. But it can also accept any type that’s recognized by the FastAPI’s dependency injection framework.

For example, here is an adapter that extra the json content from request.

async def json_resolver(request: starlette.requests.Request):
    return await request.json()

Here is an adapter that accept two HTTP query parameters.

def parse_query_args(field_a: int, field_b: str):
    return YourDataClass(field_a, field_b)

You can specify different type signatures to facilitate HTTP fields extraction include query parameters, body parameters, and many other data types. For more detail, you can take a look at FastAPI documentation.

You can use adapters in different scenarios within Serve:

  • Ray AIR ModelWrapper

  • Serve Deployment Graph DAGDriver

  • Embedded in Bring Your Own FastAPI Application

Let’s go over them one by one.

Ray AIR ModelWrapper¶

Ray Serve provides a suite of adapters to convert HTTP requests to ML inputs like numpy arrays. You can use it with Ray AI Runtime (AIR) model wrapper feature to one click deploy pre-trained models.

For example, we provide a simple adapter for n-dimensional array.

With model wrappers, you can specify it via the http_adapter field.

from ray import serve
from ray.serve.http_adapters import json_to_ndarray
from ray.serve.model_wrappers import ModelWrapperDeployment

ModelWrapperDeployment.options(name="my_model").deploy(
    my_ray_air_predictor,
    my_ray_air_checkpoint,
    http_adapter=json_to_ndarray
)

Note

Serve also supports pydantic models as a short-hand for HTTP adapters in model wrappers. Instead of functions, you can directly pass in a pydantic model class to mean ‚Äúvalidate the HTTP body with this schema‚ÄĚ. Once validated, the model instance will passed to the predictor.

from pydantic import BaseModel

class User(BaseModel):
    user_id: int
    user_name: str

...
ModelWrapperDeployment.deploy(..., http_adapter=User)

Serve Deployment Graph DAGDriver¶

In Serve Deployment Graph, you can configure ray.serve.drivers.DAGDriver to accept an http adapter via it’s http_adapter field.

For example, the json request adapters parse JSON in HTTP body:

from ray.serve.drivers import DAGDriver
from ray.serve.http_adapters import json_request
from ray.dag.input_node import InputNode

with InputNode() as input_node:
    ...
    dag = DAGDriver.bind(other_node, http_adapter=json_request)

Note

Serve also supports pydantic models as a short-hand for HTTP adapters in model wrappers. Instead of functions, you can directly pass in a pydantic model class to mean ‚Äúvalidate the HTTP body with this schema‚ÄĚ. Once validated, the model instance will passed as input_node variable.

from pydantic import BaseModel

class User(BaseModel):
    user_id: int
    user_name: str

...
DAGDriver.bind(other_node, http_adapter=User)

Embedded in Bring Your Own FastAPI Application¶

You can also bring the adapter to your own FastAPI app using Depends. The input schema will automatically be part of the generated OpenAPI schema with FastAPI.

from fastapi import FastAPI, Depends
from ray.serve.http_adapters import json_to_ndarray

app = FastAPI()

@app.post("/endpoint")
async def endpoint(np_array = Depends(json_to_ndarray)):
    ...

It has the following schema for input:

pydantic model ray.serve.http_adapters.NdArray[source]¶

Schema for numeric array input.

Show JSON schema
{
   "title": "NdArray",
   "description": "Schema for numeric array input.",
   "type": "object",
   "properties": {
      "array": {
         "title": "Array",
         "description": "The array content as a nested list. You can pass in 1D to 4D array as nested list, or flatten them. When you flatten the array, you can use the `shape` parameter to perform reshaping.",
         "anyOf": [
            {
               "type": "array",
               "items": {
                  "type": "number"
               }
            },
            {
               "type": "array",
               "items": {
                  "type": "array",
                  "items": {
                     "type": "number"
                  }
               }
            },
            {
               "type": "array",
               "items": {
                  "type": "array",
                  "items": {
                     "type": "array",
                     "items": {
                        "type": "number"
                     }
                  }
               }
            },
            {
               "type": "array",
               "items": {
                  "type": "array",
                  "items": {
                     "type": "array",
                     "items": {
                        "type": "array",
                        "items": {
                           "type": "number"
                        }
                     }
                  }
               }
            }
         ]
      },
      "shape": {
         "title": "Shape",
         "description": "The shape of the array. If present, the array will be reshaped.",
         "type": "array",
         "items": {
            "type": "integer"
         }
      },
      "dtype": {
         "title": "Dtype",
         "description": "The numpy dtype of the array. If present, the array will be cast by `astype`.",
         "type": "string"
      }
   },
   "required": [
      "array"
   ]
}

Fields
field array: Union[List[float], List[List[float]], List[List[List[float]]], List[List[List[List[float]]]]] [Required]¶

The array content as a nested list. You can pass in 1D to 4D array as nested list, or flatten them. When you flatten the array, you can use the shape parameter to perform reshaping.

field dtype: Optional[str] = None¶

The numpy dtype of the array. If present, the array will be cast by astype.

field shape: Optional[List[int]] = None¶

The shape of the array. If present, the array will be reshaped.

List of Built-in Adapters¶

Here is a list of adapters and please feel free to contribute more!

ray.serve.http_adapters.json_to_ndarray(payload: ray.serve.http_adapters.NdArray) numpy.ndarray[source]¶

Accepts an NdArray JSON from an HTTP body and converts it to a numpy array. PublicAPI (beta): This API is in beta and may change before becoming stable.

ray.serve.http_adapters.json_to_multi_ndarray(payload: Dict[str, ray.serve.http_adapters.NdArray]) Dict[str, numpy.ndarray][source]¶

Accepts a JSON of shape {str_key: NdArray} and converts it to dict of arrays. PublicAPI (beta): This API is in beta and may change before becoming stable.

ray.serve.http_adapters.starlette_request(request: starlette.requests.Request) starlette.requests.Request[source]¶

Returns the raw request object. PublicAPI (beta): This API is in beta and may change before becoming stable.

async ray.serve.http_adapters.json_request(request: starlette.requests.Request) Dict[str, Any][source]¶

Return the JSON object from request body. PublicAPI (beta): This API is in beta and may change before becoming stable.

ray.serve.http_adapters.image_to_ndarray(img: bytes = File(Ellipsis)) numpy.ndarray[source]¶

Accepts a PIL-readable file from an HTTP form and convert it to a numpy array. PublicAPI (beta): This API is in beta and may change before becoming stable.

async ray.serve.http_adapters.pandas_read_json(raw_request: starlette.requests.Request)[source]¶

Accept JSON body and converts into pandas DataFrame.

This function simply uses pandas.read_json(body, **query_params) under the hood.

PublicAPI (beta): This API is in beta and may change before becoming stable.