Using Custom Datasources

Datasets can read and write in parallel to custom datasources defined in Python. Once you have implemented YourCustomDataSource, you can use it like any other source in Ray Data:

# Read from a custom datasource.
ds = ray.data.read_datasource(YourCustomDatasource(), **read_args)

# Write to a custom datasource.
ds.write_datasource(YourCustomDatasource(), **write_args)