ray.data.read_datasource#

ray.data.read_datasource(datasource: ray.data.datasource.datasource.Datasource[ray.data.read_api.T], *, parallelism: int = - 1, ray_remote_args: Optional[Dict[str, Any]] = None, **read_args) → ray.data.dataset.Dataset[ray.data.read_api.T][source]#

Read a dataset from a custom data source.

Parameters

  • datasource – The datasource to read data from.

  • parallelism – The requested parallelism of the read. Parallelism may be limited by the available partitioning of the datasource. If set to -1, parallelism will be automatically chosen based on the available cluster resources and estimated in-memory data size.

  • read_args – Additional kwargs to pass to the datasource impl.

  • ray_remote_args – kwargs passed to ray.remote in the read tasks.

Returns

Dataset holding the data read from the datasource.

PublicAPI: This API is stable across Ray releases.