ray.data.read_datasource(datasource: ray.data.datasource.datasource.Datasource, *, parallelism: int = - 1, ray_remote_args: Dict[str, Any] = None, **read_args) ray.data.dataset.Dataset[source]#

Read a stream from a custom data source.

  • datasource – The datasource to read data from.

  • parallelism – The requested parallelism of the read. Parallelism may be limited by the available partitioning of the datasource. If set to -1, parallelism will be automatically chosen based on the available cluster resources and estimated in-memory data size.

  • read_args – Additional kwargs to pass to the datasource impl.

  • ray_remote_args – kwargs passed to ray.remote in the read tasks.


Dataset that reads data from the datasource.

PublicAPI: This API is stable across Ray releases.