ray.data.Datasource

class ray.data.Datasource

Bases: _DatasourceProjectionPushdownMixin, _DatasourcePredicatePushdownMixin

Interface for defining a custom Dataset datasource.

To read a datasource into a dataset, use read_datasource().

Methods

__init__	Initialize the datasource and its mixins.
apply_predicate	Apply a predicate to this datasource.
apply_projection	Apply a projection to this datasource.
create_reader	Deprecated: Implement get_read_tasks() and estimate_inmemory_data_size() instead.
estimate_inmemory_data_size	Return an estimate of the in-memory data size, or None if unknown.
get_column_renames	Return the column renames from the projection map.
get_name	Return a human-readable name for this datasource.
get_projection_map	Return the projection map (original column names -> final column names).
get_read_tasks	Execute the read and return read tasks.
prepare_read	Deprecated: Implement get_read_tasks() and estimate_inmemory_data_size() instead.
supports_projection_pushdown	Returns True in case Datasource supports projection operation being pushed down into the reading layer

Attributes

should_create_reader
supports_distributed_reads	If False, only launch read tasks on the driver's node.