ray.data.Datasource

class ray.data.Datasource

Interface for defining a custom Dataset datasource.

To read a datasource into a dataset, use read_datasource(). To write to a writable datasource, use write_datasource().

See RangeDatasource and DummyOutputDatasource for examples of how to implement readable and writable datasources.

Note

Datasource instances must be serializable, since write() is called in remote tasks.

Methods

__init__
create_reader	Return a Reader for the given read arguments.
estimate_inmemory_data_size	Return an estimate of the in-memory data size, or None if unknown.
get_name	Return a human-readable name for this datasource.
get_read_tasks	Execute the read and return read tasks.
on_write_complete	Callback for when a write job completes.
on_write_failed	Callback for when a write job fails.
on_write_start	Callback for when a write job starts.
prepare_read	Deprecated: Please implement create_reader() instead.
write	Write blocks out to the datasource.

Attributes

should_create_reader
supports_distributed_reads	If False, only launch read tasks on the driver's node.