ray.data.datasource.FileBasedDatasource#

class ray.data.datasource.FileBasedDatasource(paths: str | ~typing.List[str], *, filesystem: pyarrow.fs.FileSystem | None = None, schema: type | pyarrow.lib.Schema | None = None, open_stream_args: ~typing.Dict[str, ~typing.Any] | None = None, meta_provider: ~ray.data.datasource.file_meta_provider.BaseFileMetadataProvider = <ray.data.datasource.file_meta_provider.DefaultFileMetadataProvider object>, partition_filter: ~ray.data.datasource.partitioning.PathPartitionFilter = None, partitioning: ~ray.data.datasource.partitioning.Partitioning = None, ignore_missing_paths: bool = False, shuffle: ~typing.Literal['files'] | None = None, include_paths: bool = False, file_extensions: ~typing.List[str] | None = None)[source]#

Bases: Datasource

File-based datasource for reading files.

Don’t use this class directly. Instead, subclass it and implement _read_stream().

DeveloperAPI: This API may change across minor Ray releases.

Methods

create_reader

Return a Reader for the given read arguments.

get_name

Return a human-readable name for this datasource.

on_write_failed

Callback for when a write job fails.

on_write_start

Create a directory to write files to.

prepare_read

Deprecated: Please implement create_reader() instead.

write

Write blocks for a file-based datasource.

Attributes

should_create_reader

supports_distributed_reads

If False, only launch read tasks on the driver's node.