ray.data.datasource.FileBasedDatasource#
- class ray.data.datasource.FileBasedDatasource(paths: str | ~typing.List[str], *, filesystem: pyarrow.fs.FileSystem | None = None, schema: type | pyarrow.lib.Schema | None = None, open_stream_args: ~typing.Dict[str, ~typing.Any] | None = None, meta_provider: ~ray.data.datasource.file_meta_provider.BaseFileMetadataProvider = <ray.data.datasource.file_meta_provider.DefaultFileMetadataProvider object>, partition_filter: ~ray.data.datasource.partitioning.PathPartitionFilter = None, partitioning: ~ray.data.datasource.partitioning.Partitioning = None, ignore_missing_paths: bool = False, shuffle: ~typing.Literal['files'] | None = None, include_paths: bool = False, file_extensions: ~typing.List[str] | None = None)[source]#
Bases:
Datasource
File-based datasource for reading files.
Don’t use this class directly. Instead, subclass it and implement
_read_stream()
.DeveloperAPI: This API may change across minor Ray releases.
Methods
Return a Reader for the given read arguments.
Return a human-readable name for this datasource.
Callback for when a write job fails.
Create a directory to write files to.
Deprecated: Please implement create_reader() instead.
Write blocks for a file-based datasource.
Attributes
If
False
, only launch read tasks on the driver's node.