ray.data.datasource.BaseFileMetadataProvider
ray.data.datasource.BaseFileMetadataProvider#
- class ray.data.datasource.BaseFileMetadataProvider[source]#
Bases:
ray.data.datasource.file_meta_provider.FileMetadataProvider
Abstract callable that provides metadata for
FileBasedDatasource
implementations that reuse the baseprepare_read()
method.Also supports file and file size discovery in input directory paths.
- Current subclasses:
DeveloperAPI: This API may change across minor Ray releases.
Methods
__init__
()expand_paths
(paths, filesystem[, ...])Expands all paths into concrete file paths by walking directories.