ray.data.datasource.BaseFileMetadataProvider
ray.data.datasource.BaseFileMetadataProvider#
- class ray.data.datasource.BaseFileMetadataProvider[source]#
- Abstract callable that provides metadata for FileBasedDatasource
implementations that reuse the base
prepare_read
method.
Also supports file and file size discovery in input directory paths.
- Current subclasses:
DefaultFileMetadataProvider
DeveloperAPI: This API may change across minor Ray releases.
- __init__()#
Methods
__init__
()expand_paths
(paths, filesystem)Expands all paths into concrete file paths by walking directories.