ray.data.datasource.DefaultFileMetadataProvider
ray.data.datasource.DefaultFileMetadataProvider#
- class ray.data.datasource.DefaultFileMetadataProvider[source]#
Default metadata provider for FileBasedDatasource implementations that reuse the base
prepare_read
method.Calculates block size in bytes as the sum of its constituent file sizes, and assumes a fixed number of rows per file.
DeveloperAPI: This API may change across minor Ray releases.
- __init__()#
Methods
__init__
()expand_paths
(paths, filesystem)Expands all paths into concrete file paths by walking directories.