ray.data.datasource.FastFileMetadataProvider#

class ray.data.datasource.FastFileMetadataProvider[source]#

Fast Metadata provider for FileBasedDatasource implementations.

Offers improved performance vs. DefaultFileMetadataProvider by skipping directory path expansion and file size collection. While this performance improvement may be negligible for local filesystems, it can be substantial for cloud storage service providers.

This should only be used when all input paths are known to be files.

DeveloperAPI: This API may change across minor Ray releases.

__init__()#

Methods

__init__()

expand_paths(paths, filesystem)

Expands all paths into concrete file paths by walking directories.