ray.data.datasource.FastFileMetadataProvider#

class ray.data.datasource.FastFileMetadataProvider[source]#

Bases: ray.data.datasource.file_meta_provider.DefaultFileMetadataProvider

Fast Metadata provider for FileBasedDatasource implementations.

Offers improved performance vs. DefaultFileMetadataProvider by skipping directory path expansion and file size collection. While this performance improvement may be negligible for local filesystems, it can be substantial for cloud storage service providers.

This should only be used when all input paths exist and are known to be files.

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__()