ray.data.datasource.BaseFileMetadataProvider#

class ray.data.datasource.BaseFileMetadataProvider[source]#

Bases: ray.data.datasource.file_meta_provider.FileMetadataProvider

Abstract callable that provides metadata for FileBasedDatasource implementations that reuse the base prepare_read() method.

Also supports file and file size discovery in input directory paths.

Current subclasses:

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__()

expand_paths(paths, filesystem[, ...])

Expands all paths into concrete file paths by walking directories.