ray.data.datasource.BaseFileMetadataProvider#

class ray.data.datasource.BaseFileMetadataProvider[source]#
Abstract callable that provides metadata for FileBasedDatasource

implementations that reuse the base prepare_read method.

Also supports file and file size discovery in input directory paths.

Current subclasses:

DefaultFileMetadataProvider

DeveloperAPI: This API may change across minor Ray releases.

__init__()#

Methods

__init__()

expand_paths(paths, filesystem)

Expands all paths into concrete file paths by walking directories.