ray.data.datasource.BaseFileMetadataProvider#

class ray.data.datasource.BaseFileMetadataProvider[source]#

Bases: FileMetadataProvider

Abstract callable that provides metadata for FileBasedDatasource implementations that reuse the base prepare_read() method.

Also supports file and file size discovery in input directory paths.

Current subclasses:

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__

expand_paths

Expands all paths into concrete file paths by walking directories.