ray.data.datasource.DefaultFileMetadataProvider#

class ray.data.datasource.DefaultFileMetadataProvider[source]#

Default metadata provider for FileBasedDatasource implementations that reuse the base prepare_read method.

Calculates block size in bytes as the sum of its constituent file sizes, and assumes a fixed number of rows per file.

DeveloperAPI: This API may change across minor Ray releases.

__init__()#

Methods

__init__()

expand_paths(paths, filesystem)

Expands all paths into concrete file paths by walking directories.