BaseFileMetadataProvider.expand_paths(paths: List[str], filesystem: Optional[pyarrow.fs.FileSystem], partitioning: Optional[ray.data.datasource.partitioning.Partitioning] = None, ignore_missing_paths: bool = False) Iterator[Tuple[str, int]][source]#

Expands all paths into concrete file paths by walking directories.

Also returns a sidecar of file sizes.

The input paths must be normalized for compatibility with the input filesystem prior to invocation.

  • paths – A list of file and/or directory paths compatible with the given filesystem.

  • filesystem – The filesystem implementation that should be used for expanding all paths and reading their files.

  • ignore_missing_paths – If True, ignores any file paths in paths that are not found. Defaults to False.


An iterator of (file_path, file_size) pairs. None may be returned for the file size if it is either unknown or will be fetched later by _get_block_metadata(), but the length of both lists must be equal.