- BaseFileMetadataProvider.expand_paths(paths: List[str], filesystem: Optional[pyarrow.fs.FileSystem], partitioning: Optional[ray.data.datasource.partitioning.Partitioning] = None, ignore_missing_paths: bool = False) Iterator[Tuple[str, int]] #
Expands all paths into concrete file paths by walking directories.
Also returns a sidecar of file sizes.
The input paths must be normalized for compatibility with the input filesystem prior to invocation.
paths – A list of file and/or directory paths compatible with the given filesystem.
filesystem – The filesystem implementation that should be used for expanding all paths and reading their files.
ignore_missing_paths – If True, ignores any file paths in
pathsthat are not found. Defaults to False.
An iterator of
(file_path, file_size)pairs. None may be returned for the file size if it is either unknown or will be fetched later by
_get_block_metadata(), but the length of both lists must be equal.