- ray.data.read_binary_files(paths: Union[str, List[str]], *, include_paths: bool = False, filesystem: Optional[pyarrow.fs.FileSystem] = None, parallelism: int = - 1, ray_remote_args: Dict[str, Any] = None, arrow_open_stream_args: Optional[Dict[str, Any]] = None, meta_provider: Optional[ray.data.datasource.file_meta_provider.BaseFileMetadataProvider] = None, partition_filter: Optional[ray.data.datasource.partitioning.PathPartitionFilter] = None, partitioning: ray.data.datasource.partitioning.Partitioning = None, ignore_missing_paths: bool = False, shuffle: Optional[Literal['files']] = None) ray.data.dataset.Dataset #
Datasetfrom binary files of arbitrary contents.
Read a file in remote storage.
>>> import ray >>> path = "s3://anonymous@ray-example-data/pdf-sample_0.pdf" >>> ds = ray.data.read_binary_files(path) >>> ds.schema() Column Type ------ ---- bytes binary
Read multiple local files.
>>> ray.data.read_binary_files( ... ["local:///path/to/file1", "local:///path/to/file2"])
Read a file with the filepaths included as a column in the dataset.
>>> path = "s3://anonymous@ray-example-data/pdf-sample_0.pdf" >>> ds = ray.data.read_binary_files(path, include_paths=True) >>> ds.take(1)["path"] 'ray-example-data/pdf-sample_0.pdf'
paths – A single file or directory, or a list of file or directory paths. A list of paths can contain both files and directories.
include_paths – If
True, include the path to each file. File paths are stored in the
filesystem – The PyArrow filesystem implementation to read from. These filesystems are specified in the PyArrow docs. Specify this parameter if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with
ray_remote_args – kwargs passed to
remote()in the read tasks.
parallelism – The amount of parallelism to use for the dataset. Defaults to -1, which automatically determines the optimal parallelism for your configuration. You should not need to manually set this value in most cases. For details on how the parallelism is automatically determined and guidance on how to tune it, see Tuning read parallelism. Parallelism is upper bounded by the total number of files.
arrow_open_stream_args – kwargs passed to pyarrow.fs.FileSystem.open_input_file.
meta_provider – A file metadata provider. Custom metadata providers may be able to resolve file metadata more quickly and/or accurately. In most cases, you do not need to set this. If
None, this function uses a system-chosen implementation.
partition_filter – A
PathPartitionFilter. Use with a custom callback to read only selected partitions of a dataset. By default, no files are filtered. By default, this does not filter out any files.
partitioning – A
Partitioningobject that describes how paths are organized. Defaults to
ignore_missing_paths – If True, ignores any file paths in
pathsthat are not found. Defaults to False.
shuffle – If setting to “files”, randomly shuffle input files order before read. Defaults to not shuffle with
Datasetproducing rows read from the specified paths.