read_images#

ray.data.read_images(paths: str | List[str], *, filesystem: pyarrow.fs.FileSystem | None = None, parallelism: int = -1, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, arrow_open_file_args: Dict[str, Any] | None = None, partition_filter: PathPartitionFilter | None = None, partitioning: Partitioning = None, size: Tuple[int, int] | None = None, mode: str | None = None, include_paths: bool = False, ignore_missing_paths: bool = False, shuffle: Literal['files'] | FileShuffleConfig | None = None, file_extensions: List[str] | None = ['png', 'jpg', 'jpeg', 'tif', 'tiff', 'bmp', 'gif'], concurrency: int | None = None, override_num_blocks: int | None = None, label_selector: Dict[str, str] | None = None, fallback_strategy: List[Dict[str, Any]] | None = None, max_calls: int | None = None, resources: Dict[str, float] | None = None, accelerator_type: str | None = None, runtime_env: Dict[str, Any] | None = None, ray_remote_args: Dict[str, Any] = None) → Dataset[source]#

Creates a Dataset from image files.

The column name defaults to “image”.

Examples

>>> import ray
>>> path = "s3://anonymous@ray-example-data/batoidea/JPEGImages/"
>>> ds = ray.data.read_images(path)
>>> ds.schema()
Column  Type
------  ----
image   ArrowTensorTypeV2(shape=(32, 32, 3), dtype=uint8)

If you need image file paths, set include_paths=True.

>>> ds = ray.data.read_images(path, include_paths=True)
>>> ds.schema()
Column  Type
------  ----
image   ArrowTensorTypeV2(shape=(32, 32, 3), dtype=uint8)
path    string
>>> ds.take(1)[0]["path"]
'ray-example-data/batoidea/JPEGImages/1.jpeg'

If your images are arranged like:

root/dog/xxx.png
root/dog/xxy.png

root/cat/123.png
root/cat/nsdf3.png

Then you can include the labels by specifying a Partitioning.

>>> import ray
>>> from ray.data.datasource.partitioning import Partitioning
>>> root = "s3://anonymous@ray-example-data/image-datasets/dir-partitioned"
>>> partitioning = Partitioning("dir", field_names=["class"], base_dir=root)
>>> ds = ray.data.read_images(root, size=(224, 224), partitioning=partitioning)
>>> ds.schema()
Column  Type
------  ----
image   ArrowTensorTypeV2(shape=(224, 224, 3), dtype=uint8)
class   string
Parameters:

  • paths (str | List[str]) – A single file or directory, or a list of file or directory paths. A list of paths can contain both files and directories.

  • filesystem (pyarrow.fs.FileSystem | None) – The pyarrow filesystem implementation to read from. These filesystems are specified in the pyarrow docs. Specify this parameter if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with s3://, the S3FileSystem is used.

  • parallelism (int) – This argument is deprecated. Use override_num_blocks argument.

  • num_cpus (float | None) – The number of CPUs to reserve for each parallel read worker.

  • num_gpus (float | None) – The number of GPUs to reserve for each parallel read worker. For example, specify num_gpus=1 to request 1 GPU for each parallel read worker.

  • memory (float | None) – The heap memory in bytes to reserve for each parallel read worker.

  • arrow_open_file_args (Dict[str, Any] | None) – kwargs passed to pyarrow.fs.FileSystem.open_input_file. when opening input files to read.

  • partition_filter (PathPartitionFilter | None) – A PathPartitionFilter. Use with a custom callback to read only selected partitions of a dataset. By default, this filters out any file paths whose file extension does not match *.png, *.jpg, *.jpeg, *.tiff, *.bmp, or *.gif.

  • partitioning (Partitioning) – A Partitioning object that describes how paths are organized. Defaults to None.

  • size (Tuple[int, int] | None) – The desired height and width of loaded images. If unspecified, images retain their original shape.

  • mode (str | None) – A Pillow mode describing the desired type and depth of pixels. If unspecified, image modes are inferred by Pillow.

  • include_paths (bool) – If True, include the path to each image. File paths are stored in the 'path' column.

  • ignore_missing_paths (bool) – If True, ignores any file/directory paths in paths that are not found. Defaults to False.

  • shuffle (Literal['files'] | ~ray.data.datasource.file_based_datasource.FileShuffleConfig | None) – If setting to “files”, randomly shuffle input files order before read. If setting to FileShuffleConfig, you can pass a seed to shuffle the input files. Defaults to not shuffle with None.

  • file_extensions (List[str] | None) – A list of file extensions to filter files by.

  • concurrency (int | None) – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks (int | None) – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

  • label_selector (Dict[str, str] | None) – Labels required on the node where each read task runs.

  • fallback_strategy (List[Dict[str, Any]] | None) – Alternative label requirements that Ray tries in order if label_selector can’t be satisfied.

  • max_calls (int | None) – The maximum number of read tasks a worker runs before exiting.

  • resources (Dict[str, float] | None) – Custom resources to reserve for each read task, expressed as a mapping from resource name to quantity.

  • accelerator_type (str | None) – The accelerator type required for each read task.

  • runtime_env (Dict[str, Any] | None) – The runtime environment to use for each read task.

  • ray_remote_args (Dict[str, Any]) – Additional options passed to ray.remote() for each read task. This argument is deprecated and will be removed in Ray 2.64. Use the named remote parameters instead.

Returns:

A Dataset producing tensors that represent the images at the specified paths. For information on working with tensors, read the tensor data guide.

Raises:
Return type:

Dataset

PublicAPI (beta): This API is in beta and may change before becoming stable.

On this page