ray.data.datasource.RowBasedFileDatasink#

class ray.data.datasource.RowBasedFileDatasink(path: str, *, filesystem: pyarrow.fs.FileSystem | None = None, try_create_dir: bool = True, open_stream_args: Dict[str, Any] | None = None, filename_provider: FilenameProvider | None = None, block_path_provider: BlockWritePathProvider | None = None, dataset_uuid: str | None = None, file_format: str | None = None)[source]#

Bases: _FileDatasink

A datasink that writes one row to each file.

Subclasses must implement write_row_to_file and call the superclass constructor.

Examples

import io
import numpy as np
from PIL import Image
from ray.data._internal.delegating_block_builder import DelegatingBlockBuilder
from ray.data.datasource import FileBasedDatasource

class ImageDatasource(FileBasedDatasource):
    def __init__(self, paths):
        super().__init__(
            paths,
            file_extensions=["png", "jpg", "jpeg", "bmp", "gif", "tiff"],
        )

    def _read_stream(self, f, path):
        data = f.readall()
        image = Image.open(io.BytesIO(data))

        builder = DelegatingBlockBuilder()
        array = np.array(image)
        item = {"image": array}
        builder.add(item)
        yield builder.build()

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__

Initialize this datasink.

get_name

Return a human-readable name for this datasink.

on_write_failed

Callback for when a write job fails.

on_write_start

Create a directory to write files to.

write_row_to_file

Write a row to a file.

Attributes

supports_distributed_writes

If False, only launch write tasks on the driver's node.