ray.data.Dataset.write_numpy#

Dataset.write_numpy(path: str, *, column: str, filesystem: pyarrow.fs.FileSystem | None = None, try_create_dir: bool = True, arrow_open_stream_args: Dict[str, Any] | None = None, filename_provider: FilenameProvider | None = None, min_rows_per_file: int | None = None, ray_remote_args: Dict[str, Any] = None, concurrency: int | None = None, num_rows_per_file: int | None = None) → None[source]#

Writes a column of the Dataset to .npy files.

This is only supported for columns in the datasets that can be converted to NumPy arrays.

The number of files is determined by the number of blocks in the dataset. To control the number of number of blocks, call repartition().

By default, the format of the output files is {uuid}_{block_idx}.npy, where uuid is a unique id for the dataset. To modify this behavior, implement a custom FilenameProvider and pass it in as the filename_provider argument.

Note

This operation will trigger execution of the lazy transformations performed on this dataset.

Examples

>>> import ray
>>> ds = ray.data.range(100)
>>> ds.write_numpy("local:///tmp/data/", column="id")

Time complexity: O(dataset size / parallelism)

Parameters:

  • path – The path to the destination root directory, where the npy files are written to.

  • column – The name of the column that contains the data to be written.

  • filesystem – The pyarrow filesystem implementation to write to. These filesystems are specified in the pyarrow docs. Specify this if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with s3://, the S3FileSystem is used.

  • try_create_dir – If True, attempts to create all directories in destination path. Does nothing if all directories already exist. Defaults to True.

  • arrow_open_stream_args – kwargs passed to pyarrow.fs.FileSystem.open_output_stream, which is used when opening the file to write to.

  • filename_provider – A FilenameProvider implementation. Use this parameter to customize what your filenames look like.

  • min_rows_per_file – [Experimental] The target minimum number of rows to write to each file. If None, Ray Data writes a system-chosen number of rows to each file. If the number of rows per block is larger than the specified value, Ray Data writes the number of rows per block to each file. The specified value is a hint, not a strict limit. Ray Data might write more or fewer rows to each file.

  • ray_remote_args – kwargs passed to ray.remote() in the write tasks.

  • concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run. By default, concurrency is dynamically decided based on the available resources.

  • num_rows_per_file – [Deprecated] Use min_rows_per_file instead.