ray.data.Dataset.write_tfrecords
ray.data.Dataset.write_tfrecords#
- Dataset.write_tfrecords(path: str, *, tf_schema: Optional[schema_pb2.Schema] = None, filesystem: Optional[pyarrow.fs.FileSystem] = None, try_create_dir: bool = True, arrow_open_stream_args: Optional[Dict[str, Any]] = None, block_path_provider: ray.data.datasource.file_based_datasource.BlockWritePathProvider = <ray.data.datasource.file_based_datasource.DefaultBlockWritePathProvider object>, ray_remote_args: Dict[str, Any] = None) None [source]#
Write the
Dataset
to TFRecord files.The TFRecord files contain tf.train.Example records, with one Example record for each row in the dataset.
Warning
tf.train.Feature only natively stores ints, floats, and bytes, so this function only supports datasets with these data types, and will error if the dataset contains unsupported types.
The number of files is determined by the number of blocks in the dataset. To control the number of number of blocks, call
repartition()
.This method is only supported for datasets with records that are convertible to pyarrow tables.
By default, the format of the output files is
{uuid}_{block_idx}.tfrecords
, whereuuid
is a unique id for the dataset. To modify this behavior, implement a customBlockWritePathProvider
and pass it in as theblock_path_provider
argument.Note
This operation will trigger execution of the lazy transformations performed on this dataset.
Examples
>>> import ray >>> ds = ray.data.range(100) >>> ds.write_tfrecords("local:///tmp/data/")
Time complexity: O(dataset size / parallelism)
- Parameters
path – The path to the destination root directory, where tfrecords files are written to.
filesystem – The pyarrow filesystem implementation to write to. These filesystems are specified in the pyarrow docs. Specify this if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with
s3://
, theS3FileSystem
is used.try_create_dir – If
True
, attempts to create all directories in the destination path. Does nothing if all directories already exist. Defaults toTrue
.arrow_open_stream_args – kwargs passed to pyarrow.fs.FileSystem.open_output_stream, which is used when opening the file to write to.
block_path_provider – A
BlockWritePathProvider
implementation specifying the filename structure for each output parquet file. By default, the format of the output files is{uuid}_{block_idx}.tfrecords
, whereuuid
is a unique id for the dataset.ray_remote_args – kwargs passed to
remote()
in the write tasks.