ray.data.read_tfrecords#

ray.data.read_tfrecords(paths: Union[str, List[str]], *, filesystem: Optional[pyarrow.fs.FileSystem] = None, parallelism: int = - 1, arrow_open_stream_args: Optional[Dict[str, Any]] = None, meta_provider: Optional[ray.data.datasource.file_meta_provider.BaseFileMetadataProvider] = None, partition_filter: Optional[ray.data.datasource.partitioning.PathPartitionFilter] = None, ignore_missing_paths: bool = False, tf_schema: Optional[schema_pb2.Schema] = None, shuffle: Optional[Literal['files']] = None) → ray.data.dataset.Dataset[source]#

Create a Dataset from TFRecord files that contain tf.train.Example messages.

Warning

This function exclusively supports tf.train.Example messages. If a file contains a message that isn’t of type tf.train.Example, then this function fails.

Examples

>>> import ray
>>> ray.data.read_tfrecords("s3://anonymous@ray-example-data/iris.tfrecords")
Dataset(
   num_blocks=...,
   num_rows=150,
   schema={...}
)

We can also read compressed TFRecord files, which use one of the compression types supported by Arrow:

>>> ray.data.read_tfrecords(
...     "s3://anonymous@ray-example-data/iris.tfrecords.gz",
...     arrow_open_stream_args={"compression": "gzip"},
... )
Dataset(
   num_blocks=...,
   num_rows=150,
   schema={...}
)
Parameters

  • paths – A single file or directory, or a list of file or directory paths. A list of paths can contain both files and directories.

  • filesystem – The PyArrow filesystem implementation to read from. These filesystems are specified in the PyArrow docs. Specify this parameter if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with s3://, the S3FileSystem is used.

  • parallelism – The amount of parallelism to use for the dataset. Defaults to -1, which automatically determines the optimal parallelism for your configuration. You should not need to manually set this value in most cases. For details on how the parallelism is automatically determined and guidance on how to tune it, see Tuning read parallelism. Parallelism is upper bounded by the total number of records in all the CSV files.

  • arrow_open_stream_args – kwargs passed to pyarrow.fs.FileSystem.open_input_file. when opening input files to read. To read a compressed TFRecord file, pass the corresponding compression type (e.g., for GZIP or ZLIB), use arrow_open_stream_args={'compression_type': 'gzip'}).

  • meta_provider – A file metadata provider. Custom metadata providers may be able to resolve file metadata more quickly and/or accurately. In most cases, you do not need to set this. If None, this function uses a system-chosen implementation.

  • partition_filter – A PathPartitionFilter. Use with a custom callback to read only selected partitions of a dataset. By default, this filters out any file paths whose file extension does not match "*.tfrecords*".

  • ignore_missing_paths – If True, ignores any file paths in paths that are not found. Defaults to False.

  • tf_schema – Optional TensorFlow Schema which is used to explicitly set the schema of the underlying Dataset.

  • shuffle – If setting to “files”, randomly shuffle input files order before read. Defaults to not shuffle with None.

Returns

A Dataset that contains the example features.

Raises

ValueError – If a file contains a message that isn’t a tf.train.Example.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.