ray.data.from_tf#

ray.data.from_tf(dataset: tf.data.Dataset) → MaterializedDataset[source]#

Create a Dataset from a TensorFlow Dataset.

This function is inefficient. Use it to read small datasets or prototype.

Warning

If your dataset is large, this function may execute slowly or raise an out-of-memory error. To avoid issues, read the underyling data with a function like read_images().

Note

This function isn’t parallelized. It loads the entire dataset into the local node’s memory before moving the data to the distributed object store.

Examples

>>> import ray
>>> import tensorflow_datasets as tfds
>>> dataset, _ = tfds.load('cifar10', split=["train", "test"])  
>>> ds = ray.data.from_tf(dataset)  
>>> ds  
MaterializedDataset(
    num_blocks=...,
    num_rows=50000,
    schema={
        id: binary,
        image: numpy.ndarray(shape=(32, 32, 3), dtype=uint8),
        label: int64
    }
)
>>> ds.take(1)  
[{'id': b'train_16399', 'image': array([[[143,  96,  70],
[141,  96,  72],
[135,  93,  72],
...,
[ 96,  37,  19],
[105,  42,  18],
[104,  38,  20]],
...,
[[195, 161, 126],
[187, 153, 123],
[186, 151, 128],
...,
[212, 177, 147],
[219, 185, 155],
[221, 187, 157]]], dtype=uint8), 'label': 7}]

Parameters:: dataset – A TensorFlow Dataset.
Returns:: A MaterializedDataset that contains the samples stored in the TensorFlow Dataset.