Inspecting Data#

Inspect Datasets to better understand your data.

This guide shows you how to:

Describing datasets#

Datasets are tabular. To view a dataset’s column names and types, call Dataset.schema().

import ray

ds = ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")

print(ds.schema())
Column             Type
------             ----
sepal length (cm)  double
sepal width (cm)   double
petal length (cm)  double
petal width (cm)   double
target             int64

For more information like the number of rows, print the Dataset.

import ray

ds = ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")

print(ds)
Dataset(
   num_blocks=...,
   num_rows=150,
   schema={
      sepal length (cm): double,
      sepal width (cm): double,
      petal length (cm): double,
      petal width (cm): double,
      target: int64
   }
)

Inspecting rows#

To get a list of rows, call Dataset.take() or Dataset.take_all(). Ray Data represents each row as a dictionary.

import ray

ds = ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")

rows = ds.take(1)
print(rows)
[{'sepal length (cm)': 5.1, 'sepal width (cm)': 3.5, 'petal length (cm)': 1.4, 'petal width (cm)': 0.2, 'target': 0}]

For more information on working with rows, see Transforming rows and Iterating over rows.

Inspecting batches#

A batch contains data from multiple rows. To inspect batches, call Dataset.take_batch().

By default, Ray Data represents batches as dicts of NumPy ndarrays. To change the type of the returned batch, set batch_format.

import ray

ds = ray.data.read_images("s3://anonymous@ray-example-data/image-datasets/simple")

batch = ds.take_batch(batch_size=2, batch_format="numpy")
print("Batch:", batch)
print("Image shape", batch["image"].shape)
Batch: {'image': array([[[[...]]]], dtype=uint8)}
Image shape: (2, 32, 32, 3)
import ray

ds = ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")

batch = ds.take_batch(batch_size=2, batch_format="pandas")
print(batch)
   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  target
0                5.1               3.5  ...               0.2       0
1                4.9               3.0  ...               0.2       0
<BLANKLINE>
[2 rows x 5 columns]

For more information on working with batches, see Transforming batches and Iterating over batches.

Inspecting execution statistics#

Ray Data calculates statistics during execution like the wall clock time and memory usage for the different stages.

To view stats about your Datasets, call Dataset.stats() on an executed dataset. The stats are also persisted under /tmp/ray/session_*/logs/ray-data.log.

def pause(x):
    time.sleep(.0001)
    return x

ds = (
    ray.data.read_csv("s3://anonymous@air-example-data/iris.csv")
    .map(lambda x: x)
    .map(pause)
)

for batch in ds.iter_batches():
    pass

print(ds.stats())
Operator 1 ReadCSV->SplitBlocks(4): 1 tasks executed, 4 blocks produced in 0.22s
* Remote wall time: 222.1ms min, 222.1ms max, 222.1ms mean, 222.1ms total
* Remote cpu time: 15.6ms min, 15.6ms max, 15.6ms mean, 15.6ms total
* Peak heap memory usage (MiB): 157953.12 min, 157953.12 max, 157953 mean
* Output num rows: 150 min, 150 max, 150 mean, 150 total
* Output size bytes: 6000 min, 6000 max, 6000 mean, 6000 total
* Tasks per node: 1 min, 1 max, 1 mean; 1 nodes used
* Extra metrics: {'obj_store_mem_freed': 5761}

Dataset iterator time breakdown:
* Total time user code is blocked: 5.68ms
* Total time in user code: 0.96us
* Total time overall: 238.93ms
* Num blocks local: 0
* Num blocks remote: 0
* Num blocks unknown location: 1
* Batch iteration time breakdown (summed across prefetch threads):
    * In ray.get(): 2.16ms min, 2.16ms max, 2.16ms avg, 2.16ms total
    * In batch creation: 897.67us min, 897.67us max, 897.67us avg, 897.67us total
    * In batch formatting: 836.87us min, 836.87us max, 836.87us avg, 836.87us total