ray.data.Dataset.to_arrow_refs#

Dataset.to_arrow_refs() → List[ray.types.ObjectRef[pyarrow.Table]][source]#

Convert this dataset into a distributed set of Arrow tables.

This is only supported for datasets convertible to Arrow records. This function is zero-copy if the existing data is already in Arrow format. Otherwise, the data will be converted to Arrow format.

Note

This operation will trigger execution of the lazy transformations performed on this dataset.

Time complexity: O(1) unless conversion is required.

Returns

A list of remote Arrow tables created from this dataset.

DeveloperAPI: This API may change across minor Ray releases.