ray.data.Dataset.to_spark

Dataset.to_spark(spark: pyspark.sql.SparkSession) → pyspark.sql.DataFrame

Convert this dataset into a Spark dataframe.

Note

This operation will trigger execution of the lazy transformations performed on this dataset, and will block until execution completes.

Time complexity: O(dataset size / parallelism)

Returns

A Spark dataframe created from this dataset.