ray.data.from_spark#

ray.data.from_spark(df: pyspark.sql.DataFrame, *, parallelism: Optional[int] = None) → ray.data.dataset.MaterializedDataset[source]#

Create a Dataset from a Spark DataFrame.

Parameters

  • df – A Spark DataFrame, which must be created by RayDP (Spark-on-Ray).

  • parallelism – The amount of parallelism to use for the dataset. If not provided, the parallelism is equal to the number of partitions of the original Spark DataFrame.

Returns

A MaterializedDataset holding rows read from the DataFrame.