ray.data.from_spark#

ray.data.from_spark(df: pyspark.sql.DataFrame, *, parallelism: Optional[int] = None) → ray.data.dataset.Dataset[ray.data._internal.arrow_block.ArrowRow][source]#

Create a dataset from a Spark dataframe.

Parameters

  • spark – A SparkSession, which must be created by RayDP (Spark-on-Ray).

  • df – A Spark dataframe, which must be created by RayDP (Spark-on-Ray). parallelism: The amount of parallelism to use for the dataset. If not provided, it will be equal to the number of partitions of the original Spark dataframe.

Returns

Dataset holding Arrow records read from the dataframe.

PublicAPI: This API is stable across Ray releases.