ray.data.Dataset.write_snowflake#

Dataset.write_snowflake(table: str, connection_parameters: str, *, ray_remote_args: Dict[str, Any] = None, concurrency: int | None = None)[source]#

Write this Dataset to a Snowflake table.

Note

This operation will trigger execution of the lazy transformations performed on this dataset.

Examples

import ray

connection_parameters = dict(
    user=...,
    account="ABCDEFG-ABC12345",
    password=...,
    database="SNOWFLAKE_SAMPLE_DATA",
    schema="TPCDS_SF100TCL"
)
ds = ray.data.read_parquet("s3://anonymous@ray-example-data/iris.parquet")
ds.write_snowflake("MY_DATABASE.MY_SCHEMA.IRIS", connection_parameters)
Parameters:

  • table – The name of the table to write to.

  • connection_parameters – Keyword arguments to pass to snowflake.connector.connect. To view supported parameters, read https://docs.snowflake.com/developer-guide/python-connector/python-connector-api#functions.

  • ray_remote_args – Keyword arguments passed to ray.remote() in the write tasks.

  • concurrency – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run. By default, concurrency is dynamically decided based on the available resources.