Saving data#
Ray Data saves datasets to files and converts them to objects from other Python libraries. This guide shows you how to write data to files and convert datasets to other Python libraries.
Write data to files#
Ray Data writes to shared local storage and cloud storage.
Write data to cloud storage#
To save your Dataset to cloud storage, authenticate all nodes with your cloud service provider. Then, call a method such as Dataset.write_parquet and specify a URI with the appropriate scheme. The URI can point to a bucket or a folder.
To write data to formats other than Parquet, see the Saving Data API.
To save data to Amazon S3, specify a URI with the s3:// scheme.
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
ds.write_parquet("s3://my-bucket/my-folder")
Ray Data relies on PyArrow to authenticate with Amazon S3. To configure your credentials for PyArrow, see the PyArrow S3 filesystem documentation.
To save data to Google Cloud Storage, install gcsfs, the filesystem interface to Google Cloud Storage:
pip install gcsfs
Then, create a GCSFileSystem and specify a URI with the gcs:// scheme.
import gcsfs
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
filesystem = gcsfs.GCSFileSystem(project="my-google-project")
ds.write_parquet("gcs://my-bucket/my-folder", filesystem=filesystem)
Ray Data relies on PyArrow to authenticate with Google Cloud Storage. To configure your credentials for PyArrow, see the PyArrow GCS filesystem documentation.
To save data to Azure Blob Storage, install adlfs, the filesystem interface to Azure Data Lake Storage Gen1 and Gen2:
pip install adlfs
Then, create an AzureBlobFileSystem and specify a URI with the az:// scheme.
import adlfs
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
filesystem = adlfs.AzureBlobFileSystem(account_name="azureopendatastorage")
ds.write_parquet("az://my-bucket/my-folder", filesystem=filesystem)
Ray Data relies on PyArrow to authenticate with Azure Blob Storage. To configure your credentials for PyArrow, see the PyArrow documentation on fsspec-compatible filesystems.
Change the number of output files#
When you call a write method, Ray Data writes your data to several files. To control the number of output files, set min_rows_per_file.
Note
min_rows_per_file is a hint, not a strict limit. Ray Data might write more or fewer rows to each file. If the number of rows per block is larger than min_rows_per_file, Ray Data writes the number of rows per block to each file.
import os
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
ds.write_csv("/tmp/few_files/", min_rows_per_file=75)
print(os.listdir("/tmp/few_files/"))
['0_000001_000000.csv', '0_000000_000000.csv', '0_000002_000000.csv']
Write into a partitioned dataset#
To write a partitioned dataset with Hive-style, folder-based partitioning, repartition the dataset by the partition columns first. Repartitioning gives you control over the number of files and their sizes. After you repartition by the partition columns, every block holds all the rows for a particular partition. The repartitioning then determines how many files Ray creates, and write-method parameters such as max_rows_per_file can optionally limit them further. Ray writes every block independently. Without the repartition, every block can carry rows for any partition, so you can get N files per partition, where N is the number of blocks in your dataset. You then have little control over the number of files and their sizes.
Warning
Ray Data has deprecated using min_rows_per_file with non-empty partition_cols. Support for this combination ends after February 2027. Instead, call repartition() with the partition columns and an explicit num_blocks, and use max_rows_per_file. If you already repartition the dataset by the partition columns, removing min_rows_per_file leaves the output layout unchanged.
import os
import ray
import pandas as pd
def print_directory_tree(start_path: str) -> None:
"""
Prints the directory tree structure starting from the given path.
"""
for root, dirs, files in os.walk(start_path):
level = root.replace(start_path, '').count(os.sep)
indent = ' ' * 4 * (level)
print(f'{indent}{os.path.basename(root)}/')
subindent = ' ' * 4 * (level + 1)
for f in files:
print(f'{subindent}{f}')
# Sample dataset to partition by ``city`` and ``year``.
df = pd.DataFrame(
{
"city": ["SF", "SF", "NYC", "NYC", "SF", "NYC", "SF", "NYC"],
"year": [2023, 2024, 2023, 2024, 2023, 2023, 2024, 2024],
"sales": [100, 120, 90, 115, 105, 95, 130, 110],
}
)
ds = ray.data.from_pandas(df)
# Partitioned write:
# 1. Repartition so all rows with the same (city, year) land in the same
# block. This minimizes shuffling during the write.
# 2. Pass the same columns to ``partition_cols`` so Ray creates a
# Hive-style directory layout: city=<value>/year=<value>/....
# 3. Use ``max_rows_per_file`` to cap how many rows Ray puts in each
# Parquet file.
ds.repartition(keys=["city", "year"], num_blocks=4).write_parquet(
"/tmp/sales_partitioned",
partition_cols=["city", "year"],
max_rows_per_file=3,
)
print_directory_tree("/tmp/sales_partitioned")
sales_partitioned/
city=NYC/
year=2024/
1_a2b8b82cd2904a368ec39f42ae3cf830_000000_000000-0.parquet
year=2023/
1_a2b8b82cd2904a368ec39f42ae3cf830_000001_000000-0.parquet
city=SF/
year=2024/
1_a2b8b82cd2904a368ec39f42ae3cf830_000000_000000-0.parquet
year=2023/
1_a2b8b82cd2904a368ec39f42ae3cf830_000001_000000-0.parquet
Convert datasets to other Python libraries#
Convert a dataset to a pandas DataFrame, or to a DataFrame from a distributed data processing framework.
Convert datasets to pandas#
To convert a Dataset to a pandas DataFrame, call Dataset.to_pandas(). The whole dataset must fit in the memory of the process that calls to_pandas().
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
df = ds.to_pandas()
print(df)
sepal length (cm) sepal width (cm) ... petal width (cm) target
0 5.1 3.5 ... 0.2 0
1 4.9 3.0 ... 0.2 0
2 4.7 3.2 ... 0.2 0
3 4.6 3.1 ... 0.2 0
4 5.0 3.6 ... 0.2 0
.. ... ... ... ... ...
145 6.7 3.0 ... 2.3 2
146 6.3 2.5 ... 1.9 2
147 6.5 3.0 ... 2.0 2
148 6.2 3.4 ... 2.3 2
149 5.9 3.0 ... 1.8 2
<BLANKLINE>
[150 rows x 5 columns]
Convert datasets to distributed DataFrames#
Ray Data interoperates with distributed data processing frameworks such as Daft, Dask, Spark, Modin, and Mars.
To convert a Dataset to a Daft DataFrame, call Dataset.to_daft().
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
df = ds.to_daft()
print(df)
╭───────────────────┬──────────────────┬───────────────────┬──────────────────┬────────╮
│ sepal length (cm) ┆ sepal width (cm) ┆ petal length (cm) ┆ petal width (cm) ┆ target │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ Float64 ┆ Float64 ┆ Float64 ┆ Float64 ┆ Int64 │
╞═══════════════════╪══════════════════╪═══════════════════╪══════════════════╪════════╡
│ 5.1 ┆ 3.5 ┆ 1.4 ┆ 0.2 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 4.9 ┆ 3 ┆ 1.4 ┆ 0.2 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 4.7 ┆ 3.2 ┆ 1.3 ┆ 0.2 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 4.6 ┆ 3.1 ┆ 1.5 ┆ 0.2 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 5 ┆ 3.6 ┆ 1.4 ┆ 0.2 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 5.4 ┆ 3.9 ┆ 1.7 ┆ 0.4 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 4.6 ┆ 3.4 ┆ 1.4 ┆ 0.3 ┆ 0 │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 5 ┆ 3.4 ┆ 1.5 ┆ 0.2 ┆ 0 │
╰───────────────────┴──────────────────┴───────────────────┴──────────────────┴────────╯
(Showing first 8 of 150 rows)
To convert a Dataset to a Dask DataFrame, call Dataset.to_dask().
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
df = ds.to_dask()
To convert a Dataset to a Spark DataFrame, call Dataset.to_spark().
import ray
import raydp
spark = raydp.init_spark(
app_name = "example",
num_executors = 1,
executor_cores = 4,
executor_memory = "512M"
)
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
df = ds.to_spark(spark)
To convert a Dataset to a Modin DataFrame, call Dataset.to_modin().
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
mdf = ds.to_modin()
To convert a Dataset to a Mars DataFrame, call Dataset.to_mars().
import ray
ds = ray.data.read_csv("s3://anonymous@ray-example-data/iris.csv")
mdf = ds.to_mars()