download#

ray.data.expressions.download(uri_column_name: str, *, filesystem: FileSystem | None = None) → DownloadExpr[source]#

Create a download expression that downloads content from URIs.

This creates an expression that will download bytes from URIs stored in a specified column. When evaluated, it will fetch the content from each URI and return the downloaded bytes.

A missing object (404), an access-denied object (403) or an invalid URI yields None for that row, and each block logs one warning counting such rows. Transient failures (5xx, throttling, timeouts, dropped connections) are retried with backoff inside the storage client and then re-fetched up to RAY_DATA_DOWNLOAD_MAX_ATTEMPTS times per task (default 2); if they still fail, the task raises DownloadError instead of yielding None, so no row is silently lost. With the default settings a URI that never recovers holds its task for about four minutes before the error is raised: each attempt spends up to RAY_DATA_OBSTORE_RETRY_TIMEOUT_S (120 seconds) in client retries, plus the RAY_DATA_DOWNLOAD_RETRY_BACKOFF_MS pause between attempts, and DataContext.max_map_retries multiplies that again. Bad credentials raise too rather than producing a column of nulls: a 403 whose S3 error code says so (for example InvalidAccessKeyId or SignatureDoesNotMatch) on the obstore backend, and on either backend a block in which every URI is denied, which is the only signal available to the PyArrow backend because its HEAD requests carry no error code. Tune the client retry budget with the RAY_DATA_OBSTORE_RETRY_* environment variables, retry the task with DataContext.retried_map_errors and max_map_retries, or tolerate the failure with DataContext.max_errored_blocks.

Parameters:

  • uri_column_name (str) – The name of the column containing URIs to download from

  • filesystem (FileSystem | None) – PyArrow filesystem to use for reading remote files. If None, the filesystem is auto-detected from the path scheme.

Returns:

A DownloadExpr that will download content from the specified URI column

Return type:

DownloadExpr

Example

>>> from ray.data.expressions import download
>>> import ray
>>> # Create dataset with URIs
>>> ds = ray.data.from_items([
...     {"uri": "s3://bucket/file1.jpg", "id": "1"},
...     {"uri": "s3://bucket/file2.jpg", "id": "2"}
... ])
>>> # Add downloaded bytes column
>>> ds_with_bytes = ds.with_column("bytes", download("uri"))

PublicAPI (alpha): This API is in alpha and may change before becoming stable.