read_orc#

ray.data.read_orc(paths: str | List[str], *, filesystem: pyarrow.fs.FileSystem | None = None, parallelism: int = -1, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, partition_filter: PathPartitionFilter | None = None, partitioning: Partitioning = None, include_paths: bool = False, ignore_missing_paths: bool = False, shuffle: Literal['files'] | FileShuffleConfig | None = None, file_extensions: List[str] | None = ['orc'], concurrency: int | None = None, override_num_blocks: int | None = None, merge_schema: bool = False, label_selector: Dict[str, str] | None = None, fallback_strategy: List[Dict[str, Any]] | None = None, max_calls: int | None = None, resources: Dict[str, float] | None = None, accelerator_type: str | None = None, runtime_env: Dict[str, Any] | None = None, ray_remote_args: Dict[str, Any] | None = None) → Dataset[source]#

Create a Dataset from records stored in ORC files.

Note

The DataSourceV2 reader is selected through the use_datasource_v2 setting of DataContext. By default, it samples file schemas and may omit columns absent from the sample. Use merge_schema=True to infer from all eligible files. If you encounter a DataSourceV2 limitation, use the V1 reader by setting ray.data.DataContext.get_current().use_datasource_v2 = False, or RAY_DATA_USE_DATASOURCE_V2=0 in the environment before importing Ray. This shared setting also affects read_parquet().

Examples

Read an ORC file in remote storage or local storage.

>>> import ray
>>> ds = ray.data.read_orc("s3://bucket/path")
>>> ds.schema()

Read multiple local files.

>>> ray.data.read_orc(
...    ["local:///path/to/file1", "local:///path/to/file2"])
Parameters:

  • paths (str | List[str]) – A single file or directory, or a list of file or directory paths. A list of paths can contain both files and directories.

  • filesystem (pyarrow.fs.FileSystem | None) – The PyArrow filesystem implementation to read from. These filesystems are specified in the PyArrow docs. Specify this parameter if you need to provide specific configurations to the filesystem. By default, the filesystem is automatically selected based on the scheme of the paths. For example, if the path begins with s3://, the S3FileSystem is used.

  • parallelism (int) – This argument is deprecated. Use override_num_blocks argument.

  • num_cpus (float | None) – The number of CPUs to reserve for each parallel read worker.

  • num_gpus (float | None) – The number of GPUs to reserve for each parallel read worker. For example, specify num_gpus=1 to request 1 GPU for each parallel read worker.

  • memory (float | None) – The heap memory in bytes to reserve for each parallel read worker.

  • partition_filter (PathPartitionFilter | None) – A PathPartitionFilter. Use with a custom callback to read only selected partitions of a dataset. By default, this filters out any file paths whose file extension does not match "*.orc*".

  • partitioning (Partitioning) – A Partitioning object that describes how paths are organized. Defaults to None.

  • include_paths (bool) – If True, include the path to each file. File paths are stored in the 'path' column.

  • ignore_missing_paths (bool) – If True, ignores any file paths in paths that are not found. Defaults to False.

  • shuffle (Literal['files'] | ~ray.data.datasource.file_based_datasource.FileShuffleConfig | None) – If setting to “files”, randomly shuffle input files order before read. If setting to FileShuffleConfig, you can pass a seed to shuffle the input files. Defaults to not shuffle with None.

  • file_extensions (List[str] | None) – A list of file extensions to filter files by. Defaults to ["orc"]. Pass None to disable extension-based filtering.

  • concurrency (int | None) – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks (int | None) – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

  • merge_schema (bool) – If True, infer the schema from every eligible ORC file before returning the Dataset. Defaults to False, which infers from up to 16 files. Requires the V2 datasource. See the notes below for cost and file-discovery behavior.

  • label_selector (Dict[str, str] | None) – Labels required on the node where each read task runs.

  • fallback_strategy (List[Dict[str, Any]] | None) – Alternative label requirements that Ray tries in order if label_selector can’t be satisfied.

  • max_calls (int | None) – The maximum number of read tasks a worker runs before exiting.

  • resources (Dict[str, float] | None) – Custom resources to reserve for each read task, expressed as a mapping from resource name to quantity.

  • accelerator_type (str | None) – The accelerator type required for each read task.

  • runtime_env (Dict[str, Any] | None) – The runtime environment to use for each read task.

  • ray_remote_args (Dict[str, Any] | None) – Additional options passed to ray.remote() for each read task. This argument is deprecated and will be removed in Ray 2.64. Use the named remote parameters instead.

Returns:

Dataset holding records from the ORC files.

Return type:

Dataset

Note

merge_schema=True lists all eligible files and reads their schemas on the driver during planning, which can be expensive for large or remote datasets. The reader uses the merged schema, but Ray doesn’t retain per-file ORC metadata from schema inference for reuse during execution. With more than 10,000 candidate files, Ray warns before schema merging. Retained path and size metadata adds O(files) data to serialized lineage.

Follow-up work for distributed planning-time schema reads and metadata reuse is tracked in ray-project/ray#67040.

The Dataset retains those candidate paths across actions. Files appended after discovery aren’t included. Concurrent directory traversal isn’t an atomic snapshot, and replacing a file at the same path is outside this guarantee. This option uses the existing schema unifier and reader; it doesn’t add support for otherwise unsupported schema changes. In particular, evolved nested struct fields can fail at read time on PyArrow 17 (see ray-project/ray#67064).

PublicAPI (alpha): This API is in alpha and may change before becoming stable.