read_webdataset#

ray.data.read_webdataset(paths: str | List[str], *, filesystem: pyarrow.fs.FileSystem | None = None, parallelism: int = -1, num_cpus: float | None = None, num_gpus: float | None = None, memory: float | None = None, arrow_open_stream_args: Dict[str, Any] | None = None, partition_filter: PathPartitionFilter | None = None, decoder: bool | str | callable | list | None = True, fileselect: list | callable | None = None, filerename: list | callable | None = None, suffixes: list | callable | None = None, verbose_open: bool = False, shuffle: Literal['files'] | FileShuffleConfig | None = None, include_paths: bool = False, file_extensions: List[str] | None = None, concurrency: int | None = None, override_num_blocks: int | None = None, expand_json: bool = False, label_selector: Dict[str, str] | None = None, fallback_strategy: List[Dict[str, Any]] | None = None, max_calls: int | None = None, resources: Dict[str, float] | None = None, accelerator_type: str | None = None, runtime_env: Dict[str, Any] | None = None, ray_remote_args: Dict[str, Any] | None = None) → Dataset[source]#

Create a Dataset from WebDataset files.

Parameters:

  • paths (str | List[str]) – A single file/directory path or a list of file/directory paths. A list of paths can contain both files and directories.

  • filesystem (pyarrow.fs.FileSystem | None) – The filesystem implementation to read from.

  • parallelism (int) – This argument is deprecated. Use override_num_blocks argument.

  • num_cpus (float | None) – The number of CPUs to reserve for each parallel read worker.

  • num_gpus (float | None) – The number of GPUs to reserve for each parallel read worker. For example, specify num_gpus=1 to request 1 GPU for each parallel read worker.

  • memory (float | None) – The heap memory in bytes to reserve for each parallel read worker.

  • arrow_open_stream_args (Dict[str, Any] | None) – Key-word arguments passed to pyarrow.fs.FileSystem.open_input_stream. To read a compressed TFRecord file, pass the corresponding compression type (e.g. for GZIP or ZLIB, use arrow_open_stream_args={'compression': 'gzip'}).

  • partition_filter (PathPartitionFilter | None) – Path-based partition filter, if any. Can be used with a custom callback to read only selected partitions of a dataset.

  • decoder (bool | str | callable | list | None) – A function or list of functions to decode the data.

  • fileselect (list | callable | None) – A callable or list of glob patterns to select files.

  • filerename (list | callable | None) – A function or list of tuples to rename files prior to grouping.

  • suffixes (list | callable | None) – A function or list of suffixes to select for creating samples.

  • verbose_open (bool) – Whether to print the file names as they are opened.

  • shuffle (Literal['files'] | ~ray.data.datasource.file_based_datasource.FileShuffleConfig | None) – If setting to “files”, randomly shuffle input files order before read. if setting to FileShuffleConfig, the random seed can be passed toshuffle the input files, i.e. FileShuffleConfig(seed = 42). Defaults to not shuffle with None.

  • include_paths (bool) – If True, include the path to each file. File paths are stored in the 'path' column.

  • file_extensions (List[str] | None) – A list of file extensions to filter files by.

  • concurrency (int | None) – The maximum number of Ray tasks to run concurrently. Set this to control number of tasks to run concurrently. This doesn’t change the total number of tasks run or the total number of output blocks. By default, concurrency is dynamically decided based on the available resources.

  • override_num_blocks (int | None) – Override the number of output blocks from all read tasks. By default, the number of output blocks is dynamically decided based on input data size and available resources. You shouldn’t manually set this value in most cases.

  • expand_json (bool) – If True, expand JSON objects into individual samples. Defaults to False.

  • label_selector (Dict[str, str] | None) – Labels required on the node where each read task runs.

  • fallback_strategy (List[Dict[str, Any]] | None) – Alternative label requirements that Ray tries in order if label_selector can’t be satisfied.

  • max_calls (int | None) – The maximum number of read tasks a worker runs before exiting.

  • resources (Dict[str, float] | None) – Custom resources to reserve for each read task, expressed as a mapping from resource name to quantity.

  • accelerator_type (str | None) – The accelerator type required for each read task.

  • runtime_env (Dict[str, Any] | None) – The runtime environment to use for each read task.

  • ray_remote_args (Dict[str, Any] | None) – Additional options passed to ray.remote() for each read task. This argument is deprecated and will be removed in Ray 2.64. Use the named remote parameters instead.

Returns:

A Dataset that contains the example features.

Raises:

ValueError – If a file contains a message that isn’t a tf.train.Example.

Return type:

Dataset

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

On this page