__init__

DataConfig.__init__(datasets_to_split: Literal['all'] | List[str] = 'all', execution_options: ExecutionOptions | Dict[str, ExecutionOptions] | None = None, enable_shard_locality: bool = True)

Construct a DataConfig.

Parameters:

• datasets_to_split – Specifies which datasets should be split among workers. Can be set to “all” or a list of dataset names. Defaults to “all”, i.e. split all datasets.

• execution_options –

  Optional Ray Data execution options. When set, they are applied to dataset shards. When None (the default), Train applies default_ingest_options() to each dataset shard. Can be either:

  1. A single ExecutionOptions object applied to all datasets.

  2. A dict mapping dataset names to ExecutionOptions for per-dataset overrides. Datasets not present in the dict use default_ingest_options().

  NOTE: For exclude_resources and resource_limits, those options only affect Ray Data after train performs its cluster resource reservation. So if you specify exclude_resources, it will exclude the resources from data’s reservation, not train’s reservation.

• enable_shard_locality – If true, dataset sharding across Train workers will consider locality to minimize cross-node data transfer. Enabled by default.