ray.train.DataConfig.configure#

DataConfig.configure(datasets: Dict[str, Dataset], world_size: int, worker_handles: List[ActorHandle] | None, worker_node_ids: List[str] | None, **kwargs) → List[Dict[str, DataIterator]][source]#

Configure how Train datasets should be assigned to workers.

Parameters:

  • datasets – The datasets dict passed to Train by the user.

  • world_size – The number of Train workers in total.

  • worker_handles – The actor handles of the Train workers.

  • worker_node_ids – The node ids of the Train workers.

  • kwargs – Forwards compatibility placeholder.

Returns:

A list of dataset splits for each worker. The size of the list must be equal to world_size. Each element of the list contains the assigned DataIterator instances by name for the worker.

DeveloperAPI: This API may change across minor Ray releases.