configure

DataConfig.configure(datasets: Dict[str, Dataset], world_size: int, worker_handles: List[ActorHandle] | None, worker_node_ids: List[NodeIdStr] | None, **kwargs) → List[Dict[str, DataIterator]]

Configure how Train datasets should be assigned to workers.

Parameters:

• datasets (Dict[str, Dataset]) – The datasets dict passed to Train by the user.

• world_size (int) – The number of Train workers in total.

• worker_handles (List[ActorHandle] | None) – The actor handles of the Train workers.

• worker_node_ids (List[NodeIdStr] | None) – The node ids of the Train workers.

• **kwargs – Forwards compatibility placeholder.

Returns:

A list of dataset splits for each worker. The size of the list must be equal to world_size. Each element of the list contains the assigned DataIterator instances by name for the worker.

Return type:

List[Dict[str, DataIterator]]

DeveloperAPI: This API may change across minor Ray releases.