Ray Collective API

For usage, see Ray Collective Communication Lib.

Collective groups

ray.util.collective.collective.init_collective_group	Initialize a collective group inside an actor process.
ray.util.collective.collective.create_collective_group	Declare a list of actors as a collective group.
ray.util.collective.collective.destroy_collective_group	Destroy a collective group given its group name.
ray.util.collective.collective.is_group_initialized	Check if the group is initialized in this process by the group name.
ray.util.collective.collective.get_rank	Return the rank of this process in the given group.
ray.util.collective.collective.get_collective_group_size	Return the size of the collective group with the given name.
ray.util.collective.collective.get_group_handle	Check if the group is initialized and return the group handle.
ray.util.collective.collective.GroupManager	Use this class to manage the collective groups we created so far.

Collective communication

ray.util.collective.collective.allreduce	Collective allreduce the tensor across the group.
ray.util.collective.collective.allreduce_multigpu	Collective allreduce a list of tensors across the group.
ray.util.collective.collective.barrier	Barrier all processes in the collective group.
ray.util.collective.collective.reduce	Reduce the tensor across the group to the destination rank.
ray.util.collective.collective.reduce_multigpu	Reduce the tensor across the group to the destination rank and destination tensor.
ray.util.collective.collective.broadcast	Broadcast the tensor from a source process to all others.
ray.util.collective.collective.broadcast_multigpu	Broadcast the tensor from a source GPU to all other GPUs.
ray.util.collective.collective.allgather	Allgather tensors from each process of the group into a list.
ray.util.collective.collective.allgather_multigpu	Allgather tensors from each gpus of the group into lists.
ray.util.collective.collective.reducescatter	Reducescatter a list of tensors across the group.
ray.util.collective.collective.reducescatter_multigpu	Reducescatter a list of tensors across all GPUs.

Point-to-point communication

ray.util.collective.collective.send	Send a tensor to a remote process synchronously.
ray.util.collective.collective.send_multigpu	Send a tensor to a remote GPU synchronously.
ray.util.collective.collective.recv	Receive a tensor from a remote process synchronously.
ray.util.collective.collective.recv_multigpu	Receive a tensor from a remote GPU synchronously.

Backends

ray.util.collective.collective.is_backend_available	Check if a collective backend is available.
ray.util.collective.backend_registry.register_collective_backend	Register a custom collective backend with Ray.

Utilities

ray.util.collective.collective.get_address_and_port	Returns the IP address and a free port on this node.
ray.util.collective.collective.synchronize	Synchronize the current process to a give device.