allreduce_multigpu

ray.util.collective.collective.allreduce_multigpu(tensor_list: list, group_name: str = 'default', op: ReduceOp = ReduceOp.SUM)

Collective allreduce a list of tensors across the group.

Parameters:

• tensor_list (list) – list of tensors to be allreduced, each on a GPU.

• group_name (str) – the collective group name to perform allreduce.

• op (ReduceOp) – The reduce operation.