reducescatter_multigpu#

ray.util.collective.collective.reducescatter_multigpu(output_tensor_list: list, input_tensor_lists: list, group_name: str = 'default', op: ReduceOp = ReduceOp.SUM)[source]#

Reducescatter a list of tensors across all GPUs.

Parameters:

  • output_tensor_list (list) – the resulted list of tensors, with shape: num_gpus * shape(tensor).

  • input_tensor_lists (list) – the original tensors, with shape: num_gpus * world_size * shape(tensor).

  • group_name (str) – the name of the collective group.

  • op (ReduceOp) – The reduce operation.