pylibwholegraph.torch.comm.create_group_communicator

pylibwholegraph.torch.comm.create_group_communicator(group_size: int = -1, comm_stride: int = 1)

Create WholeMemory Communicator. For example: 24 ranks with group_size = 4 and comm_stride = 2 will create following groups: [0, 2, 4, 6], [1, 3, 5, 7], [8, 10, 12, 14], [9, 11, 13, 15], [16, 18, 20, 22], [17, 19, 21, 23] :param group_size: Size of each group, -1 means to use all ranks in just one single group. :param comm_stride: Stride of each rank in each group :return: WholeMemoryCommunicator