How to send data use mpi4py across multiple nodes?

everyone: I run my code on the hpc,but I can't transmit data between nodes. I write a simple code to test the communications between cores across nodes. First, I use one node 8 cores, my code is(test.py)

from mpi4py import MPI import sys import numpy as np def print_hello(rank, size, name): msg = "Hello World! I am process {0} of {1} on {2}.\n" sys.stdout.write(msg.format(rank, size, name)) if __name__ == "__main__": comm = MPI.COMM_WORLD size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() if rank == 0: data = np.random.random((8,64,64,64)) print(data.shape) else: data = None data = comm.scatter(data,root=0) print(data.shape) print_hello(rank, size, name) 

and I run it use the srun -N 1 -n 8 python3 test.py 2>&1 | tee out.txt, it just like mpirun -np 8 python3 test.py 1>&1 | tee out.txt, it runs only 5 seconds The out.txt file is :

(64, 64, 64) Hello World! I am process 4 of 8 on cn3478. (64, 64, 64) Hello World! I am process 5 of 8 on cn3478. (64, 64, 64) Hello World! I am process 6 of 8 on cn3478. (64, 64, 64) Hello World! I am process 7 of 8 on cn3478. (64, 64, 64) Hello World! I am process 1 of 8 on cn3478. (64, 64, 64) Hello World! I am process 2 of 8 on cn3478. (64, 64, 64) Hello World! I am process 3 of 8 on cn3478. (8, 64, 64, 64) (64, 64, 64) Hello World! I am process 0 of 8 on cn3478. 

everything looks good! However, When I use two nodes 48 cores, it went wrong! The file is that(test48.py):

from mpi4py import MPI import sys import numpy as np def print_hello(rank, size, name): msg = "Hello World! I am process {0} of {1} on {2}.\n" sys.stdout.write(msg.format(rank, size, name)) if __name__ == "__main__": comm = MPI.COMM_WORLD size = MPI.COMM_WORLD.Get_size() rank = MPI.COMM_WORLD.Get_rank() name = MPI.Get_processor_name() if rank == 0: data = np.random.random((48,64,64,64)) print(data.shape) else: data = None data = comm.scatter(data,root=0) print(data.shape) print_hello(rank, size, name) 

I run it through yhrun -N 2 -n 48 python3 test48.py 2>&1 | tee out2.txt It runs a lot of time(more than 1 hour) and did't print anything. I guess that the data transmission went wrong because I comment out the two lines:

 data = comm.scatter(data,root=0) print(data.shape) 

The code is completed quickly and the output is :

Hello World! I am process 25 of 48 on cn3598. Hello World! I am process 28 of 48 on cn3598. Hello World! I am process 30 of 48 on cn3598. Hello World! I am process 31 of 48 on cn3598. Hello World! I am process 32 of 48 on cn3598. Hello World! I am process 40 of 48 on cn3598. Hello World! I am process 41 of 48 on cn3598. Hello World! I am process 44 of 48 on cn3598. Hello World! I am process 24 of 48 on cn3598. Hello World! I am process 26 of 48 on cn3598. Hello World! I am process 27 of 48 on cn3598. Hello World! I am process 29 of 48 on cn3598. Hello World! I am process 33 of 48 on cn3598. Hello World! I am process 34 of 48 on cn3598. Hello World! I am process 35 of 48 on cn3598. Hello World! I am process 36 of 48 on cn3598. Hello World! I am process 37 of 48 on cn3598. Hello World! I am process 38 of 48 on cn3598. Hello World! I am process 42 of 48 on cn3598. Hello World! I am process 43 of 48 on cn3598. Hello World! I am process 45 of 48 on cn3598. Hello World! I am process 46 of 48 on cn3598. Hello World! I am process 47 of 48 on cn3598. Hello World! I am process 39 of 48 on cn3598. Hello World! I am process 1 of 48 on cn3597. Hello World! I am process 3 of 48 on cn3597. Hello World! I am process 4 of 48 on cn3597. Hello World! I am process 9 of 48 on cn3597. Hello World! I am process 12 of 48 on cn3597. Hello World! I am process 15 of 48 on cn3597. Hello World! I am process 16 of 48 on cn3597. Hello World! I am process 17 of 48 on cn3597. Hello World! I am process 20 of 48 on cn3597. Hello World! I am process 2 of 48 on cn3597. Hello World! I am process 5 of 48 on cn3597. Hello World! I am process 6 of 48 on cn3597. Hello World! I am process 7 of 48 on cn3597. Hello World! I am process 8 of 48 on cn3597. Hello World! I am process 10 of 48 on cn3597. Hello World! I am process 11 of 48 on cn3597. Hello World! I am process 13 of 48 on cn3597. Hello World! I am process 14 of 48 on cn3597. Hello World! I am process 18 of 48 on cn3597. Hello World! I am process 19 of 48 on cn3597. Hello World! I am process 21 of 48 on cn3597. Hello World! I am process 22 of 48 on cn3597. Hello World! I am process 23 of 48 on cn3597. (48, 64, 64, 64) Hello World! I am process 0 of 48 on cn3597. 

What's wrong is the code? or the data transmission between nodes is not allowed? Any suggestion is welcomed!

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.