OpenMPI over RoCEE
I'm running OFED-1.5.1 with the RoCEE mlx4 drivers. I can run low level verbs programs ok, but when running open mpi, I'm getting this error. Anybody seen this?
-----
[ompi@escher ~]$ mpirun -np 2 -host 10.192.176.111,10.192.176.112 --mca btl openib,sm,self /usr/mpi/gcc/openmpi-1.4.1/tests/IMB-3.2/IMB-MPI1 -msglen msglen.txt -iter 1000000 pingpong [escher][[36356,1],1][connect/btl_openib_connect_oob.c:325:qp_connect_all] error modifing QP to RTR errno says Invalid argument [escher][[36356,1],1][connect/btl_openib_connect_oob.c:809:rml_recv_cb] error in endpoint reply start connect
--------------------------------------------------------------------------
mpirun has exited due to process rank 1 with PID 4894 on
node escher exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4C3B794E.7010701-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
Does it work with Open MPI v1.4.2?
Quoted 21 lines · Steve Wise
I'm running OFED-1.5.1 with the RoCEE mlx4 drivers. I can run low level verbs programs ok, but when running open mpi, I'm getting this error. Anybody seen this?
-----
[ompi@escher ~]$ mpirun -np 2 -host 10.192.176.111,10.192.176.112 --mca btl openib,sm,self
/usr/mpi/gcc/openmpi-1.4.1/tests/IMB-3.2/IMB-MPI1-msglen msglen.txt -iter 1000000 pingpong [escher][[36356,1],1][connect/btl_openib_connect_oob.c:325:qp_connect_all] error modifing QP to RTR errno says Invalid argument [escher][[36356,1],1][connect/btl_openib_connect_oob.c:809:rml_recv_cb] error in endpoint reply start connect
--------------------------------------------------------------------------
mpirun has exited due to process rank 1 with PID 4894 on
node escher exiting without calling "finalize". This may
have caused other processes in the application to be terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
Signature · 9 lines
Jeff Squyres [email protected] For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 48CED3A4-25F4-4D43-9948-881C0856225B-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org
You know, I got it running by adding this: --mca btl_openib_cpc_include rdmacm
Which basically sez use only the rdmacm to setup the connection.
Thanks,
Steve.
Quoted 30 lines · Jeff Squyres
Does it work with Open MPI v1.4.2?
Quoted 21 lines · Steve Wise
I'm running OFED-1.5.1 with the RoCEE mlx4 drivers. I can run low level verbs programs ok, but when running open mpi, I'm getting this error. Anybody seen this?
-----
[ompi@escher ~]$ mpirun -np 2 -host 10.192.176.111,10.192.176.112 --mca btl openib,sm,self
/usr/mpi/gcc/openmpi-1.4.1/tests/IMB-3.2/IMB-MPI1-msglen msglen.txt -iter 1000000 pingpong [escher][[36356,1],1][connect/btl_openib_connect_oob.c:325:qp_connect_all] error modifing QP to RTR errno says Invalid argument [escher][[36356,1],1][connect/btl_openib_connect_oob.c:809:rml_recv_cb] error in endpoint reply start connect
--------------------------------------------------------------------------
mpirun has exited due to process rank 1 with PID 4894 on
node escher exiting without calling "finalize". This may
have caused other processes in the application to be terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html