bug 1918 - openmpi broken due to rdma-cm changes
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines
OpenMPI uses
rdma_bind_addr()to figure out which ip addresses are valid
for which IB devices. This logic is now broken. Regardless of whether
OpenMPI should use another method for determining which IP address belong to which interfaces, we should probably rethink whether we're breaking rdma-cm semantics in a bad way on a point release.
The changes to the rdma_cm have been merged upstream. These were fixes specifically to enable using the loopback address with RDMA devices.
At first thought, we can extend enum ib_device_cap_flags to indicate if a device supports loopback capabilities or not. The rdma_cm could then skip over such devices when dealing with a loopback address.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): DE586AB7003E4FC4B3850004A0B3CAF6-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org
Quoted 7 lines · Sean Hefty
Quoted 6 lines
OpenMPI uses
rdma_bind_addr()to figure out which ip addresses are valid
for which IB devices. This logic is now broken. Regardless of whether
OpenMPI should use another method for determining which IP address belong to which interfaces, we should probably rethink whether we're breaking rdma-cm semantics in a bad way on a point release.
Quoted 9 lines
The changes to the
rdma_cmhave been merged upstream. These were fixes specifically to enable using the loopback address with RDMA devices.At first thought, we can extend enum
ib_device_cap_flagsto indicate if a device supports loopback capabilities or not. Therdma_cmcould then skip over such devices when dealing with a loopback address.- Sean
But how can you determine _which_ rdma device should be used if and app binds to 127.0.0.1? I think this is busted...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4B6B4C9B.8070804-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
Quoted 2 lines
But how can you determine _which_ rdma device should be used if and app binds to 127.0.0.1? I think this is busted...
The code just picks the first rdma device available. To me, this is preferable than simply disallowing the loopback device from working at all. I personally use it all the time, so I don't have to figure out what the ip address is of the system that I'm trying to test on.
Loopback support has always been in the rdma_cm and was intended to work; it just didn't work very well...
A message that is not in the archive (sent off-list or not yet received): EEF75E01C32544C688989E66FECDC422-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org
Quoted 9 lines · Sean Hefty
Quoted 3 lines
But how can you determine _which_ rdma device should be used if and app binds to 127.0.0.1? I think this is busted...
The code just picks the first rdma device available. To me, this is preferable than simply disallowing the loopback device from working at all. I personally use it all the time, so I don't have to figure out what the ip address is of the system that I'm trying to test on.
Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop...
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4B6B4EFE.3010205-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
Quoted 2 lines
Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop...
The problem sounds like it's more than just whether 127.0.0.1 is usable. That check may fix openmpi, but it sounds more like the app needs to know whether the device can actually support loopback, regardless of what addresses are used. Is this correct?
What would openmpi do if there were two addresses assigned to the T3 device? Does openmpi simply bypass RDMA for all connections on the local machine?
Basically, I'm not sure that this is *just* an rdma_cm issue. Although it definitely appears that some sort of change needs to be made to the rdma_cm.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): DB361897EE7C40C5AB494AAC2D8625AC-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org
Quoted 11 lines · Sean Hefty
Quoted 3 lines
Well then the rdma-cm needs to know which devices support hw loopback. Cuz on a T3-only system, no hwloop...
The problem sounds like it's more than just whether 127.0.0.1 is usable. That check may fix openmpi, but it sounds more like the app needs to know whether the device can actually support loopback, regardless of what addresses are used. Is this correct?
What would openmpi do if there were two addresses assigned to the T3 device?
It would use them and might even create two connections.
Quoted 3 lines
Does openmpi simply bypass RDMA for all connections on the local machine?
OpenMPI can be run to use hw loopback if its available. For T3 clusters, OMPI is run in a mode to use shared memory for intra-node communications.
Quoted 4 lines
Basically, I'm not sure that this is *just* an
rdma_cmissue. Although it definitely appears that some sort of change needs to be made to therdma_cm.
I think the OpenMPI rdmacm code needs to skip 127.0.0.1, in this particular case. Prior to ofed-1.5.1, however, the bind would fail and thus OpenMPI would not advertise 127.0.0.1 to its peer. I will work to get that change done.
But lets also add a device attribute so the rdmacm can know if a device supports loopback. Clearly, if the rdma-cm allows binds to T3, loopback connections will fail at connect time.
Hey Roland, are you ok with a device attribute to indicate hw-loopback support?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines
Hey Roland, are you ok with a device attribute to indicate hw-loopback support?
Sigh, I guess so. Can we have the rdma-cm handle this somewhat automagically, eg only choose devices that do handle loopback when binding/connecting to 127.0.0.1? Or maybe can we put the handling of this into the device-driver-specific iwcm methods, to fail listen/connect requests that are loopback?
But if a device cap flag is what it takes, that's OK I guess.
Signature · 9 lines
Roland Dreier <[email protected]> Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): aday6j8efqs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 6 lines · Roland Dreier
Quoted 2 lines
Hey Roland, are you ok with a device attribute to indicate hw-loopback support?
Sigh, I guess so. Can we have the rdma-cm handle this somewhat automagically, eg only choose devices that do handle loopback when binding/connecting to 127.0.0.1?
That's the plan.
Quoted 4 lines
Or maybe can we put the handling of
this into the device-driver-specific iwcm methods, to fail listen/connect requests that are loopback?
Is this only an iwarp issue? IE do all IB devices support hw loopback? And will all future devices support it (IE is it an IBTA requirement)?
If so, then we could add a iwcm capabilities field maybe. But currently there's no bind method. We could add a bind method I suppose, but I'd rather do it with a simple capabilities.
Quoted 2 lines
But if a device cap flag is what it takes, that's OK I guess.
IMO the device cap flag is the way to go, and the rdma-cm should never bind 127.0.0.1 to a device that doesn't support loopback.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4B6B5BD7.9090301-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
Quoted 3 lines
Is this only an iwarp issue? IE do all IB devices support hw loopback? And will all future devices support it (IE is it an IBTA requirement)?
I do think IBA requires loopback to work. Can't quote chapter & verse off the top of my head.
Signature · 9 lines
Roland Dreier <[email protected]> Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adatytwef1i.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
I can. Chapter 17 verse 3.1
17.3.1 Loopback
"An HCA shall be able to internally loopback a packet sent to itself. That is,
the verbs layer can specify a packet to be delivered to the same port (possibly
a different QP though). The packet shall be delivered without the packet appearing on the port's physical link. This loopback shall be able to function without requiring the presence of an external switch. InfiniBand does not reserve a special LID value to indicate loopback. Instead,
the DLID (and DGID if present) of a loopback packet should be the LID (and GID) of the port on which the packet was emitted. For loopback packets, a channel adapter implementation may ignore other path information, such as MTU, that is not otherwise needed for the receive buffer or for the completion queue as specified in section 11.4.2.1 Poll for Completion
on page 629."
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Roland Dreier
Sent: Thursday, February 04, 2010 3:51 PM
To: Steve Wise
Cc: Sean Hefty; linux-rdma; OpenFabrics EWG; Jeff Squyres
Subject: Re: bug 1918 - openmpi broken due to rdma-cm changes
Quoted 3 lines
Is this only an iwarp issue? IE do all IB devices support hw loopback? And will all future devices support it (IE is it an IBTA requirement)?
I do think IBA requires loopback to work. Can't quote chapter & verse off the top of my head.
Signature · 9 lines
Roland Dreier <[email protected]> Cisco.com - http://www.cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Sean Hefty
At first thought, we can extend enum
ib_device_cap_flagsto indicate if a device supports loopback capabilities or not. Therdma_cmcould then skip over such devices when dealing with a loopback address.
This solution would work. Will you code it up?
Stevo
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4B6B5048.5020707-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org
Quoted 1 line
This solution would work. Will you code it up?
I can do that. I just want to make sure that we address the full scope of the problem.
- Sean