issues with the rdma-cm server side mapping of IP to GID
Hi Sean,
We came across a pretty deadly situation with rdma-cm based client/server application where the client set their RC QP to send to HCA X on the server node but the server app opened their QP on HCA Y. The result was un-acked RC packets and RC session failure.
This happened because the mapping between destination IP to destination GID as seen by the client was different from what's present in the server IP stack at the time the connection request arrived -- the server side rdma-cm IP --> GID mapping is established by the cma_translate_addr() call in cma_new_conn_id() which is done on the destination IP taken from the RDMA-CM header in the CM REQ.
Such situation can happen in the following cases:
1. net.ipv4.conf.default.arp_ignore equals 0 (the default)
2. server side bonding/teaming fail-over when the Gratitous ARP sent was lost
3. re-order of ibM net-devices mapping to HCA PCI devices after server boot/crash
4. etc more
Basically, when the rdma-cm observes difference between the destination GID as present in the IB path within
the CM REQ to the one resolved locally, we should at least print a warning. Perhaps, we should reject the connection request? (in that case, I wasn't sure what would be the appropriate reject reason), any more ideas?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 530C51EF.2000509-VPRAkNaXOzVWk0Htik3J/[email protected]
Quoted 9 lines
Such situation can happen in the following cases:
1. net.ipv4.conf.default.arp_ignore equals 0 (the default)
2. server side bonding/teaming fail-over when the Gratitous ARP sent was lost
3. re-order of ibM net-devices mapping to HCA PCI devices after server boot/crash
4. etc more
Quoted 6 lines
Basically, when the rdma-cm observes difference between the destination GID as present in the IB path within
the CM REQ to the one resolved locally, we should at least print a warning. Perhaps, we should reject the connection request? (in that case, I wasn't sure what would be the appropriate reject reason), any more ideas?
I'm not sure that this results in a single error case.
Can the kernel rdma_cm check for net.ipv4.default.arp_ignore on startup and at least print a warning if that is wrong? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237388D1F0BD-P5GAC/[email protected]
Quoted 11 lines · Sean
Quoted 10 lines
Such situation can happen in the following cases:
1. net.ipv4.conf.default.arp_ignore equals 0 (the default)
2. server side bonding/teaming fail-over when the Gratitous ARP sent was lost
3. re-order of ibM net-devices mapping to HCA PCI devices after server boot/crash
4. etc moreBasically, when the rdma-cm observes difference between the destination GID as present in the IB path within the CM REQ to the one resolved locally, we should at least print a warning. Perhaps, we should reject the connection request? (in that case, I wasn't sure what would be the appropriate reject reason), any more ideas?
I'm not sure that this results in a single error case.
Sorry... I'm not sure to follow, can you elaborate a bit more?
Quoted 1 line
Can the kernel
rdma_cmcheck for net.ipv4.default.arp_ignore on startup and at least print a warning if that is wrong?
I am not sure, and anyway, please note that I brought at least two more use cases where the problem happens
- following server side bonding fail-over
- following server side reboot after which the PCI ordering changes between two HCAs and hence ibM devices change their PCI association
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 5312F107.2000404-VPRAkNaXOzVWk0Htik3J/[email protected]
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
1. net.ipv4.conf.default.arp_ignore equals 0 (the default)
2. server side bonding/teaming fail-over when the Gratitous ARP sent was lost
3. re-order of ibM net-devices mapping to HCA PCI devices after server boot/crash
4. etc more
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Basically, when the rdma-cm observes difference between the destination
GID as present in the IB path within the CM REQ to the one resolved locally, we should at least print a warning. Perhaps, we should reject the connection request? (in that case, I wasn't sure what would be the appropriate reject reason), any more ideas?
Quoted 1 line
I'm not sure that this results in a single error case.
Sorry... I'm not sure to follow, can you elaborate a bit more?
We don't know what type of device responds to the ARP query. It could come from an ethernet device.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237388D1F39F-P5GAC/[email protected]
Quoted 5 lines · Sean
Quoted 3 lines
Quoted 1 line
I'm not sure that this results in a single error case.
Sorry... I'm not sure to follow, can you elaborate a bit more?
We don't know what type of device responds to the ARP query. It could come from an ethernet device.
Yep, this issue is really nasty..., but wait, you mentioned Ethernet, well, if the fabric is IB we do know that the GID in the REQ belongs to an HCA of that server node, b/c the client did route (== path query) resolution based on this DGID and their CM REQ landed in our hands, right?
We could come and say that we adhere to the IP --> GID mapping as present in the CM REQ (GID in the path, IP in the CMA header) and associate the newly created CMA ID with the device/port where this GID resides, no matter what the local IP stack has to say. This would work, but for people that seek HA for their apps, multiple sessions can be created over the same server hca/port where they wanted them to be spreaded... what we can when such inconsistency is observed by the rdma-cm is the following
1. print warning to the system log
2. reject the connection request
3. send Gratuitous ARP to update client nodes IPoIB neighbour IP --> GID mapping
I suggest that we 1st debate/agree on something that makes sense with IB and later see how it would work for RoCE
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZ+ZC62FeZCy17ZMkzkxqrTdTrNRDs+nWQQ4Xjb9Sx5T3A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 22 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I'm not sure that this results in a single error case.
Sorry... I'm not sure to follow, can you elaborate a bit more?
We don't know what type of device responds to the ARP query. It could
come from an ethernet device.
Yep, this issue is really nasty..., but wait, you mentioned Ethernet, well, if the fabric is IB we do know that the GID in the REQ belongs to an HCA of that server node, b/c the client did route (== path query) resolution based on this DGID and their CM REQ landed in our hands, right?
We could come and say that we adhere to the IP --> GID mapping as present in the CM REQ (GID in the path, IP in the CMA header) and associate the newly created CMA ID with the device/port where this GID resides, no matter what the local IP stack has to say. This would work, but for people that seek HA for their apps, multiple sessions can be created over the same server hca/port where they wanted them to be spreaded... what we can when such inconsistency is observed by the rdma-cm is the following
Quoted 7 lines
1. print warning to the system log
2. reject the connection request
3. send Gratuitous ARP to update client nodes IPoIB neighbour IP --> GID mappingI suggest that we 1st debate/agree on something that makes sense with IB and later see how it would work for RoCE
I wasn't referring to RoCE. I was simply saying that this problem may result in multiple errors. It's possible that we may never reach the point of sending a CM REQ. It seems best to try to detect this problem as early as possible.
If a CM REQ actually gets to the remote side, it could be rejected as an invalid GID, and the client could retry the request. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237388D1F440-P5GAC/[email protected]
On Mon, Mar 3, 2014 at 7:15 PM, Hefty, Sean <[email protected]> wrote:
[...]
Quoted 1 line
If a CM REQ actually gets to the remote side, it could be rejected as an invalid GID, and the client could retry the request.
Retrying the request is practically calling
1. rdma_resolve_addr
2. rdma_resolve_route
3. rdma_connect
where step #1 would get wrong GID from the client arp cache and hence step #3 would result in a reject and so on...
I am starting to think that the rdma address resolution must not make use of the arp cache, or at least provide applications
the API to dictate that ARP request must be sent, how this sounds? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZKT0oGfx99PhX9OP5_qaa2QyRVEJqE+hSkM7tykN23GOw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 8 lines
Quoted 1 line
If a CM REQ actually gets to the remote side, it could be rejected as an
invalid GID, and the client could retry the request.
Retrying the request is practically calling
1.
rdma_resolve_addr
2.rdma_resolve_route
3.rdma_connect
Yep - the entire setup is broken. If the wrong remote GID was resolved, then the wrong local GID _may_ have been selected. There's no easy guaranteed recovery here.
Quoted 5 lines
where step #1 would get wrong GID from the client arp cache and hence step #3 would result in a reject and so on...
I am starting to think that the rdma address resolution must not make use of the arp cache, or at least provide applications
the API to dictate that ARP request must be sent, how this sounds?
Clients should not be made aware of how the resolution was done. The RDMA CM needs to abstract that problem. Alternate mechanisms may be usable, but there aren't exactly a whole lot of options available. The client could use native IB addressing at this point if it does not want to rely on address translation.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237388D200B2-P5GAC/[email protected]
Quoted 6 lines
Quoted 1 line
Quoted 1 line
If a CM REQ actually gets to the remote side, it could be rejected as
an
Quoted 4 lines
invalid GID, and the client could retry the request.
Retrying the request is practically calling
Quoted 7 lines
Quoted 3 lines
1.
rdma_resolve_addr
2.rdma_resolve_route
3.rdma_connectYep - the entire setup is broken. If the wrong remote GID was resolved, then the wrong local GID _may_ have been selected. There's no easy guaranteed recovery here.
On second thought, I don't think this is true. The SGID is selected based on the IP address, not the DGID. If the remote CM rejects the connection, the RDMA CM may be able to recover by restarting at step 2. Query for a new PR, then re-issue the connect request. It may be possible to do this without the application's involvement. I'm not sure how the librdmacm would handle this, since the initial resolve_route would be redone. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Sean
Clients should not be made aware of how the resolution was done. The RDMA CM needs to abstract that problem. Alternate mechanisms may be usable, but there aren't exactly a whole lot of options available. The client could use native IB addressing at this point if it does not want to rely on address translation.
No, native IB addressing isn't an option for this and many other applications. they want to use IP addressing. We have to make sure that our IP --> RDMA address resolving which uses neighbour lookup is robust under the real life schemes I brought which makes the RC connection / UD session established based on this resolution to be totally broken.
So given the fact that we have a solution for large scale through native IB addressing that offloads the address and route resolution -- what's your thinking on my suggestion to make address resolution for rdma-cm endpoints creates with the non native IB port space to use solicit neighbour lookup (e.g avoid arp cache usage?)
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html