RDMA: fix cross-NIC same-host IPv6 RDMA-CM connect

RDMA-CM cannot establish an IPv6 RoCEv2 connection between two NICs that live on the same host. This shows up on hosts that pin one process per NUMA-local NIC and let those processes talk to each other over each NIC's global IPv6 GID (e.g. a storage daemon with one engine per NUMA node on dual ConnectX-7). rdma_resolve_addr() and ib_send_cm_req() both return success, but the destination NIC silently drops the frame and the peer never sees the REQ; the connection times out.

The bug has two halves, one on each side of the connection:

1) Send side (patch 1, drivers/infiniband/core/addr.c)

When the destination address is local, addr_resolve_neigh() copies the *source* device's MAC into the path record's destination MAC. That is right for true loopback (same netdev), but for a destination that lives on a different netdev of the same host the destination NIC will not accept a frame addressed to the source NIC's MAC and drops it in HW. The fix resolves the netdev that owns the destination address and uses its MAC.

2) Receive side (patch 2, drivers/infiniband/core/cma.c)

Once the REQ does reach the peer, validate_ipv6_net_dev() rejects it:
rt6_lookup() of a same-host destination collapses onto the loopback netdev, so the strict rt6i_idev->dev == net_dev check fails with -EHOSTUNREACH even though the REQ arrived on the right net_dev. The fix accepts an RTF_LOCAL route when net_dev itself owns the listener address. This half is only observable once patch 1 lets the REQ arrive.

Both halves are needed for a working connection; patch 1 alone makes the REQ reach the peer but it is then rejected by the unfixed receive side.

Verification
------------
Measured on two RoCEv2 ConnectX-7 ports on the same host, each with a global IPv6 GID (port A "src", port B "dst"), driving a cross-NIC RDMA-CM connect (rping, src GID on port A -> dst GID on port B) while tracing the destination MAC resolved in addr_resolve():

without the series: resolved dst MAC = port A's MAC (the *source* NIC)

                        -> frame dropped, connect times out
  with the series:     resolved dst MAC = port B's MAC (the *dest* NIC)
                        -> connect completes

The kernel under test carried c31e4038c97f and its dst_rtable() prereq (i.e. the same addr_resolve_neigh()/is_dst_local() shape as for-next); the change applies unmodified to rdma.git for-next.

Note on stable: the Fixes: tags bound the backport to where each construct exists in its current form. Trees predating c31e4038c97f have the equivalent send-side gap in the older IFF_LOOPBACK form of addr_resolve_neigh() and would need a separately shaped backport.

The patches are independent files but should be applied as a pair so the connection works end to end.

Alex Timofeyev (2):
RDMA/core: use destination netdev MAC for cross-NIC same-host local
dst
RDMA/cma: accept cross-NIC same-host local dst in
validate_ipv6_net_dev

 drivers/infiniband/core/addr.c | 22 +++++++++++++++++++---
 drivers/infiniband/core/cma.c  | 15 ++++++++++++++-
 2 files changed, 33 insertions(+), 4 deletions(-)

base-commit: 20ff9350862468af21b46cae2c22d17d6ec637f9

2.40.4

addr_resolve_neigh() treats every is_dst_local() destination as loopback and copies the source device's MAC into the path record's destination MAC (dst_dev_addr <- src_dev_addr). That is correct for true loopback (source and destination on the same netdev), but wrong when the local destination address lives on a different netdev of the same host.

In that cross-NIC same-host case the destination NIC will not accept a frame whose destination MAC is the source NIC's MAC, and drops it in hardware before it reaches the peer. rdma_resolve_addr() and ib_send_cm_req() both return success, but the CM REQ never arrives and the connection times out.

Look up the netdev that owns the destination address and copy its MAC into dst_dev_addr instead. Fall back to the source MAC when no netdev claims the address (true loopback), preserving the existing behaviour.

This was observed with two RoCEv2 ConnectX-7 ports on the same host, each holding a global IPv6 GID, when one process pinned per NUMA NIC connected to the other over RDMA-CM: the resolved destination MAC was the source port's MAC instead of the destination port's, and the REQ was silently dropped. With the fix the resolved MAC is the destination port's and the connection completes.

---
 drivers/infiniband/core/addr.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

index 7e62b5b1ffaa..84aa43436bfe 100644
@@ -451,10 +451,26 @@ static int addr_resolve_neigh(const struct dst_entry *dst,
 			      u32 seq)
 {
 	if (is_dst_local(dst)) {
-		/* When the destination is local entry, source and destination
-		 * are same. Skip the neighbour lookup.
+		struct net_device *dst_ndev;
+
+		/* When the destination is local, source and destination are on
+		 * the same host. For true loopback (same netdev) the source and
+		 * destination MACs are equal, but when the destination address
+		 * lives on a different netdev of the same host the destination
+		 * MAC must be that netdev's MAC -- otherwise the destination NIC
+		 * silently drops the frame. Look up the netdev that owns the
+		 * destination address and copy its MAC; fall back to the source
+		 * MAC if no netdev claims the address.
 		 */
-		memcpy(addr->dst_dev_addr, addr->src_dev_addr, MAX_ADDR_LEN);
+		rcu_read_lock();
+		dst_ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(dst->dev), dst_in);
+		if (!IS_ERR(dst_ndev))
+			memcpy(addr->dst_dev_addr, dst_ndev->dev_addr,
+			       MAX_ADDR_LEN);
+		else
+			memcpy(addr->dst_dev_addr, addr->src_dev_addr,
+			       MAX_ADDR_LEN);
+		rcu_read_unlock();
 		return 0;
 	}
 

2.40.4

validate_ipv6_net_dev() confirms an incoming CM REQ was delivered on the correct net_dev with an rt6_lookup() that requires rt->rt6i_idev->dev == net_dev. For an IPv6 destination that is local to a different netdev of the same host, the FIB resolves the lookup onto the loopback netdev, so rt6i_idev->dev is lo regardless of which physical netdev owns the listener address. The strict comparison then rejects the REQ with -EHOSTUNREACH even though it was correctly delivered on net_dev.

Accept the request when the resolved route is RTF_LOCAL and net_dev itself owns the address the listener was bound to (src_addr). This is the receive-side counterpart to the cross-NIC same-host send-side fix in addr_resolve_neigh().

---
 drivers/infiniband/core/cma.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

index 9480d1a51c11..872c57943362 100644
@@ -1635,7 +1635,20 @@ static bool validate_ipv6_net_dev(struct net_device *net_dev,
 	if (!rt)
 		return false;
 
-	ret = rt->rt6i_idev->dev == net_dev;
+	if (rt->rt6i_idev->dev == net_dev) {
+		ret = true;
+	} else if (rt->rt6i_flags & RTF_LOCAL) {
+		/* For a destination that is local to another netdev of the same
+		 * host, the FIB collapses the lookup onto the loopback netdev,
+		 * so rt6i_idev->dev is not net_dev even though the request was
+		 * correctly delivered on net_dev. Accept it when net_dev itself
+		 * owns the address we were listening on.
+		 */
+		ret = ipv6_chk_addr(dev_net(net_dev), &src_addr->sin6_addr,
+				    net_dev, 1);
+	} else {
+		ret = false;
+	}
 	ip6_rt_put(rt);
 
 	return ret;

2.40.4

I'm not sure about this, you need to have policy routing or VRF setup so these local routes don't show up.. Do you have that?

A local route result should result only in a local loopback AH, it should never result in a packet on the wire, and we shouldn't be trying to mangle loopback routes at all.

Same answer here, if you have proper routing you won't get a loopback route to match and you won't fail on this check. Removing the check does not seem correct.

Jason

You're right, and thanks for pushing back on this.

I've been testing VRF all day and it solves the problem completely, with no kernel changes. Putting each NIC in its own VRF moves that NIC's addresses out of the global v6 local table (255), so the cross-NIC same-host destination no longer collapses to lo -- it resolves on-wire to the real egress device:

  # enp49s0np0 -> vrfdata0 (table 100), enp193s0np0 -> vrfdata1 (101)
  $ ip -6 route get <dst-on-other-local-nic> from <src> oif enp49s0np0
  ... dev enp49s0np0 ...        # was: local ... dev lo

So neither addr_resolve_neigh() nor validate_ipv6_net_dev() ever hits the local shortcut my patches were rewriting. The check stays correct and the loopback routes stay untouched, exactly as you said.

I validated it with the actual workload that motivated the series, not just rping: a 3-node DAOS cluster (two engines per host, one per NUMA NIC) on a kernel WITHOUT these two patches -- only the upstream prereqs fa29d1e8877b + c31e4038c97f. All 6 ranks join and stay joined, and a pool created across all of them comes up healthy. That covers both the same-host cross-NIC path and the cross-host path, all over v6 RoCEv2.

One observation in case it's useful to others who hit this: VRF self-RPC relies on c31e4038c97f -- once an address is homed in a VRF table, is_dst_local() needs the RTF_LOCAL test rather than the old IFF_LOOPBACK one to still recognize it as local. That commit is in v6.18 but not in linux-6.6.y, which is the stable base we (and presumably other RoCE-on-6.6 users) are on; might be worth a stable backport on its own merits, independent of this series.

Given all that, please drop this series -- I'm withdrawing it. The right fix here is configuration, not a kernel patch. Appreciate the review and the steer toward VRF.

Thanks,
Alex

Feel free to propose backports to stable if it is valuable to you, RDMA maintainers are not managing stable at all, so you are welcome to do so.

Jason