IB/cm: Consider local communication ID when check for a stale connection

From: Mark Zhang <[email protected]>

Made the check for duplicate/stale CM more strict by adding comparison for remote communication ID field.

The absence of such strict check causes to the following flows not being handled properly:
1. Client tries to setup more than one connection with same QP in a server (e.g., when use external QP), the client would reject the reply.
2. Client node reboots, and when it gets the same QP number as that of before the reboot time, the server would rejects the request.

---
 drivers/infiniband/core/cm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

index 0d1377232933..23dfba947357 100644
@@ -775,12 +775,17 @@ static struct cm_timewait_info * cm_insert_remote_qpn(struct cm_timewait_info
 	struct cm_timewait_info *cur_timewait_info;
 	__be64 remote_ca_guid = timewait_info->remote_ca_guid;
 	__be32 remote_qpn = timewait_info->remote_qpn;
+	__be32 remote_id = timewait_info->work.remote_id;
 
 	while (*link) {
 		parent = *link;
 		cur_timewait_info = rb_entry(parent, struct cm_timewait_info,
 					     remote_qp_node);
-		if (be32_lt(remote_qpn, cur_timewait_info->remote_qpn))
+		if (be32_lt(remote_id, cur_timewait_info->work.remote_id))
+			link = &(*link)->rb_left;
+		else if (be32_gt(remote_id, cur_timewait_info->work.remote_id))
+			link = &(*link)->rb_right;
+		else if (be32_lt(remote_qpn, cur_timewait_info->remote_qpn))
 			link = &(*link)->rb_left;
 		else if (be32_gt(remote_qpn, cur_timewait_info->remote_qpn))
 			link = &(*link)->rb_right;

2.26.2

This is correct and required behavior by IBA:

12.9.8.3.1 REQ RECEIVED / REP RECEIVED

(RC, UC, XRC) A CM may receive a REQ/REP specifying a remote QPN in “REQ:local QPN”/”REP:local QPN” that the CM already considers connected to a local QP.

IBA has a specific flow that should be followed for this case:

When a CM receives such a REQ/REP it shall abort the connection establishment by issuing REJ to the REQ/REP. It shall then issue DREQ, with “DREQ:remote QPN” set to the remote QPN from the REQ/REP, until DREP is received or Max Retries is exceeded, and place the local QP in the TimeWait state.

The fundmental issue is IBA does not include the SRC QPN in any RC packets. It is the responsiblity of each end port to ensure that only one RC QP is setup sending traffic to a given DLID/DQPN pair.

This is what the timewait mechanism and these checks in the CM are for. The proper CM use will ensure that the local QP targetting the DLID/DQPN is destroyed before the cm_id enters timedwait, and the timedwait will prevent a new QP from being established with the same parameters until the network has flushed all packets related to the old sending QP.

Jason