Skip to contentSkip to conversation

Subject: RDMA/rxe: Handle zero length rdma

PATCH Applied Bob Pearson 5 participants 11 messages Feb 2, 2023 – Feb 16, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 Subject: RDMA/rxe: Handle zero length rdma

Currently the rxe driver does not handle all cases of zero length rdma operations correctly. The client does not have to provide an rkey for zero length RDMA read or write operations so the rkey provided may be invalid and should not be used to lookup an mr.

This patch corrects the driver to ignore the provided rkey if the reth length is zero for read or write operations and make sure to set the mr to NULL. In read_reply() if length is zero rxe_recheck_mr() is not called. Warnings are added in the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.

---
v4:
Fixed a regression in flush operations because the rkey must be valid in that case even if the length is zero which maybe the case for selectivity level of the entire memory region.

v3:
Fixed my fat finger typing on v2. Mangled the patch.

v2:
Rebased to current for-next.
Cleaned up description to be a little more accurate.

---
 drivers/infiniband/sw/rxe/rxe_mr.c   |  7 ++++
 drivers/infiniband/sw/rxe/rxe_resp.c | 59 +++++++++++++++++++++-------
 2 files changed, 51 insertions(+), 15 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +7 -0
index c80458634962..5e9a03831bf9 100644
@@ -314,6 +314,9 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr,
 	if (length == 0)
 		return 0;
 
+	if (WARN_ON(!mr))
+		return -EINVAL;
+
 	if (mr->ibmr.type == IB_MR_TYPE_DMA) {
 		rxe_mr_copy_dma(mr, iova, addr, length, dir);
 		return 0;
@@ -432,6 +435,10 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 	int err;
 	u8 *va;
 
+	/* mr must be valid even if length is zero */
+	if (WARN_ON(!mr))
+		return -EINVAL;
+
 	if (length == 0)
 		return 0;
 
drivers/infiniband/sw/rxe/rxe_resp.c +44 -15

125 lines not on this page · raw message

base-commit: 01884059c09fa770206ca03bf3660502f97d5fdd

Signature · 1 line
2.37.2
#1 lore raw
Quoted 12 lines · Bob Pearson

Currently the rxe driver does not handle all cases of zero length rdma operations correctly. The client does not have to provide an rkey for zero length RDMA read or write operations so the rkey provided may be invalid and should not be used to lookup an mr.

This patch corrects the driver to ignore the provided rkey if the reth length is zero for read or write operations and make sure to set the mr to NULL. In read_reply() if length is zero rxe_recheck_mr() is not called. Warnings are added in the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.

<...>

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) · line 440

Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_mr.c +3 -0
@@ -432,6 +435,10 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 	int err;
 	u8 *va;
 
+	/* mr must be valid even if length is zero */
+	if (WARN_ON(!mr))
+		return -EINVAL;

If 'mr' is NULL, NULL pointer dereference can occur in process_flush() before reaching here. Isn't it better to do the check in process_flush()?

Quoted 18 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -0
+
 	if (length == 0)
 		return 0;
 
drivers/infiniband/sw/rxe/rxe_resp.c +5 -0
index cccf7c6c21e9..c8e7b4ca456b 100644
@@ -420,13 +420,23 @@ static enum resp_states rxe_resp_check_length(struct rxe_qp *qp,
 	return RESPST_CHK_RKEY;
 }
 
+/* if the reth length field is zero we can assume nothing
+ * about the rkey value and should not validate or use it.
+ * Instead set qp->resp.rkey to 0 which is an invalid rkey
+ * value since the minimum index part is 1.
+ */
 static void qp_resp_from_reth(struct rxe_qp *qp, struct rxe_pkt_info *pkt)

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states rxe_resp_check_length(struct rxe_qp *qp, · line 435

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -3
 {
+	unsigned int length = reth_len(pkt);
+
 	qp->resp.va = reth_va(pkt);
 	qp->resp.offset = 0;
-	qp->resp.rkey = reth_rkey(pkt);
-	qp->resp.resid = reth_len(pkt);
-	qp->resp.length = reth_len(pkt);
+	qp->resp.resid = length;
+	qp->resp.length = length;

As you know, the comment above this function is applicable only to RDMA Read and Write. What do you say to mentioning FLUSH here rather than at the one in rxe_flush_pmem_iova().

Thanks,
Daisuke

Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+	if (pkt->mask & RXE_READ_OR_WRITE_MASK && length == 0)
+		qp->resp.rkey = 0;
+	else
+		qp->resp.rkey = reth_rkey(pkt);
 }
 

<...>

#2 lore raw
Quoted 63 lines · Daisuke Matsuda (Fujitsu)
Quoted 12 lines · Bob Pearson

Currently the rxe driver does not handle all cases of zero length rdma operations correctly. The client does not have to provide an rkey for zero length RDMA read or write operations so the rkey provided may be invalid and should not be used to lookup an mr.

This patch corrects the driver to ignore the provided rkey if the reth length is zero for read or write operations and make sure to set the mr to NULL. In read_reply() if length is zero rxe_recheck_mr() is not called. Warnings are added in the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.

<...>

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) · line 440

Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_mr.c +3 -0
@@ -432,6 +435,10 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 	int err;
 	u8 *va;
 
+	/* mr must be valid even if length is zero */
+	if (WARN_ON(!mr))
+		return -EINVAL;

If 'mr' is NULL, NULL pointer dereference can occur in process_flush() before reaching here. Isn't it better to do the check in process_flush()?

Quoted 18 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -0
+
 	if (length == 0)
 		return 0;
 
drivers/infiniband/sw/rxe/rxe_resp.c +5 -0
index cccf7c6c21e9..c8e7b4ca456b 100644
@@ -420,13 +420,23 @@ static enum resp_states rxe_resp_check_length(struct rxe_qp *qp,
 	return RESPST_CHK_RKEY;
 }
 
+/* if the reth length field is zero we can assume nothing
+ * about the rkey value and should not validate or use it.
+ * Instead set qp->resp.rkey to 0 which is an invalid rkey
+ * value since the minimum index part is 1.
+ */
 static void qp_resp_from_reth(struct rxe_qp *qp, struct rxe_pkt_info *pkt)

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states rxe_resp_check_length(struct rxe_qp *qp, · line 435

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -3
 {
+	unsigned int length = reth_len(pkt);
+
 	qp->resp.va = reth_va(pkt);
 	qp->resp.offset = 0;
-	qp->resp.rkey = reth_rkey(pkt);
-	qp->resp.resid = reth_len(pkt);
-	qp->resp.length = reth_len(pkt);
+	qp->resp.resid = length;
+	qp->resp.length = length;

As you know, the comment above this function is applicable only to RDMA Read and Write. What do you say to mentioning FLUSH here rather than at the one in rxe_flush_pmem_iova().

Thanks,
Daisuke

Quoted 9 lines
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+	if (pkt->mask & RXE_READ_OR_WRITE_MASK && length == 0)
+		qp->resp.rkey = 0;
+	else
+		qp->resp.rkey = reth_rkey(pkt);
 }
 

<...>

Daisuke,

The updated patch no longer sets mr = NULL for flush. This check is mainly to defend against someone changing it again in the future and is near where mr is dereferenced. You are correct about the comment I can change it. Will give it a day or two to see if anything else comes in.

Regards,

Bob Pearson

#3 lore raw
Quoted 9 lines · Bob Pearson
Quoted 7 lines · Daisuke Matsuda (Fujitsu)
Quoted 6 lines · Bob Pearson

Currently the rxe driver does not handle all cases of zero length rdma operations correctly. The client does not have to provide an rkey for zero length RDMA read or write operations so the rkey provided may be invalid and should not be used to lookup an mr.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

This patch corrects the driver to ignore the provided rkey if the reth length is zero for read or write operations and make sure to set the mr to NULL. In read_reply() if length is zero rxe_recheck_mr() is not called. Warnings are added in the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.

The change looks good. I will leave it to you whether to fix the comment.

Thanks,
Daisuke

Quoted 68 lines
Quoted 50 lines

<...>

drivers/infiniband/sw/rxe/rxe_mr.c int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) · line 440

Quoted 7 lines
drivers/infiniband/sw/rxe/rxe_mr.c +3 -0
@@ -432,6 +435,10 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length)
 	int err;
 	u8 *va;
 
+	/* mr must be valid even if length is zero */
+	if (WARN_ON(!mr))
+		return -EINVAL;

If 'mr' is NULL, NULL pointer dereference can occur in process_flush() before reaching here. Isn't it better to do the check in process_flush()?

Quoted 18 lines
drivers/infiniband/sw/rxe/rxe_mr.c +1 -0
+
 	if (length == 0)
 		return 0;
 
drivers/infiniband/sw/rxe/rxe_resp.c +5 -0
index cccf7c6c21e9..c8e7b4ca456b 100644
@@ -420,13 +420,23 @@ static enum resp_states rxe_resp_check_length(struct rxe_qp *qp,
 	return RESPST_CHK_RKEY;
 }
 
+/* if the reth length field is zero we can assume nothing
+ * about the rkey value and should not validate or use it.
+ * Instead set qp->resp.rkey to 0 which is an invalid rkey
+ * value since the minimum index part is 1.
+ */
 static void qp_resp_from_reth(struct rxe_qp *qp, struct rxe_pkt_info *pkt)

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states rxe_resp_check_length(struct rxe_qp *qp, · line 435

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -3
 {
+	unsigned int length = reth_len(pkt);
+
 	qp->resp.va = reth_va(pkt);
 	qp->resp.offset = 0;
-	qp->resp.rkey = reth_rkey(pkt);
-	qp->resp.resid = reth_len(pkt);
-	qp->resp.length = reth_len(pkt);
+	qp->resp.resid = length;
+	qp->resp.length = length;

As you know, the comment above this function is applicable only to RDMA Read and Write. What do you say to mentioning FLUSH here rather than at the one in rxe_flush_pmem_iova().

Thanks,
Daisuke

Quoted 9 lines
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+	if (pkt->mask & RXE_READ_OR_WRITE_MASK && length == 0)
+		qp->resp.rkey = 0;
+	else
+		qp->resp.rkey = reth_rkey(pkt);
 }
 

<...>

Daisuke,

The updated patch no longer sets mr = NULL for flush. This check is mainly to defend against someone changing it again in the future and is near where mr is dereferenced. You are correct about the comment I can change it. Will give it a day or two to see if anything else comes in.

Regards,

Bob Pearson

#4 lore raw
In reply to Bob Pearson
Quoted 10 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
  static enum resp_states check_rkey(struct rxe_qp *qp,
  				   struct rxe_pkt_info *pkt)
  {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
  		return RESPST_EXECUTE;
  	}

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 10 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
 
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
  	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment when its value is not NULL.

Thanks
Zhijian

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
  		return RESPST_EXECUTE;
  	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
  	return RESPST_EXECUTE;
 
  err:
+	qp->resp.mr = NULL;
  	if (mr)
#5 lore raw
Quoted 13 lines · Zhijian/李 智坚
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
  static enum resp_states check_rkey(struct rxe_qp *qp,
  				   struct rxe_pkt_info *pkt)
  {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
  		return RESPST_EXECUTE;
  	}
 
Quoted 12 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
  	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment when its value is not NULL.

I do not see what you mean by " the previous qp->resp.mr ". As far as I understand, 'qp->resp.mr' is set to NULL in cleanup() before responder completes its work, so it is not supposed to be reused unlike 'res->read.rkey' being retained for multi-packet Read responses. Could you elaborate on your view?

Daisuke

Quoted 14 lines

Thanks
Zhijian

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
  		return RESPST_EXECUTE;
  	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
  	return RESPST_EXECUTE;
 
  err:
+	qp->resp.mr = NULL;
  	if (mr)
#6 lore raw
Quoted 27 lines · Daisuke/松田 大輔
Quoted 13 lines · Zhijian/李 智坚
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
   static enum resp_states check_rkey(struct rxe_qp *qp,
   				   struct rxe_pkt_info *pkt)
   {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   		return RESPST_EXECUTE;
   	}
 
Quoted 12 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
   	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment when its value is not NULL.

Quoted 5 lines

I do not see what you mean by " the previous qp->resp.mr ". As far as I understand, 'qp->resp.mr' is set to NULL in cleanup() before responder completes its work, so it is not supposed to be reused unlike 'res->read.rkey' being retained for multi-packet Read responses. Could you elaborate on your view?

IMO every 'qp->resp.mr' assignment will take a mr reference, so we should drop the this reference if we want to assign it a new mr again.

Theoretically, it should be changed to something like

if (qp->resp.mr) {
	rxe_put(qp->resp.mr)
         qp->resp.mr = NULL;
}
Quoted 9 lines
Quoted 9 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 574

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
   		return RESPST_EXECUTE;
   	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   	return RESPST_EXECUTE;
 
   err:
+	qp->resp.mr = NULL;

ditto

Thanks
Zhijian

Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +0 -0
   	if (mr)
#7 lore raw
Quoted 42 lines · Zhijian/李 智坚
Quoted 14 lines · Daisuke/松田 大輔
Quoted 13 lines · Zhijian/李 智坚
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
   static enum resp_states check_rkey(struct rxe_qp *qp,
   				   struct rxe_pkt_info *pkt)
   {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   		return RESPST_EXECUTE;
   	}
 
Quoted 11 lines
Quoted 11 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
   	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment

when its

Quoted 7 lines
Quoted 1 line

value is not NULL.

I do not see what you mean by " the previous qp->resp.mr ". As far as I understand, 'qp->resp.mr' is set to NULL in cleanup() before responder completes its work, so it is not supposed to be reused unlike 'res->read.rkey' being retained for multi-packet Read responses. Could you elaborate on your view?

IMO every 'qp->resp.mr' assignment will take a mr reference, so we should drop the this reference if we want to assign it a new mr again.

Quoted 5 lines

Theoretically, it should be changed to something like

if (qp->resp.mr) {
	rxe_put(qp->resp.mr)
         qp->resp.mr = NULL;
}

You are correct about what you wrote.
' qp->resp.mr = NULL;' and 'rxe_put(qp->resp.mr);' come in pairs in other parts of rxe. I agree the same principle should be applied here.

However, it seems to me that this 'qp->resp.mr' is always NULL. For all operations other than multi-packet Read responses, cleanup() must be executed before responder completes the work. On the other hand, for multi-packet Read replies, responder does not execute cleanup() and thus does not drop the reference until the last reply is sent, but it does not execute check_rkey() either since it uses the fast path jumping from get_req() to read_reply().

From practical perspective, we can just remove ' qp->resp.mr = NULL;', and then 'qp->resp.mr' will still be NULL. From theoretical perspective, we can make the change as Zhijian suggested. What other guys think about this?

Thanks,
Daisuke

Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 574

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
   		return RESPST_EXECUTE;
   	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   	return RESPST_EXECUTE;
 
   err:
+	qp->resp.mr = NULL;

ditto

Thanks
Zhijian

Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +0 -0
   	if (mr)
#8 lore raw
In reply to [email protected]
Quoted 36 lines · On 2/9/23 04:20, [email protected] wrote:
Quoted 27 lines · Daisuke/松田 大輔
Quoted 13 lines · Zhijian/李 智坚
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
   static enum resp_states check_rkey(struct rxe_qp *qp,
   				   struct rxe_pkt_info *pkt)
   {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   		return RESPST_EXECUTE;
   	}
 
Quoted 12 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
   	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment when its value is not NULL.

Quoted 5 lines

I do not see what you mean by " the previous qp->resp.mr ". As far as I understand, 'qp->resp.mr' is set to NULL in cleanup() before responder completes its work, so it is not supposed to be reused unlike 'res->read.rkey' being retained for multi-packet Read responses. Could you elaborate on your view?

Quoted 9 lines

IMO every 'qp->resp.mr' assignment will take a mr reference, so we should drop the this reference if we want to assign it a new mr again.

Theoretically, it should be changed to something like

if (qp->resp.mr) {
	rxe_put(qp->resp.mr)
         qp->resp.mr = NULL;
}

Generally I agree with the idea that a (few) assignments like this, even if after minutes of looking over the code can be shown to be unneeded because it is already NULL or will get deleted, are helpful if it makes the code easier to understand and prevents later changes to unintentionally break things.

Bob

Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 574

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
   		return RESPST_EXECUTE;
   	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
   	return RESPST_EXECUTE;
 
   err:
+	qp->resp.mr = NULL;

ditto

Thanks
Zhijian

Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +0 -0
   	if (mr)
#9 lore raw
Quoted 37 lines · Bob Pearson
Quoted 36 lines · On 2/9/23 04:20, [email protected] wrote:
Quoted 32 lines · Daisuke/松田 大輔
Quoted 13 lines · Zhijian/李 智坚
Quoted 11 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_resp.c +4 -0
+/* resolve the packet rkey to qp->resp.mr or set qp->resp.mr to NULL
+ * if an invalid rkey is received or the rdma length is zero. For middle
+ * or last packets use the stored value of mr.
+ */
    static enum resp_states check_rkey(struct rxe_qp *qp,
    				   struct rxe_pkt_info *pkt)
    {
@@ -473,10 +487,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
    		return RESPST_EXECUTE;
    	}
 
Quoted 12 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 495

Quoted 9 lines
drivers/infiniband/sw/rxe/rxe_resp.c +5 -3
-	/* A zero-byte op is not required to set an addr or rkey. See C9-88 */
+	/* A zero-byte read or write op is not required to
+	 * set an addr or rkey. See C9-88
+	 */
    	if ((pkt->mask & RXE_READ_OR_WRITE_MASK) &&
-	    (pkt->mask & RXE_RETH_MASK) &&
-	    reth_len(pkt) == 0) {
+	    (pkt->mask & RXE_RETH_MASK) && reth_len(pkt) == 0) {
+		qp->resp.mr = NULL;

You are making sure 'qp->resp.mr = NULL', but I doubt the previous qp->resp.mr will leak after this assignment when its value is not NULL.

I do not see what you mean by " the previous qp->resp.mr ". As far as I understand, 'qp->resp.mr' is set to NULL in cleanup() before responder completes its work, so it is not supposed to be reused unlike 'res->read.rkey' being retained for multi-packet Read responses. Could you elaborate on your view?

Quoted 8 lines
Quoted 8 lines

IMO every 'qp->resp.mr' assignment will take a mr reference, so we should drop the this reference if we want to assign it a new mr again.

Theoretically, it should be changed to something like

if (qp->resp.mr) {
	rxe_put(qp->resp.mr)
          qp->resp.mr = NULL;
}

Without "qp->resp.mr = NULL;", it is possible to cause use after free error if qp->resp.mr is used again.

Zhu Yanjun

Quoted 26 lines
Quoted 1 line

Generally I agree with the idea that a (few) assignments like this, even if after minutes of looking over the code can be shown to be unneeded because it is already NULL or will get deleted, are helpful if it makes the code easier to understand and prevents later changes to unintentionally break things.

Bob

Quoted 19 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 8 lines

drivers/infiniband/sw/rxe/rxe_resp.c static enum resp_states check_rkey(struct rxe_qp *qp, · line 574

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_resp.c +1 -0
    		return RESPST_EXECUTE;
    	}
 
@@ -555,6 +571,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
    	return RESPST_EXECUTE;
 
    err:
+	qp->resp.mr = NULL;

ditto

Thanks
Zhijian

Quoted 1 line
Quoted 1 line
Quoted 1 line
(quoted diff) +0 -0
    	if (mr)
maintainer Applied
#10 lore raw
In reply to Bob Pearson
Quoted 13 lines · Bob Pearson

Currently the rxe driver does not handle all cases of zero length rdma operations correctly. The client does not have to provide an rkey for zero length RDMA read or write operations so the rkey provided may be invalid and should not be used to lookup an mr.

This patch corrects the driver to ignore the provided rkey if the reth length is zero for read or write operations and make sure to set the mr to NULL. In read_reply() if length is zero rxe_recheck_mr() is not called. Warnings are added in the routines in rxe_mr.c to catch NULL MRs when the length is non-zero.

Quoted 7 lines

---
v4:
Fixed a regression in flush operations because the rkey must be valid in that case even if the length is zero which maybe the case for selectivity level of the entire memory region.

Applied to for-next, I added a fixes line for day 0

Jason