Skip to contentSkip to conversation

RDMA/siw: Clear association under lock if siw_qp_modify fails in siw_accept

PATCH Superseded Guoqing Jiang 2 participants 5 messages Aug 25 – Aug 26

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/siw: Clear association under lock if siw_qp_modify fails in siw_accept

We need to clear qp and cep before release state_lock as siw_qp_llp_close and siw_qp_modify->siw_qp_llp_close did.

Otherwise if siw_qp_modify() fails in siw_accept(), the QP's state_lock is released before the error path cleanup. A concurrent ibv_modify_qp() transitioning the QP to ERROR can race in this window:

siw_accept() ibv_modify_qp(ERROR)
---------------------- ----------------------
siw_qp_modify() fails
up_write(&qp->state_lock)

                                     down_write(&qp->state_lock)
                                     nextstate_from_idle():
				     if (qp->cep)
                                       siw_cep_put(qp->cep) <- frees cep
                                       qp->cep = NULL
  goto error
    cep->qp = NULL                   <- UAF

Clear qp->cep and cep->qp, and drop the association reference taken by siw_cep_get(), all under the write lock held from the initial down_write(&qp->state_lock). Thread B therefore sees qp->cep == NULL, skips its own put, and cannot free the cep before siw_accept() is done with it.

---
 drivers/infiniband/sw/siw/siw_cm.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +6 -2
index 0245b25e7271..1573f2e888b2 100644
@@ -1719,9 +1719,13 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 			   SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE |
 				   SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD |
 				   SIW_QP_ATTR_MPA);
+	if (rv) {
+		cep->qp = NULL;
+		qp->cep = NULL;
+		siw_cep_put(cep);
+		goto error_unlock;
+	}
 	up_write(&qp->state_lock);
-	if (rv)
-		goto error;
 
 	siw_dbg_cep(cep, "[QP %u]: send mpa reply, %d byte pdata\n",
 		    qp_id(qp), params->private_data_len);
Signature · 1 line
2.35.3
maintainer
#1 lore raw
Quoted 37 lines · Guoqing Jiang

We need to clear qp and cep before release state_lock as siw_qp_llp_closeo and siw_qp_modify->siw_qp_llp_close did.

Otherwise if siw_qp_modify() fails in siw_accept(), the QP's state_lock is released before the error path cleanup. A concurrent ibv_modify_qp() transitioning the QP to ERROR can race in this window:

   siw_accept()                       ibv_modify_qp(ERROR)
   ----------------------             ----------------------
   siw_qp_modify() fails
   up_write(&qp->state_lock)
                                      down_write(&qp->state_lock)
                                      nextstate_from_idle():
				     if (qp->cep)
                                        siw_cep_put(qp->cep) <- frees cep
                                        qp->cep = NULL
   goto error
     cep->qp = NULL                   <- UAF

Clear qp->cep and cep->qp, and drop the association reference taken by siw_cep_get(), all under the write lock held from the initial down_write(&qp->state_lock). Thread B therefore sees qp->cep == NULL, skips its own put, and cannot free the cep before siw_accept() is done with it.

---
  drivers/infiniband/sw/siw/siw_cm.c | 8 ++++++--
  1 file changed, 6 insertions(+), 2 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +0 -0
index 0245b25e7271..1573f2e888b2 100644

drivers/infiniband/sw/siw/siw_cm.c int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) · line 1723

Quoted 5 lines
drivers/infiniband/sw/siw/siw_cm.c +2 -0
@@ -1719,9 +1719,13 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
  			   SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE |
  				   SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD |
  				   SIW_QP_ATTR_MPA);
+	if (rv) {
+		cep->qp = NULL;

This can better be omitted since done in error_unlock path anyway and is redundant otherwise. It would also better retain the logic of calling siw_qp_put(qp) right after clearing cep's reference to the qp, as done in error path.

Thank you!
Bernard.> + qp->cep = NULL;

Quoted 9 lines
drivers/infiniband/sw/siw/siw_cm.c +3 -2
+		siw_cep_put(cep);
+		goto error_unlock;
+	}
  	up_write(&qp->state_lock);
-	if (rv)
-		goto error;
 
  	siw_dbg_cep(cep, "[QP %u]: send mpa reply, %d byte pdata\n",
  		    qp_id(qp), params->private_data_len);
#2 lore raw

Hi Bernard,

Quoted 33 lines · Bernard Metzler
Quoted 32 lines · Guoqing Jiang

We need to clear qp and cep before release state_lock as siw_qp_llp_closeo and siw_qp_modify->siw_qp_llp_close did.

Otherwise if siw_qp_modify() fails in siw_accept(), the QP's state_lock is released before the error path cleanup. A concurrent ibv_modify_qp() transitioning the QP to ERROR can race in this window:

   siw_accept()                       ibv_modify_qp(ERROR) ----------------------             ---------------------- siw_qp_modify() fails
   up_write(&qp->state_lock)
down_write(&qp->state_lock)
                                      nextstate_from_idle():
                     if (qp->cep)
                                        siw_cep_put(qp->cep) <- frees cep qp->cep = NULL
   goto error
     cep->qp = NULL                   <- UAF

Clear qp->cep and cep->qp, and drop the association reference taken by siw_cep_get(), all under the write lock held from the initial down_write(&qp->state_lock). Thread B therefore sees qp->cep == NULL, skips its own put, and cannot free the cep before siw_accept() is done with it.

---
  drivers/infiniband/sw/siw/siw_cm.c | 8 ++++++--

  1 file changed, 6 insertions(+), 2 deletions(-)
Quoted 13 lines
Quoted 11 lines
drivers/infiniband/sw/siw/siw_cm.c +0 -0
index 0245b25e7271..1573f2e888b2 100644
@@ -1719,9 +1719,13 @@ int siw_accept(struct iw_cm_id *id, struct 
 iw_cm_conn_param *params)

                 SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE | SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD | SIW_QP_ATTR_MPA);
+    if (rv) {
+        cep->qp = NULL;

This can better be omitted since done in error_unlock path anyway and is redundant otherwise.

Right, thanks for the review! And probably call siw_cep_put before set qp->cep to NULL as other places did.

Quoted 2 lines

It would also better retain the logic of calling siw_qp_put(qp) right after clearing cep's reference to the qp, as done in error path.

I suppose this change could be okay.

@@ -1719,9 +1719,12 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE | SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD | SIW_QP_ATTR_MPA);
+       if (rv) {
+               siw_cep_put(cep);
+               qp->cep = NULL;
+               goto error_unlock;
+       }
        up_write(&qp->state_lock);
-       if (rv)
-               goto error;

I am wondering if there is another use-after-free case in siw_qp_cm_drop. Because the beginning of siw_qp_cm_drop borrows the qp->cep pointer without holding a reference. If another thread put the last reference of cep, then siw_qp_cm_drop wakes and deferences cep which has been freed.

Quoted 11 lines

Thank you!
Bernard.> +        qp->cep = NULL;

Quoted 8 lines

+        siw_cep_put(cep);
+        goto error_unlock;
+    }
      up_write(&qp->state_lock);
-    if (rv)
-        goto error;
        siw_dbg_cep(cep, "[QP %u]: send mpa reply, %d byte pdata\n", qp_id(qp), params->private_data_len);

Thanks,
Guoqing

maintainer
#3 lore raw
Quoted 60 lines · Guoqing Jiang

Hi Bernard,

Quoted 33 lines · Bernard Metzler
Quoted 32 lines · Guoqing Jiang

We need to clear qp and cep before release state_lock as siw_qp_llp_closeo and siw_qp_modify->siw_qp_llp_close did.

Otherwise if siw_qp_modify() fails in siw_accept(), the QP's state_lock is released before the error path cleanup. A concurrent ibv_modify_qp() transitioning the QP to ERROR can race in this window:

   siw_accept()                       ibv_modify_qp(ERROR) ----------------------             ---------------------- siw_qp_modify() fails
   up_write(&qp->state_lock)
down_write(&qp->state_lock)
                                      nextstate_from_idle():
                     if (qp->cep)
                                        siw_cep_put(qp->cep) <- frees cep qp->cep = NULL
   goto error
     cep->qp = NULL                   <- UAF

Clear qp->cep and cep->qp, and drop the association reference taken by siw_cep_get(), all under the write lock held from the initial down_write(&qp->state_lock). Thread B therefore sees qp->cep == NULL, skips its own put, and cannot free the cep before siw_accept() is done with it.

---
  drivers/infiniband/sw/siw/siw_cm.c | 8 ++++++--

  1 file changed, 6 insertions(+), 2 deletions(-)
Quoted 12 lines
Quoted 10 lines
drivers/infiniband/sw/siw/siw_cm.c +0 -0
index 0245b25e7271..1573f2e888b2 100644
@@ -1719,9 +1719,13 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)

                 SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE | SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD | SIW_QP_ATTR_MPA);
+    if (rv) {
+        cep->qp = NULL;

This can better be omitted since done in error_unlock path anyway and is redundant otherwise.

Right, thanks for the review! And probably call siw_cep_put before set qp->cep to NULL as other places did.

Quoted 2 lines

It would also better retain the logic of calling siw_qp_put(qp) right after clearing cep's reference to the qp, as done in error path.

I suppose this change could be okay.

drivers/infiniband/sw/siw/siw_cm.c +0 -0
@@ -1719,9 +1719,12 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
                            SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE |
                                    SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD |

drivers/infiniband/sw/siw/siw_cm.c int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) · line 1728

Quoted 10 lines
drivers/infiniband/sw/siw/siw_cm.c +5 -2
                                    SIW_QP_ATTR_MPA);
+       if (rv) {
+               siw_cep_put(cep);
+               qp->cep = NULL;
+               goto error_unlock;
+       }
         up_write(&qp->state_lock);
-       if (rv)
-               goto error;
 

That looks good, while I think we should always NULL a pointer to something before releasing a hold/reference to it. Putting the reference implies a memory barrier and so it's guaranteed others see that NULL as well. You are right - there are places where this is not done as it should.

Quoted 5 lines

I am wondering if there is another use-after-free case in siw_qp_cm_drop. Because the beginning of siw_qp_cm_drop borrows the qp->cep pointer without holding a reference. If another thread put the last reference of cep, then siw_qp_cm_drop wakes and deferences cep which has been freed.

Let's put this in an extra thread. You may have a point ;)

Thanks,
Bernard.

Quoted 14 lines
Quoted 11 lines

Thank you!
Bernard.> +        qp->cep = NULL;

Quoted 8 lines

+        siw_cep_put(cep);
+        goto error_unlock;
+    }
      up_write(&qp->state_lock);
-    if (rv)
-        goto error;
        siw_dbg_cep(cep, "[QP %u]: send mpa reply, %d byte pdata\n", qp_id(qp), params->private_data_len);

Thanks,
Guoqing

#4 lore raw
Quoted 68 lines · Bernard Metzler
Quoted 67 lines · Guoqing Jiang

Hi Bernard,

Quoted 33 lines · Bernard Metzler
Quoted 32 lines · Guoqing Jiang

We need to clear qp and cep before release state_lock as siw_qp_llp_closeo and siw_qp_modify->siw_qp_llp_close did.

Otherwise if siw_qp_modify() fails in siw_accept(), the QP's state_lock is released before the error path cleanup. A concurrent ibv_modify_qp() transitioning the QP to ERROR can race in this window:

   siw_accept()                       ibv_modify_qp(ERROR) ----------------------             ---------------------- siw_qp_modify() fails
   up_write(&qp->state_lock) down_write(&qp->state_lock)
                                      nextstate_from_idle():
                     if (qp->cep) siw_cep_put(qp->cep) <- frees cep
                                        qp->cep = NULL goto error
     cep->qp = NULL                   <- UAF

Clear qp->cep and cep->qp, and drop the association reference taken by siw_cep_get(), all under the write lock held from the initial down_write(&qp->state_lock). Thread B therefore sees qp->cep == NULL, skips its own put, and cannot free the cep before siw_accept() is done with it.

---
  drivers/infiniband/sw/siw/siw_cm.c | 8 ++++++--

  1 file changed, 6 insertions(+), 2 deletions(-)
Quoted 13 lines
Quoted 11 lines
drivers/infiniband/sw/siw/siw_cm.c +0 -0
index 0245b25e7271..1573f2e888b2 100644
@@ -1719,9 +1719,13 @@ int siw_accept(struct iw_cm_id *id, struct 
 iw_cm_conn_param *params)

                 SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE | SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD | SIW_QP_ATTR_MPA);
+    if (rv) {
+        cep->qp = NULL;

This can better be omitted since done in error_unlock path anyway and is redundant otherwise.

Right, thanks for the review! And probably call siw_cep_put before set qp->cep to NULL as other places did.

Quoted 2 lines

It would also better retain the logic of calling siw_qp_put(qp) right after clearing cep's reference to the qp, as done in error path.

I suppose this change could be okay.

(quoted diff) +0 -0
@@ -1719,9 +1719,12 @@ int siw_accept(struct iw_cm_id *id, struct 
 iw_cm_conn_param *params)

                            SIW_QP_ATTR_STATE | SIW_QP_ATTR_LLP_HANDLE | SIW_QP_ATTR_ORD | SIW_QP_ATTR_IRD | SIW_QP_ATTR_MPA);
+       if (rv) {
+               siw_cep_put(cep);
+               qp->cep = NULL;
+               goto error_unlock;

Quoted 8 lines
Quoted 5 lines

+       }
         up_write(&qp->state_lock);
-       if (rv)
-               goto error;

That looks good, while I think we should always NULL a pointer to something before releasing a hold/reference to it. Putting the reference implies a memory barrier and so it's guaranteed others see that NULL as well.

Hmm, good point.

Quoted 8 lines

You are right - there are places where this is not done as it should.

Quoted 5 lines

I am wondering if there is another use-after-free case in siw_qp_cm_drop. Because the beginning of siw_qp_cm_drop borrows the qp->cep pointer without holding a reference. If another thread put the last reference of cep, then siw_qp_cm_drop wakes and deferences cep which has been freed.

Let's put this in an extra thread. You may have a point ;)

I will dig a little bit more before start a new thread.

Thanks,
Guoqing