Skip to contentSkip to conversation

libceph: wait for con->work to finish when cancelling con

RFC PATCH Superseded [email protected] 2 participants 3 messages Mar 8, 2022 Ceph common code (libceph)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 libceph: wait for con->work to finish when cancelling con

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

URL: https://tracker.ceph.com/issues/54461

---
 net/ceph/messenger.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/messenger.c +1 -1
index d3bb656308b4..32eb5dc00583 100644
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
 static void cancel_con(struct ceph_connection *con)
 {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
 		dout("%s %p\n", __func__, con);
 		con->ops->put(con);
 	}
Signature · 1 line
2.27.0
#1 lore raw
Quoted 20 lines · On Tue, 2022-03-08 at 17:59 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

URL: https://tracker.ceph.com/issues/54461

---
 net/ceph/messenger.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..32eb5dc00583 100644

net/ceph/messenger.c static void queue_con(struct ceph_connection *con) · line 1422

Quoted 8 lines
net/ceph/messenger.c +1 -1
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
 static void cancel_con(struct ceph_connection *con)
 {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
 		dout("%s %p\n", __func__, con);
 		con->ops->put(con);
 	}

Won't this deadlock?

This function is called from ceph_con_close with the con->mutex held. The work will try to take the same mutex and will get stuck. If you want to do this, then you may also need to change it to call cancel_con after dropping the mutex.

Signature · 1 line
Jeff Layton <[email protected]>
author
#2 lore raw
Quoted 31 lines · Jeff Layton
Quoted 20 lines · On Tue, 2022-03-08 at 17:59 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

URL: https://tracker.ceph.com/issues/54461

---
  net/ceph/messenger.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..32eb5dc00583 100644

net/ceph/messenger.c static void queue_con(struct ceph_connection *con) · line 1422

Quoted 8 lines
net/ceph/messenger.c +1 -1
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
  static void cancel_con(struct ceph_connection *con)
  {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
  		dout("%s %p\n", __func__, con);
  		con->ops->put(con);
  	}

Won't this deadlock?

Quoted 4 lines

This function is called from ceph_con_close with the con->mutex held. The work will try to take the same mutex and will get stuck. If you want to do this, then you may also need to change it to call cancel_con after dropping the mutex.

Yeah, correct :-)

- Xiubo