Skip to contentSkip to conversation

libceph: wait for con->work to finish when cancelling con

PATCH In review [email protected] 3 participants 7 messages Mar 8, 2022 – Mar 9, 2022 Ceph common code (libceph)

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 libceph: wait for con->work to finish when cancelling con

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

 net/ceph/messenger.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +2 -2
index d3bb656308b4..62e39f63f94c 100644
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
 	ceph_con_reset_protocol(con);
 	ceph_con_reset_session(con);
-	cancel_con(con);
 	mutex_unlock(&con->mutex);
+	cancel_con(con);
 }
 EXPORT_SYMBOL(ceph_con_close);
 
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
 static void cancel_con(struct ceph_connection *con)
 {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
 		dout("%s %p\n", __func__, con);
 		con->ops->put(con);
 	}
Signature · 1 line
2.27.0
#1 lore raw
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

 net/ceph/messenger.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
 	ceph_con_reset_protocol(con);
 	ceph_con_reset_session(con);
-	cancel_con(con);
 	mutex_unlock(&con->mutex);
+	cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

Quoted 12 lines
net/ceph/messenger.c +1 -1
 }
 EXPORT_SYMBOL(ceph_con_close);
 
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
 static void cancel_con(struct ceph_connection *con)
 {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
 		dout("%s %p\n", __func__, con);
 		con->ops->put(con);
 	}
Signature · 1 line
Jeff Layton <[email protected]>
author
#2 lore raw
Quoted 36 lines · Jeff Layton
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

  net/ceph/messenger.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
  	ceph_con_reset_protocol(con);
  	ceph_con_reset_session(con);
-	cancel_con(con);
  	mutex_unlock(&con->mutex);
+	cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

Quoted 6 lines

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

IMO it should be okay, since the 'queue_con(con)', which doing the similar things, also outside the mutex.

- Xiubo

Quoted 13 lines
Quoted 13 lines
net/ceph/messenger.c +1 -1
  }
  EXPORT_SYMBOL(ceph_con_close);
 
@@ -1416,7 +1416,7 @@ static void queue_con(struct ceph_connection *con)
 
  static void cancel_con(struct ceph_connection *con)
  {
-	if (cancel_delayed_work(&con->work)) {
+	if (cancel_delayed_work_sync(&con->work)) {
  		dout("%s %p\n", __func__, con);
  		con->ops->put(con);
  	}
maintainer
#3 lore raw
Quoted 43 lines · Xiubo Li
Quoted 40 lines · Jeff Layton
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

  net/ceph/messenger.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
     ceph_con_reset_protocol(con);
     ceph_con_reset_session(con);
-    cancel_con(con);
     mutex_unlock(&con->mutex);
+    cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

Quoted 5 lines
Quoted 2 lines

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

IMO it should be okay, since the 'queue_con(con)', which doing the similar things, also outside the mutex.

Hi Xiubo,

I read the patch description and skimmed through the linked trackers but I don't understand the issue. ceph_con_workfn() holds con->mutex for most of the time it's running and cancel_delayed_work() is called under the same con->mutex. It's true that ceph_con_workfn() work may not be finished by the time ceph_con_close() returns but I don't see how that can result in anything bad happening.

Can you explain the issue in more detail, with pointers to specific code snippets in the MDS client and the messenger? Where exactly is the "new data" (what data, please be specific) gets misused by the "stale work"?

Thanks,

Ilya

author
#4 lore raw
Quoted 55 lines · Ilya Dryomov
Quoted 37 lines · Xiubo Li
Quoted 35 lines · Jeff Layton
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

   net/ceph/messenger.c | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
      ceph_con_reset_protocol(con);
      ceph_con_reset_session(con);
-    cancel_con(con);
      mutex_unlock(&con->mutex);
+    cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

Quoted 8 lines
Quoted 6 lines

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

IMO it should be okay, since the 'queue_con(con)', which doing the similar things, also outside the mutex.

Hi Xiubo,

I read the patch description and skimmed through the linked trackers but I don't understand the issue. ceph_con_workfn() holds con->mutex for most of the time it's running and cancel_delayed_work() is called under the same con->mutex. It's true that ceph_con_workfn() work may not be finished by the time ceph_con_close() returns but I don't see how that can result in anything bad happening.

Quoted 4 lines

Can you explain the issue in more detail, with pointers to specific code snippets in the MDS client and the messenger? Where exactly is the "new data" (what data, please be specific) gets misused by the "stale work"?

The tracker I attached in V1 is not exact, please ignore that.

From the current code, there has one case that for ceph fs in send_mds_reconnect():

4256         ceph_con_close(&session->s_con);
4257         ceph_con_open(&session->s_con,
4258                       CEPH_ENTITY_TYPE_MDS, mds, 4259                       ceph_mdsmap_get_addr(mdsc->mdsmap, mds));

If in ceph_con_close() just before cancelling the con->work, it was already fired but then the queue thread was just scheduled out when the con->work was trying to take the con->mutex.

And then in ceph_con_open() it will update the con->state to 'CEPH_CON_S_PREOPEN' and other members then queue the con->work again.

That means the con->work will be run twice with the state 'CEPH_CON_S_PREOPEN'.

I am not sure whether will this cause strange issues like the URL I attached in V1.

- Xiubo

Quoted 4 lines

Thanks,

Ilya

maintainer
#5 lore raw
Quoted 73 lines · Xiubo Li
Quoted 55 lines · Ilya Dryomov
Quoted 37 lines · Xiubo Li
Quoted 35 lines · Jeff Layton
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

   net/ceph/messenger.c | 4 ++--
   1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
      ceph_con_reset_protocol(con);
      ceph_con_reset_session(con);
-    cancel_con(con);
      mutex_unlock(&con->mutex);
+    cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

Quoted 8 lines
Quoted 6 lines

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

IMO it should be okay, since the 'queue_con(con)', which doing the similar things, also outside the mutex.

Hi Xiubo,

I read the patch description and skimmed through the linked trackers but I don't understand the issue. ceph_con_workfn() holds con->mutex for most of the time it's running and cancel_delayed_work() is called under the same con->mutex. It's true that ceph_con_workfn() work may not be finished by the time ceph_con_close() returns but I don't see how that can result in anything bad happening.

Quoted 4 lines

Can you explain the issue in more detail, with pointers to specific code snippets in the MDS client and the messenger? Where exactly is the "new data" (what data, please be specific) gets misused by the "stale work"?

The tracker I attached in V1 is not exact, please ignore that.

From the current code, there has one case that for ceph fs in
send_mds_reconnect():

4256 ceph_con_close(&session->s_con);
4257 ceph_con_open(&session->s_con,
4258 CEPH_ENTITY_TYPE_MDS, mds,
4259 ceph_mdsmap_get_addr(mdsc->mdsmap, mds));

Quoted 6 lines

If in ceph_con_close() just before cancelling the con->work, it was already fired but then the queue thread was just scheduled out when the con->work was trying to take the con->mutex.

And then in ceph_con_open() it will update the con->state to 'CEPH_CON_S_PREOPEN' and other members then queue the con->work again.

But ceph_con_close() releases con->mutex before returning, so the work that was trying to grab con->mutex would immediately grab it, encounter CEPH_CON_S_CLOSED and bail.

Quoted 3 lines

That means the con->work will be run twice with the state 'CEPH_CON_S_PREOPEN'.

... so this is very unlikely. But even it happens somehow, again I don't see how that can result in anything bad happening: whoever sees CEPH_CON_S_PREOPEN first would transition con to the initial "opening" state (e.g. CEPH_CON_S_V1_BANNER for msgr1).

Quoted 3 lines

I am not sure whether will this cause strange issues like the URL I attached in V1.

Until you can pin point the messenger as the root cause of those issues, I'd recommend dropping this patch.

Thanks,

Ilya

author
#6 lore raw
Quoted 88 lines · Ilya Dryomov
Quoted 70 lines · Xiubo Li
Quoted 54 lines · Ilya Dryomov
Quoted 36 lines · Xiubo Li
Quoted 35 lines · Jeff Layton
Quoted 24 lines · On Tue, 2022-03-08 at 21:23 +0800, [email protected] wrote:

From: Xiubo Li <[email protected]>

When reconnecting MDS it will reopen the con with new ip address, but the when opening the con with new address it couldn't be sure that the stale work has finished. So it's possible that the stale work queued will use the new data.

This will use cancel_delayed_work_sync() instead.

---

V2:
- Call cancel_con() after dropping the mutex

    net/ceph/messenger.c | 4 ++--
    1 file changed, 2 insertions(+), 2 deletions(-)
net/ceph/messenger.c +0 -0
index d3bb656308b4..62e39f63f94c 100644

net/ceph/messenger.c void ceph_con_close(struct ceph_connection *con) · line 585

Quoted 6 lines
net/ceph/messenger.c +1 -1
@@ -581,8 +581,8 @@ void ceph_con_close(struct ceph_connection *con)
 
       ceph_con_reset_protocol(con);
       ceph_con_reset_session(con);
-    cancel_con(con);
       mutex_unlock(&con->mutex);
+    cancel_con(con);

Now the question is: Is it safe to cancel this work outside the mutex or will this open up any races. Unfortunately with coarse-grained locks like this, it's hard to tell what the lock actually protects.

Quoted 8 lines
Quoted 6 lines

If we need to keep the cancel inside the lock for some reason, you could instead just add a "flush_workqueue()" after dropping the mutex in the above function.

So, this looks reasonable to me at first glance, but I'd like Ilya to ack this before we merge it.

IMO it should be okay, since the 'queue_con(con)', which doing the similar things, also outside the mutex.

Hi Xiubo,

I read the patch description and skimmed through the linked trackers but I don't understand the issue. ceph_con_workfn() holds con->mutex for most of the time it's running and cancel_delayed_work() is called under the same con->mutex. It's true that ceph_con_workfn() work may not be finished by the time ceph_con_close() returns but I don't see how that can result in anything bad happening.

Quoted 4 lines

Can you explain the issue in more detail, with pointers to specific code snippets in the MDS client and the messenger? Where exactly is the "new data" (what data, please be specific) gets misused by the "stale work"?

The tracker I attached in V1 is not exact, please ignore that.

From the current code, there has one case that for ceph fs in
send_mds_reconnect():

4256 ceph_con_close(&session->s_con);
4257 ceph_con_open(&session->s_con,
4258 CEPH_ENTITY_TYPE_MDS, mds,
4259 ceph_mdsmap_get_addr(mdsc->mdsmap, mds));

Quoted 6 lines

If in ceph_con_close() just before cancelling the con->work, it was already fired but then the queue thread was just scheduled out when the con->work was trying to take the con->mutex.

And then in ceph_con_open() it will update the con->state to 'CEPH_CON_S_PREOPEN' and other members then queue the con->work again.

But ceph_con_close() releases con->mutex before returning, so the work that was trying to grab con->mutex would immediately grab it, encounter CEPH_CON_S_CLOSED and bail.

Quoted 2 lines

That means the con->work will be run twice with the state 'CEPH_CON_S_PREOPEN'.

... so this is very unlikely. But even it happens somehow, again I don't see how that can result in anything bad happening: whoever sees CEPH_CON_S_PREOPEN first would transition con to the initial "opening" state (e.g. CEPH_CON_S_V1_BANNER for msgr1).

Quoted 4 lines
Quoted 2 lines

I am not sure whether will this cause strange issues like the URL I attached in V1.

Until you can pin point the messenger as the root cause of those issues, I'd recommend dropping this patch.

Sure, let's drop it for now, that tracker was caused by other issue and I have fixed it.

- Xiubo

Quoted 4 lines

Thanks,

Ilya