Skip to contentSkip to conversation

libceph: make calc_target() set t->paused, not just clear it

PATCH In review Ilya Dryomov 2 participants 3 messages Jan 5 – Jan 6 Ceph common code (libceph)

lore
Share & tools
lore

Ilya Dryomov

author maintainer
#0 lore raw

PATCH 1/1 libceph: make calc_target() set t->paused, not just clear it

Currently calc_target() clears t->paused if the request shouldn't be paused anymore, but doesn't ever set t->paused even though it's able to determine when the request should be paused. Setting t->paused is left to __submit_request() which is fine for regular requests but doesn't work for linger requests -- since __submit_request() doesn't operate on linger requests, there is nowhere for lreq->t.paused to be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished, rbd_reregister_watch() gets stuck with rbd_dev->watch_mutex held:

rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_wait

It's waiting for lreq->reg_commit_wait to be completed, but for that to happen the respective request needs to end up on need_resend_linger list and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.

The fact that rbd_dev->watch_mutex remains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.

---
 net/ceph/osd_client.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +9 -2
index 1a7be2f615dc..610e584524d1 100644
@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
 	struct ceph_pg_pool_info *pi;
 	struct ceph_pg pgid, last_pgid;
 	struct ceph_osds up, acting;
+	bool should_be_paused;
 	bool is_read = t->flags & CEPH_OSD_FLAG_READ;
 	bool is_write = t->flags & CEPH_OSD_FLAG_WRITE;
 	bool force_resend = false;
@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
 				 &last_pgid))
 		force_resend = true;
 
-	if (t->paused && !target_should_be_paused(osdc, t, pi)) {
-		t->paused = false;
+	should_be_paused = target_should_be_paused(osdc, t, pi);
+	if (t->paused && !should_be_paused) {
 		unpaused = true;
 	}
+	if (t->paused != should_be_paused) {
+		dout("%s t %p paused %d -> %d\n", __func__, t, t->paused,
+		     should_be_paused);
+		t->paused = should_be_paused;
+	}
+
 	legacy_change = ceph_pg_compare(&t->pgid, &pgid) ||
 			ceph_osds_changed(&t->acting, &acting,
 					  t->used_replica || any_change);
Signature · 1 line
2.49.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 47 lines · Ilya Dryomov

Currently calc_target() clears t->paused if the request shouldn't be paused anymore, but doesn't ever set t->paused even though it's able to determine when the request should be paused. Setting t->paused is left to __submit_request() which is fine for regular requests but doesn't work for linger requests -- since __submit_request() doesn't operate on linger requests, there is nowhere for lreq->t.paused to be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished, rbd_reregister_watch() gets stuck with rbd_dev->watch_mutex held:

rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_wait

It's waiting for lreq->reg_commit_wait to be completed, but for that to happen the respective request needs to end up on need_resend_linger list and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.

The fact that rbd_dev->watch_mutex remains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.

---
 net/ceph/osd_client.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +1 -0
index 1a7be2f615dc..610e584524d1 100644
@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
 	struct ceph_pg_pool_info *pi;
 	struct ceph_pg pgid, last_pgid;
 	struct ceph_osds up, acting;
+	bool should_be_paused;
 	bool is_read = t->flags & CEPH_OSD_FLAG_READ;
 	bool is_write = t->flags & CEPH_OSD_FLAG_WRITE;
 	bool force_resend = false;

net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1659

Quoted 7 lines
net/ceph/osd_client.c +2 -2
@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
 				 &last_pgid))
 		force_resend = true;
 
-	if (t->paused && !target_should_be_paused(osdc, t, pi)) {
-		t->paused = false;
+	should_be_paused = target_should_be_paused(osdc, t, pi);
+	if (t->paused && !should_be_paused) {

Do we need dout() message here as symmetric to the added one?

net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1670

Quoted 11 lines
net/ceph/osd_client.c +6 -0
 		unpaused = true;
 	}
+	if (t->paused != should_be_paused) {
+		dout("%s t %p paused %d -> %d\n", __func__, t, t->paused,
+		     should_be_paused);
+		t->paused = should_be_paused;
+	}
+
 	legacy_change = ceph_pg_compare(&t->pgid, &pgid) ||
 			ceph_osds_changed(&t->acting, &acting,
 					  t->used_replica || any_change);

Looks good.

Thanks,
Slava.

author maintainer
#2 lore raw
Quoted 52 lines · Viacheslav Dubeyko
Quoted 50 lines · Ilya Dryomov

Currently calc_target() clears t->paused if the request shouldn't be paused anymore, but doesn't ever set t->paused even though it's able to determine when the request should be paused. Setting t->paused is left to __submit_request() which is fine for regular requests but doesn't work for linger requests -- since __submit_request() doesn't operate on linger requests, there is nowhere for lreq->t.paused to be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished, rbd_reregister_watch() gets stuck with rbd_dev->watch_mutex held:

rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_wait

It's waiting for lreq->reg_commit_wait to be completed, but for that to happen the respective request needs to end up on need_resend_linger list and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.

The fact that rbd_dev->watch_mutex remains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.

---
 net/ceph/osd_client.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +1 -0
index 1a7be2f615dc..610e584524d1 100644
@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
      struct ceph_pg_pool_info *pi;
      struct ceph_pg pgid, last_pgid;
      struct ceph_osds up, acting;
+     bool should_be_paused;
      bool is_read = t->flags & CEPH_OSD_FLAG_READ;
      bool is_write = t->flags & CEPH_OSD_FLAG_WRITE;
      bool force_resend = false;
@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,
                               &last_pgid))
              force_resend = true;
 
Quoted 6 lines

net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1659

Quoted 4 lines
net/ceph/osd_client.c +2 -2
-     if (t->paused && !target_should_be_paused(osdc, t, pi)) {
-             t->paused = false;
+     should_be_paused = target_should_be_paused(osdc, t, pi);
+     if (t->paused && !should_be_paused) {

Do we need dout() message here as symmetric to the added one?

No -- the value of unpaused is already logged before returning.

Thanks,

Ilya