libceph: make calc_target() set t->paused, not just clear it
Ilya Dryomov
maintainerPATCH 1/1 libceph: make calc_target() set t->paused, not just clear it
Currently calc_target() clears t->paused if the request shouldn't be paused anymore, but doesn't ever set t->paused even though it's able to determine when the request should be paused. Setting t->paused is left to __submit_request() which is fine for regular requests but doesn't work for linger requests -- since __submit_request() doesn't operate on linger requests, there is nowhere for lreq->t.paused to be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished, rbd_reregister_watch() gets stuck with rbd_dev->watch_mutex held:
rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_wait
It's waiting for lreq->reg_commit_wait to be completed, but for that to happen the respective request needs to end up on need_resend_linger list and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.
The fact that rbd_dev->watch_mutex remains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.
--- net/ceph/osd_client.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
net/ceph/osd_client.c +9 -2
@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,struct ceph_pg_pool_info *pi;struct ceph_pg pgid, last_pgid;struct ceph_osds up, acting; +bool should_be_paused;bool is_read = t->flags &CEPH_OSD_FLAG_READ ;bool is_write = t->flags &CEPH_OSD_FLAG_WRITE ;bool force_resend = false;
@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc, &last_pgid)) force_resend = true; -if (t->paused && !target_should_be_paused (osdc, t, pi)) { - t->paused = false; + should_be_paused =target_should_be_paused (osdc, t, pi); +if (t->paused && !should_be_paused) { unpaused = true; } +if (t->paused != should_be_paused) { +dout ("%s t %p paused %d -> %d\n" , __func__, t, t->paused, + should_be_paused); + t->paused = should_be_paused; + } + legacy_change =ceph_pg_compare (&t->pgid, &pgid) ||ceph_osds_changed (&t->acting, &acting, t->used_replica || any_change);
Signature · 1 line
2.49.0
Quoted 47 lines · Ilya Dryomov
Currently
calc_target()clearst->pausedif the request shouldn't be paused anymore, but doesn't ever sett->pausedeven though it's able to determine when the request should be paused. Settingt->pausedis left to__submit_request()which is fine for regular requests but doesn't work for linger requests -- since__submit_request()doesn't operate on linger requests, there is nowhere forlreq->t.pausedto be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished,rbd_reregister_watch()gets stuck withrbd_dev->watch_mutexheld:
rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_waitIt's waiting for
lreq->reg_commit_waitto be completed, but for that to happen the respective request needs to end up onneed_resend_lingerlist and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.The fact that
rbd_dev->watch_mutexremains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.Reported-by Raphael Zimmer <[email protected]>Signed-off-by Ilya Dryomov <[email protected]>--- net/ceph/osd_client.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)net/ceph/osd_client.c +1
-0@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,struct ceph_pg_pool_info *pi;struct ceph_pg pgid, last_pgid;struct ceph_osds up, acting; +bool should_be_paused;bool is_read = t->flags &CEPH_OSD_FLAG_READ ;bool is_write = t->flags &CEPH_OSD_FLAG_WRITE ;bool force_resend = false;
net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1659
Quoted 7 lines
net/ceph/osd_client.c +2
-2@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc, &last_pgid)) force_resend = true; -if (t->paused && !target_should_be_paused (osdc, t, pi)) { - t->paused = false; + should_be_paused =target_should_be_paused (osdc, t, pi); +if (t->paused && !should_be_paused) {
Do we need dout() message here as symmetric to the added one?
net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1670
Quoted 11 lines
net/ceph/osd_client.c +6
-0unpaused = true; } +if (t->paused != should_be_paused) { +dout ("%s t %p paused %d -> %d\n" , __func__, t, t->paused, + should_be_paused); + t->paused = should_be_paused; + } + legacy_change =ceph_pg_compare (&t->pgid, &pgid) ||ceph_osds_changed (&t->acting, &acting, t->used_replica || any_change);
Looks good.
Thanks,
Slava.
Ilya Dryomov
maintainerQuoted 52 lines · Viacheslav Dubeyko
Quoted 50 lines · Ilya Dryomov
Currently
calc_target()clearst->pausedif the request shouldn't be paused anymore, but doesn't ever sett->pausedeven though it's able to determine when the request should be paused. Settingt->pausedis left to__submit_request()which is fine for regular requests but doesn't work for linger requests -- since__submit_request()doesn't operate on linger requests, there is nowhere forlreq->t.pausedto be set. One consequence of this is that watches don't get reestablished on paused -> unpaused transitions in cases where requests have been paused long enough for the (paused) unwatch request to time out and for the subsequent (re)watch request to enter the paused state. On top of the watch not getting reestablished,rbd_reregister_watch()gets stuck withrbd_dev->watch_mutexheld:
rbd_register_watch
__rbd_register_watch
ceph_osdc_watch
linger_reg_commit_waitIt's waiting for
lreq->reg_commit_waitto be completed, but for that to happen the respective request needs to end up onneed_resend_lingerlist and be kicked when requests are unpaused. There is no chance for that if the request in question is never marked paused in the first place.The fact that
rbd_dev->watch_mutexremains taken out forever then prevents the image from getting unmapped -- "rbd unmap" would inevitably hang in D state on an attempt to grab the mutex.Reported-by Raphael Zimmer <[email protected]>Signed-off-by Ilya Dryomov <[email protected]>--- net/ceph/osd_client.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)net/ceph/osd_client.c +1
-0@@ -1586,6 +1586,7 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc,struct ceph_pg_pool_info *pi;struct ceph_pg pgid, last_pgid;struct ceph_osds up, acting; +bool should_be_paused;bool is_read = t->flags &CEPH_OSD_FLAG_READ ;bool is_write = t->flags &CEPH_OSD_FLAG_WRITE ;bool force_resend = false;@@ -1654,10 +1655,16 @@ static enum calc_target_result calc_target(struct ceph_osd_client *osdc, &last_pgid)) force_resend = true;
Quoted 6 lines
net/ceph/osd_client.c static enum calc_target_result calc_target(struct ceph_osd_client *osdc, · line 1659
Quoted 4 lines
net/ceph/osd_client.c +2
-2-if (t->paused && !target_should_be_paused (osdc, t, pi)) { - t->paused = false; + should_be_paused =target_should_be_paused (osdc, t, pi); +if (t->paused && !should_be_paused) {Do we need
dout()message here as symmetric to the added one?
No -- the value of unpaused is already logged before returning.
Thanks,
Ilya