ceph: defer stopping the mdsc delayed_work
Subsystems (2)
PATCH v3 1/1 ceph: defer stopping the mdsc delayed_work
From: Xiubo Li <[email protected]>
Flushing the dirty buffer may take a long time if the Rados is overloaded or if there is network issue. So we should ping the MDSs periodically to keep alive, else the MDS will blocklist the kclient.
Cc: [email protected]
Cc: Venky Shankar <[email protected]>
URL: https://tracker.ceph.com/issues/61843
---
V3:
- Rebased to the master branch
fs/ceph/mds_client.c | 4 ++-- fs/ceph/mds_client.h | 5 +++++ fs/ceph/super.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 2 deletions(-)
fs/ceph/mds_client.c +2 -2
@@ -4764,7 +4764,7 @@ static void delayed_work(struct work_struct *work)dout ("mdsc delayed_work\n" ); -if (mdsc->stopping) +if (mdsc->stopping >=CEPH_MDSC_STOPPING_FLUSHED )return ;mutex_lock (&mdsc->mutex);
@@ -4943,7 +4943,7 @@ void send_flush_mdlog(struct ceph_mds_session *s)void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) {dout ("pre_umount\n" ); - mdsc->stopping =1 ; + mdsc->stopping =CEPH_MDSC_STOPPING_BEGIN ;ceph_mdsc_iterate_sessions (mdsc, send_flush_mdlog, true);ceph_mdsc_iterate_sessions (mdsc, lock_unlock_session, false);
fs/ceph/mds_client.h +5 -0
@@ -380,6 +380,11 @@ struct cap_wait {int want; }; +enum { +CEPH_MDSC_STOPPING_BEGIN =1 , +CEPH_MDSC_STOPPING_FLUSHED =2 , +}; +/* * mds client state */
fs/ceph/super.c +10 -0
@@ -1374,6 +1374,16 @@ static void ceph_kill_sb(struct super_block *s)ceph_mdsc_pre_umount (fsc->mdsc);flush_fs_workqueues (fsc); +/* +* Though the kill_anon_super() will finally trigger the +* sync_filesystem() anyway, we still need to do it here +* and then bump the stage of shutdown to stop the work +* queue as earlier as possible. +*/ +sync_filesystem (s); + + fsc->mdsc->stopping =CEPH_MDSC_STOPPING_FLUSHED ; +kill_anon_super (s); fsc->client->extra_mon_dispatch =NULL ;
Signature · 1 line
2.39.1
Quoted 29 lines · On Tue, Jul 25, 2023 at 9:36 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Flushing the dirty buffer may take a long time if the Rados is overloaded or if there is network issue. So we should ping the MDSs periodically to keep alive, else the MDS will blocklist the kclient.
Cc: [email protected]
Cc: Venky Shankar <[email protected]>
URL: https://tracker.ceph.com/issues/61843Reviewed-by Milind Changire <[email protected]>Signed-off-by Xiubo Li <[email protected]>---
V3:
- Rebased to the master branchfs/ceph/mds_client.c | 4 ++-- fs/ceph/mds_client.h | 5 +++++ fs/ceph/super.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 2 deletions(-)fs/ceph/mds_client.c +0
-0
fs/ceph/mds_client.c static void delayed_work(struct work_struct *work) · line 4768
Quoted 6 lines
fs/ceph/mds_client.c +1
-1@@ -4764,7 +4764,7 @@ static void delayed_work(struct work_struct *work)dout ("mdsc delayed_work\n" ); -if (mdsc->stopping) +if (mdsc->stopping >=CEPH_MDSC_STOPPING_FLUSHED )return ;
Do we want to continue to accept/perform delayed work when mdsc->stopping is set to CEPH_MDSC_STOPPING_BEGIN ?
I thought setting the STOPPING_BEGIN flag would immediately bar any further new activity and STOPPING_FLUSHED would mark safe deletion of the superblock.
Quoted 51 lines
fs/ceph/mds_client.c +1
-1mutex_lock (&mdsc->mutex);@@ -4943,7 +4943,7 @@ void send_flush_mdlog(struct ceph_mds_session *s)void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) {dout ("pre_umount\n" ); - mdsc->stopping =1 ; + mdsc->stopping =CEPH_MDSC_STOPPING_BEGIN ;ceph_mdsc_iterate_sessions (mdsc, send_flush_mdlog, true);ceph_mdsc_iterate_sessions (mdsc, lock_unlock_session, false);fs/ceph/mds_client.h +5
-0@@ -380,6 +380,11 @@ struct cap_wait {int want; }; +enum { +CEPH_MDSC_STOPPING_BEGIN =1 , +CEPH_MDSC_STOPPING_FLUSHED =2 , +}; +/* * mds client state */ fs/ceph/super.c +10
-1@@ -1374,6 +1374,16 @@ static void ceph_kill_sb(struct super_block *s)ceph_mdsc_pre_umount (fsc->mdsc);flush_fs_workqueues (fsc); +/* +* Though the kill_anon_super() will finally trigger the +* sync_filesystem() anyway, we still need to do it here +* and then bump the stage of shutdown to stop the work +* queue as earlier as possible. +*/ +sync_filesystem (s); + + fsc->mdsc->stopping =CEPH_MDSC_STOPPING_FLUSHED ; +kill_anon_super (s); fsc->client->extra_mon_dispatch =NULL ; --2.39.1
Signature · 1 line
Milind
Quoted 38 lines · Milind Changire
Quoted 28 lines · On Tue, Jul 25, 2023 at 9:36 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Flushing the dirty buffer may take a long time if the Rados is overloaded or if there is network issue. So we should ping the MDSs periodically to keep alive, else the MDS will blocklist the kclient.
Cc: [email protected]
Cc: Venky Shankar <[email protected]>
URL: https://tracker.ceph.com/issues/61843Reviewed-by Milind Changire <[email protected]>Signed-off-by Xiubo Li <[email protected]>---
V3:
- Rebased to the master branchfs/ceph/mds_client.c | 4 ++-- fs/ceph/mds_client.h | 5 +++++ fs/ceph/super.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 2 deletions(-)fs/ceph/mds_client.c +0
-0fs/ceph/mds_client.c static void delayed_work(struct work_struct *work) · line 4768
Quoted 6 lines
fs/ceph/mds_client.c +1
-1@@ -4764,7 +4764,7 @@ static void delayed_work(struct work_struct *work)dout ("mdsc delayed_work\n" ); -if (mdsc->stopping) +if (mdsc->stopping >=CEPH_MDSC_STOPPING_FLUSHED )return ;Do we want to continue to accept/perform delayed work when
mdsc->stoppingis set toCEPH_MDSC_STOPPING_BEGIN?
Quoted 3 lines
I thought setting the
STOPPING_BEGINflag would immediately bar any further new activity andSTOPPING_FLUSHEDwould mark safe deletion of the superblock.
Yes, we need.
Locally I can reproduce this very easy with fsstress.sh script, please see https://tracker.ceph.com/issues/61843#note-1.
That's because when umounting and flushing the dirty buffer it could be blocked by the Rados dues to the lower disk space or MM reasons. And during this we need to ping MDS to keep alive to make sure the MDS won't evict us before we close the sessions later.
Thanks
- Xiubo
Quoted 53 lines
Quoted 52 lines
fs/ceph/mds_client.c +1
-1mutex_lock (&mdsc->mutex);@@ -4943,7 +4943,7 @@ void send_flush_mdlog(struct ceph_mds_session *s)void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) {dout ("pre_umount\n" ); - mdsc->stopping =1 ; + mdsc->stopping =CEPH_MDSC_STOPPING_BEGIN ;ceph_mdsc_iterate_sessions (mdsc, send_flush_mdlog, true);ceph_mdsc_iterate_sessions (mdsc, lock_unlock_session, false);fs/ceph/mds_client.h +5
-0@@ -380,6 +380,11 @@ struct cap_wait {int want; }; +enum { +CEPH_MDSC_STOPPING_BEGIN =1 , +CEPH_MDSC_STOPPING_FLUSHED =2 , +}; +/* * mds client state */ fs/ceph/super.c +10
-1@@ -1374,6 +1374,16 @@ static void ceph_kill_sb(struct super_block *s)ceph_mdsc_pre_umount (fsc->mdsc);flush_fs_workqueues (fsc); +/* +* Though the kill_anon_super() will finally trigger the +* sync_filesystem() anyway, we still need to do it here +* and then bump the stage of shutdown to stop the work +* queue as earlier as possible. +*/ +sync_filesystem (s); + + fsc->mdsc->stopping =CEPH_MDSC_STOPPING_FLUSHED ; +kill_anon_super (s); fsc->client->extra_mon_dispatch =NULL ; --2.39.1
Milind Changire
Reviewed-byLooks good to me.
Quoted 113 lines · Xiubo Li
Quoted 38 lines · Milind Changire
Quoted 28 lines · On Tue, Jul 25, 2023 at 9:36 AM <[email protected]> wrote:
From: Xiubo Li <[email protected]>
Flushing the dirty buffer may take a long time if the Rados is overloaded or if there is network issue. So we should ping the MDSs periodically to keep alive, else the MDS will blocklist the kclient.
Cc: [email protected]
Cc: Venky Shankar <[email protected]>
URL: https://tracker.ceph.com/issues/61843Reviewed-by Milind Changire <[email protected]>Signed-off-by Xiubo Li <[email protected]>---
V3:
- Rebased to the master branchfs/ceph/mds_client.c | 4 ++-- fs/ceph/mds_client.h | 5 +++++ fs/ceph/super.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 2 deletions(-)fs/ceph/mds_client.c +0
-0fs/ceph/mds_client.c static void delayed_work(struct work_struct *work) · line 4768
Quoted 6 lines
fs/ceph/mds_client.c +1
-1@@ -4764,7 +4764,7 @@ static void delayed_work(struct work_struct *work)dout ("mdsc delayed_work\n" ); -if (mdsc->stopping) +if (mdsc->stopping >=CEPH_MDSC_STOPPING_FLUSHED )return ;Do we want to continue to accept/perform delayed work when
mdsc->stoppingis set toCEPH_MDSC_STOPPING_BEGIN?Quoted 3 lines
I thought setting the
STOPPING_BEGINflag would immediately bar any further new activity andSTOPPING_FLUSHEDwould mark safe deletion of the superblock.Yes, we need.
Locally I can reproduce this very easy with
fsstress.shscript, please see https://tracker.ceph.com/issues/61843#note-1.That's because when umounting and flushing the dirty buffer it could be blocked by the Rados dues to the lower disk space or MM reasons. And during this we need to ping MDS to keep alive to make sure the MDS won't evict us before we close the sessions later.
Thanks
- Xiubo
Quoted 53 lines
Quoted 52 lines
fs/ceph/mds_client.c +1
-1mutex_lock (&mdsc->mutex);@@ -4943,7 +4943,7 @@ void send_flush_mdlog(struct ceph_mds_session *s)void ceph_mdsc_pre_umount (struct ceph_mds_client *mdsc) {dout ("pre_umount\n" ); - mdsc->stopping =1 ; + mdsc->stopping =CEPH_MDSC_STOPPING_BEGIN ;ceph_mdsc_iterate_sessions (mdsc, send_flush_mdlog, true);ceph_mdsc_iterate_sessions (mdsc, lock_unlock_session, false);fs/ceph/mds_client.h +5
-0@@ -380,6 +380,11 @@ struct cap_wait {int want; }; +enum { +CEPH_MDSC_STOPPING_BEGIN =1 , +CEPH_MDSC_STOPPING_FLUSHED =2 , +}; +/* * mds client state */ fs/ceph/super.c +10
-1@@ -1374,6 +1374,16 @@ static void ceph_kill_sb(struct super_block *s)ceph_mdsc_pre_umount (fsc->mdsc);flush_fs_workqueues (fsc); +/* +* Though the kill_anon_super() will finally trigger the +* sync_filesystem() anyway, we still need to do it here +* and then bump the stage of shutdown to stop the work +* queue as earlier as possible. +*/ +sync_filesystem (s); + + fsc->mdsc->stopping =CEPH_MDSC_STOPPING_FLUSHED ; +kill_anon_super (s); fsc->client->extra_mon_dispatch =NULL ; --2.39.1
Signature · 1 line
Milind