ceph: ensure we flush delayed caps when unmounting
PATCH 1/1 ceph: ensure we flush delayed caps when unmounting
I've seen some warnings when testing recently that indicate that there are caps still delayed on the delayed list even after we've started unmounting.
When checking delayed caps, process the whole list if we're unmounting, and check for delayed caps after setting the stopping var and flushing dirty caps.
--- fs/ceph/caps.c | 3 ++- fs/ceph/mds_client.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
fs/ceph/caps.c +2 -1
@@ -4236,7 +4236,8 @@ void ceph_check_delayed_caps(struct ceph_mds_client *mdsc) ci =list_first_entry (&mdsc->cap_delay_list, struct ceph_inode_info, i_cap_delay_list); -if ((ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 && +if (!mdsc->stopping && + (ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 &&time_before (jiffies, ci->i_hold_caps_max))break ;list_del_init (&ci->i_cap_delay_list);
fs/ceph/mds_client.c +1 -0
@@ -4691,6 +4691,7 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc); +ceph_check_delayed_caps (mdsc);wait_requests (mdsc);/*
Signature · 1 line
2.31.1
Quoted 33 lines · Jeff Layton
I've seen some warnings when testing recently that indicate that there are caps still delayed on the delayed list even after we've started unmounting.
When checking delayed caps, process the whole list if we're unmounting, and check for delayed caps after setting the stopping var and flushing dirty caps.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/caps.c | 3 ++- fs/ceph/mds_client.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)fs/ceph/caps.c +2
-1@@ -4236,7 +4236,8 @@ void ceph_check_delayed_caps(struct ceph_mds_client *mdsc) ci =list_first_entry (&mdsc->cap_delay_list, struct ceph_inode_info, i_cap_delay_list); -if ((ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 && +if (!mdsc->stopping && + (ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 &&time_before (jiffies, ci->i_hold_caps_max))break ;list_del_init (&ci->i_cap_delay_list);fs/ceph/mds_client.c +0
-0
fs/ceph/mds_client.c void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc) · line 4697
Quoted 7 lines
fs/ceph/mds_client.c +1
-0@@ -4691,6 +4691,7 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc); +ceph_check_delayed_caps (mdsc);wait_requests (mdsc);/*
I'm going to self-NAK this patch for now. Initially this looked good in testing, but I think it's just papering over the real problem, which is that ceph_async_iput can queue a job to a workqueue after the point where we've flushed that workqueue on umount.
I think the right approach is to look at how to ensure that calling iput doesn't end up taking these coarse-grained locks so we don't need to queue it in so many codepaths.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 40 lines · Jeff Layton
Quoted 39 lines · Jeff Layton
I've seen some warnings when testing recently that indicate that there are caps still delayed on the delayed list even after we've started unmounting.
When checking delayed caps, process the whole list if we're unmounting, and check for delayed caps after setting the stopping var and flushing dirty caps.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/caps.c | 3 ++- fs/ceph/mds_client.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)fs/ceph/caps.c +2
-1@@ -4236,7 +4236,8 @@ void ceph_check_delayed_caps(struct ceph_mds_client *mdsc) ci =list_first_entry (&mdsc->cap_delay_list, struct ceph_inode_info, i_cap_delay_list); -if ((ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 && +if (!mdsc->stopping && + (ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 &&time_before (jiffies, ci->i_hold_caps_max))break ;list_del_init (&ci->i_cap_delay_list);fs/ceph/mds_client.c +1
-0@@ -4691,6 +4691,7 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc); +ceph_check_delayed_caps (mdsc);wait_requests (mdsc);
Quoted 6 lines
fs/ceph/mds_client.c void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc) · line 4697
Quoted 1 line
fs/ceph/mds_client.c +0
-0/* I'm going to self-NAK this patch for now. Initially this looked good in testing, but I think it's just papering over the real problem, which is that
ceph_async_iputcan queue a job to a workqueue after the point where we've flushed that workqueue on umount.
Ah, yeah. I think I saw this a few times with generic/014 (and I believe we chatted about it on irc). I've been on and off trying to figure out the way to fix it but it's really tricky.
Cheers,
--
Luís
Quoted 6 lines
I think the right approach is to look at how to ensure that calling iput doesn't end up taking these coarse-grained locks so we don't need to queue it in so many codepaths.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 48 lines · Luis Henriques
Quoted 40 lines · Jeff Layton
Quoted 39 lines · Jeff Layton
I've seen some warnings when testing recently that indicate that there are caps still delayed on the delayed list even after we've started unmounting.
When checking delayed caps, process the whole list if we're unmounting, and check for delayed caps after setting the stopping var and flushing dirty caps.
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/caps.c | 3 ++- fs/ceph/mds_client.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)fs/ceph/caps.c +2
-1@@ -4236,7 +4236,8 @@ void ceph_check_delayed_caps(struct ceph_mds_client *mdsc) ci =list_first_entry (&mdsc->cap_delay_list, struct ceph_inode_info, i_cap_delay_list); -if ((ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 && +if (!mdsc->stopping && + (ci->i_ceph_flags &CEPH_I_FLUSH ) ==0 &&time_before (jiffies, ci->i_hold_caps_max))break ;list_del_init (&ci->i_cap_delay_list);fs/ceph/mds_client.c +1
-0@@ -4691,6 +4691,7 @@ void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc)lock_unlock_sessions (mdsc);ceph_flush_dirty_caps (mdsc); +ceph_check_delayed_caps (mdsc);wait_requests (mdsc);Quoted 6 lines
fs/ceph/mds_client.c void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc) · line 4697
Quoted 1 line
fs/ceph/mds_client.c +0
-0/* I'm going to self-NAK this patch for now. Initially this looked good in testing, but I think it's just papering over the real problem, which is that
ceph_async_iputcan queue a job to a workqueue after the point where we've flushed that workqueue on umount.
Quoted 4 lines
Ah, yeah. I think I saw this a few times with generic/014 (and I believe we chatted about it on irc). I've been on and off trying to figure out the way to fix it but it's really tricky.
Yeah, that's putting it mildly.
The biggest issue here is the session->s_mutex, which is held over large swaths of the code, but it's not fully clear what it protects. The original patch that added ceph_async_iput did it to avoid the session mutex that gets held for ceph_iterate_session_caps.
My current thinking is that we probably don't need to hold the session mutex over that function in some cases, if we can guarantee that the ceph_cap objects we're iterating over don't go away when the lock is dropped. So, I'm trying to add some refcounting to the ceph_cap structures themselves to see if that helps.
It may turn out to be a dead end, but if we don't chip away at the edges of the fundamental problem, we'll never get there...
Signature · 1 line
Jeff Layton <[email protected]>