ceph: fix spurious recover_session=clean errors
Subsystems (2)
v4 cover ceph: fix spurious recover_session=clean errors
v4: test for CEPH_MOUNT_RECOVER in more places
v3: add RECOVER mount_state and allow dumping pagecache when it's set
shrink size of mount_state field
v2: fix handling of async requests in patch to queue requests
This is the fourth revision of this patchset. The main difference from v3 is that this one converts more "==" tests for SHUTDOWN state into ">=", so that the RECOVER state is treated the same way.
Original cover letter:
Ilya noticed that he would get spurious EACCES errors on calls done just after blocklisting the client on mounts with recover_session=clean. The session would get marked as REJECTED and that caused in-flight calls to die with EACCES. This patchset seems to smooth over the problem, but I'm not fully convinced it's the right approach.
The potential issue I see is that the client could take cap references to do a call on a session that has been blocklisted. We then queue the message and reestablish the session, but we may not have been granted the same caps by the MDS at that point.
If this is a problem, then we probably need to rework it so that we return a distinct error code in this situation and have the upper layers issue a completely new mds request (with new cap refs, etc.)
Obviously, that's a much more invasive approach though, so it would be nice to avoid that if this would suffice.
Jeff Layton (5):
ceph: don't WARN when removing caps due to blocklisting
ceph: make fsc->mount_state an int
ceph: add new RECOVER mount_state when recovering session
ceph: remove timeout on allowing reconnect after blocklisting
ceph: queue MDS requests to REJECTED sessions when CLEANRECOVER is set
fs/ceph/addr.c | 4 ++-- fs/ceph/caps.c | 4 ++-- fs/ceph/inode.c | 2 +- fs/ceph/mds_client.c | 27 ++++++++++++++++----------- fs/ceph/super.c | 14 ++++++++++---- fs/ceph/super.h | 3 +-- include/linux/ceph/libceph.h | 1 + 7 files changed, 33 insertions(+), 22 deletions(-)
Signature · 1 line
2.26.2
PATCH v4 1/5 ceph: don't WARN when removing caps due to blocklisting
We expect to remove dirty caps when the client is blocklisted. Don't throw a warning in that case.
--- fs/ceph/caps.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/caps.c +1 -1
@@ -1149,7 +1149,7 @@ void __ceph_remove_cap(struct ceph_cap *cap, bool queue_release)/* remove from inode's cap rbtree, and clear auth cap */ rb_erase (&cap->ci_node, &ci->i_caps);if (ci->i_auth_cap == cap) { -WARN_ON_ONCE (!list_empty (&ci->i_dirty_item)); +WARN_ON_ONCE (!list_empty (&ci->i_dirty_item) && !mdsc->fsc->blocklisted); ci->i_auth_cap =NULL ; }
Signature · 1 line
2.26.2
PATCH v4 2/5 ceph: make fsc->mount_state an int
In reply to Jeff LaytonThis field is an unsigned long currently, which is a bit of a waste on most arches since this just holds an enum. Make it (signed) int instead.
--- fs/ceph/super.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/ceph/super.h +1 -1
@@ -106,7 +106,7 @@ struct ceph_fs_client {struct ceph_mount_options *mount_options;struct ceph_client *client; -unsigned long mount_state; +int mount_state;unsigned long last_auto_reconnect;bool blocklisted;
Signature · 1 line
2.26.2
PATCH v4 3/5 ceph: add new RECOVER mount_state when recovering session
In reply to Jeff LaytonWhen recovering a session (a'la recover_session=clean), we want to do all of the operations that we do on a forced umount, but changing the mount state to SHUTDOWN is can cause queued MDS requests to fail when the session comes back. Most of those can idle until the session is recovered in this situation.
Reserve SHUTDOWN state for forced umount, and make a new RECOVER state for the forced reconnect situation. Change several tests for equality with SHUTDOWN to test for that or RECOVER.
--- fs/ceph/addr.c | 4 ++-- fs/ceph/caps.c | 2 +- fs/ceph/inode.c | 2 +- fs/ceph/mds_client.c | 4 ++-- fs/ceph/super.c | 14 ++++++++++---- include/linux/ceph/libceph.h | 1 + 6 files changed, 17 insertions(+), 10 deletions(-)
fs/ceph/addr.c +2 -2
@@ -841,7 +841,7 @@ static int ceph_writepages_start(struct address_space *mapping, wbc->sync_mode ==WB_SYNC_NONE ?"NONE" : (wbc->sync_mode == WB_SYNC_ALL ? "ALL" : "HOLD")); -if (READ_ONCE (fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) { +if (READ_ONCE (fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN ) {if (ci->i_wrbuffer_ref >0 ) { pr_warn_ratelimited( "writepage_start %p %lld forced umount\n",
@@ -1265,7 +1265,7 @@ ceph_find_incompatible(struct page *page)struct ceph_fs_client *fsc =ceph_inode_to_client (inode);struct ceph_inode_info *ci =ceph_inode (inode); -if (READ_ONCE (fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) { +if (READ_ONCE (fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN ) {dout (" page %p forced umount\n" , page);return ERR_PTR (-EIO ); }
fs/ceph/caps.c +1 -1
@@ -2735,7 +2735,7 @@ static int try_get_cap_refs(struct inode *inode, int need, int want, goto out_unlock; } -if (READ_ONCE (mdsc->fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) { +if (READ_ONCE (mdsc->fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN ) {dout ("get_cap_refs %p forced umount\n" , inode); ret = -EIO ; goto out_unlock;
fs/ceph/inode.c +1 -1
@@ -1888,7 +1888,7 @@ static void ceph_do_invalidate_pages(struct inode *inode)mutex_lock (&ci->i_truncate_mutex); -if (READ_ONCE (fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) { +if (READ_ONCE (fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN ) {pr_warn_ratelimited ("invalidate_pages %p %lld forced umount\n" , inode,ceph_ino (inode));mapping_set_error (inode->i_mapping, -EIO );
fs/ceph/mds_client.c +2 -2
@@ -1595,7 +1595,7 @@ static int remove_session_caps_cb(struct inode *inode, struct ceph_cap *cap,struct ceph_cap_flush *cf;struct ceph_mds_client *mdsc = fsc->mdsc; -if (READ_ONCE (fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) { +if (READ_ONCE (fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN ) {if (inode->i_data.nrpages >0 ) invalidate = true;if (ci->i_wrbuffer_ref >0 )
@@ -4658,7 +4658,7 @@ void ceph_mdsc_sync(struct ceph_mds_client *mdsc) {u64 want_tid, want_flush; -if (READ_ONCE (mdsc->fsc->mount_state) ==CEPH_MOUNT_SHUTDOWN ) +if (READ_ONCE (mdsc->fsc->mount_state) >=CEPH_MOUNT_SHUTDOWN )return ;dout ("sync\n" );
fs/ceph/super.c +10 -4
@@ -832,6 +832,13 @@ static void destroy_caches(void)ceph_fscache_unregister (); } +static void __ceph_umount_begin (struct ceph_fs_client *fsc) +{ +ceph_osdc_abort_requests (&fsc->client->osdc, -EIO ); +ceph_mdsc_force_umount (fsc->mdsc); + fsc->filp_gen++;// invalidate open files +} +/* * ceph_umount_begin - initiate forced umount. Tear down the * mount, skipping steps that may hang while waiting for server(s).
@@ -844,9 +851,7 @@ static void ceph_umount_begin(struct super_block *sb)if (!fsc)return ; fsc->mount_state =CEPH_MOUNT_SHUTDOWN ; -ceph_osdc_abort_requests (&fsc->client->osdc, -EIO ); -ceph_mdsc_force_umount (fsc->mdsc); - fsc->filp_gen++;// invalidate open files +__ceph_umount_begin (fsc); }static const struct super_operations ceph_super_ops = {
@@ -1235,7 +1240,8 @@ int ceph_force_reconnect(struct super_block *sb)struct ceph_fs_client *fsc =ceph_sb_to_client (sb);int err =0 ; -ceph_umount_begin (sb); + fsc->mount_state =CEPH_MOUNT_RECOVER ; +__ceph_umount_begin (fsc);/* Make sure all page caches get invalidated. * see remove_session_caps_cb() */
include/linux/ceph/libceph.h +1 -0
@@ -104,6 +104,7 @@ enum {CEPH_MOUNT_UNMOUNTING ,CEPH_MOUNT_UNMOUNTED ,CEPH_MOUNT_SHUTDOWN , + CEPH_MOUNT_RECOVER, };static inline unsigned long ceph_timeout_jiffies (unsigned long timeout)
Signature · 1 line
2.26.2
PATCH v4 4/5 ceph: remove timeout on allowing reconnect after blocklisting
In reply to Jeff Layton30 minutes is a long time to wait, and this makes it difficult to test the feature by manually blocklisting clients. Remove the timeout infrastructure and just allow the client to reconnect at will.
--- fs/ceph/mds_client.c | 5 ----- fs/ceph/super.h | 1 - 2 files changed, 6 deletions(-)
fs/ceph/mds_client.c +0 -5
@@ -4374,12 +4374,7 @@ static void maybe_recover_session(struct ceph_mds_client *mdsc)if (!READ_ONCE (fsc->blocklisted))return ; -if (fsc->last_auto_reconnect && -time_before (jiffies, fsc->last_auto_reconnect +HZ *60 *30 )) -return ; -pr_info ("auto reconnect after blocklisted\n" ); - fsc->last_auto_reconnect = jiffies;ceph_force_reconnect (fsc->sb); }
fs/ceph/super.h +0 -1
@@ -108,7 +108,6 @@ struct ceph_fs_client {int mount_state; -unsigned long last_auto_reconnect;bool blocklisted;bool have_copy_from2;
Signature · 1 line
2.26.2
PATCH v4 5/5 ceph: queue MDS requests to REJECTED sessions when CLEANRECOVER is set
In reply to Jeff LaytonIlya noticed that the first access to a blacklisted mount would often get back -EACCES, but then subsequent calls would be OK. The problem is in __do_request. If the session is marked as REJECTED, a hard error is returned instead of waiting for a new session to come into being.
When the session is REJECTED and the mount was done with recover_session=clean, queue the request to the waiting_for_map queue, which will be awoken after tearing down the old session. We can only do this for sync requests though, so check for async ones first and just let the callers redrive a sync request.
URL: https://tracker.ceph.com/issues/47385
--- fs/ceph/mds_client.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
fs/ceph/mds_client.c +14 -4
@@ -2818,10 +2818,6 @@ static void __do_request(struct ceph_mds_client *mdsc, ceph_session_state_name(session->s_state));if (session->s_state !=CEPH_MDS_SESSION_OPEN && session->s_state !=CEPH_MDS_SESSION_HUNG ) { -if (session->s_state ==CEPH_MDS_SESSION_REJECTED ) { - err = -EACCES ; - goto out_session; - }/* * We cannot queue async requests since the caps and delegated * inodes are bound to the session. Just return -EJUKEBOX and
@@ -2831,6 +2827,20 @@ static void __do_request(struct ceph_mds_client *mdsc, err = -EJUKEBOX ; goto out_session; } + +/* +* If the session has been REJECTED, then return a hard error, +* unless it's a CLEANRECOVER mount, in which case we'll queue +* it to the mdsc queue. +*/ +if (session->s_state ==CEPH_MDS_SESSION_REJECTED ) { +if (ceph_test_mount_opt (mdsc->fsc,CLEANRECOVER )) +list_add (&req->r_wait, &mdsc->waiting_for_map); +else + err = -EACCES ; + goto out_session; + } +if (session->s_state ==CEPH_MDS_SESSION_NEW || session->s_state ==CEPH_MDS_SESSION_CLOSING ) { err =__open_session (mdsc, session);
Signature · 1 line
2.26.2
Quoted 36 lines · Jeff Layton
v4: test for
CEPH_MOUNT_RECOVERin more places
v3: add RECOVERmount_stateand allow dumping pagecache when it's set
shrink size ofmount_statefield
v2: fix handling of async requests in patch to queue requestsThis is the fourth revision of this patchset. The main difference from v3 is that this one converts more "==" tests for SHUTDOWN state into ">=", so that the RECOVER state is treated the same way.
Original cover letter:
Ilya noticed that he would get spurious EACCES errors on calls done just after blocklisting the client on mounts with recover_session=clean. The session would get marked as REJECTED and that caused in-flight calls to die with EACCES. This patchset seems to smooth over the problem, but I'm not fully convinced it's the right approach.
The potential issue I see is that the client could take cap references to do a call on a session that has been blocklisted. We then queue the message and reestablish the session, but we may not have been granted the same caps by the MDS at that point.
If this is a problem, then we probably need to rework it so that we return a distinct error code in this situation and have the upper layers issue a completely new mds request (with new cap refs, etc.)
Obviously, that's a much more invasive approach though, so it would be nice to avoid that if this would suffice.
Jeff Layton (5):
ceph: don't WARN when removing caps due to blocklisting
ceph: makefsc->mount_statean int
ceph: add new RECOVERmount_statewhen recovering session
ceph: remove timeout on allowing reconnect after blocklisting
ceph: queue MDS requests to REJECTED sessions when CLEANRECOVER is set
Quoted 9 lines
fs/ceph/addr.c | 4 ++-- fs/ceph/caps.c | 4 ++-- fs/ceph/inode.c | 2 +- fs/ceph/mds_client.c | 27 ++++++++++++++++----------- fs/ceph/super.c | 14 ++++++++++---- fs/ceph/super.h | 3 +-- include/linux/ceph/libceph.h | 1 + 7 files changed, 33 insertions(+), 22 deletions(-)
Quoted 30 lines · Jeff Layton
v4: test for
CEPH_MOUNT_RECOVERin more places
v3: add RECOVERmount_stateand allow dumping pagecache when it's set
shrink size ofmount_statefield
v2: fix handling of async requests in patch to queue requestsThis is the fourth revision of this patchset. The main difference from v3 is that this one converts more "==" tests for SHUTDOWN state into ">=", so that the RECOVER state is treated the same way.
Original cover letter:
Ilya noticed that he would get spurious EACCES errors on calls done just after blocklisting the client on mounts with recover_session=clean. The session would get marked as REJECTED and that caused in-flight calls to die with EACCES. This patchset seems to smooth over the problem, but I'm not fully convinced it's the right approach.
The potential issue I see is that the client could take cap references to do a call on a session that has been blocklisted. We then queue the message and reestablish the session, but we may not have been granted the same caps by the MDS at that point.
If this is a problem, then we probably need to rework it so that we return a distinct error code in this situation and have the upper layers issue a completely new mds request (with new cap refs, etc.)
Obviously, that's a much more invasive approach though, so it would be nice to avoid that if this would suffice.
Quoted 7 lines
Jeff Layton (5):
ceph: don't WARN when removing caps due to blocklisting
ceph: makefsc->mount_statean int
ceph: add new RECOVERmount_statewhen recovering session
ceph: remove timeout on allowing reconnect after blocklisting
ceph: queue MDS requests to REJECTED sessions when CLEANRECOVER is set
series
Quoted 12 lines
fs/ceph/addr.c | 4 ++-- fs/ceph/caps.c | 4 ++-- fs/ceph/inode.c | 2 +- fs/ceph/mds_client.c | 27 ++++++++++++++++----------- fs/ceph/super.c | 14 ++++++++++---- fs/ceph/super.h | 3 +-- include/linux/ceph/libceph.h | 1 + 7 files changed, 33 insertions(+), 22 deletions(-)--
2.26.2