NFSD: Initial implementation of NFSv4 Courteous Server
Subsystems (2)
Dai Ngo
reviewerv14 cover NFSD: Initial implementation of NFSv4 Courteous Server
Hi Chuck, Bruce
This series of patches implement the NFSv4 Courteous Server.
A server which does not immediately expunge the state on lease expiration is known as a Courteous Server. A Courteous Server continues to recognize previously generated state tokens as valid until conflict arises between the expired state and the requests from another client, or the server reboots.
v2 patch includes:
. add new callback, lm_expire_lock, to lock_manager_operations to
allow the lock manager to take appropriate action with conflict lock.
. handle conflicts of NFSv4 locks with NFSv3/NLM and local locks.
. expire courtesy client after 24hr if client has not reconnected.
. do not allow expired client to become courtesy client if there are
waiters for client's locks.
. modify client_info_show to show courtesy client and seconds from
last renew.
. fix a problem with NFSv4.1 server where the it keeps returning
SEQ4_STATUS_CB_PATH_DOWN in the successful SEQUENCE reply, after the courtesy client reconnects, causing the client to keep sending BCTS requests to server.
v3 patch includes:
. modified posix_test_lock to check and resolve conflict locks
to handle NLM TEST and NFSv4 LOCKT requests.
. separate out fix for back channel stuck in SEQ4_STATUS_CB_PATH_DOWN.
v4 patch includes:
. rework nfsd_check_courtesy to avoid dead lock of fl_lock and client_lock
by asking the laudromat thread to destroy the courtesy client.
. handle NFSv4 share reservation conflicts with courtesy client. This
includes conflicts between access mode and deny mode and vice versa.
. drop the patch for back channel stuck in SEQ4_STATUS_CB_PATH_DOWN.
v5 patch includes:
. fix recursive locking of file_rwsem from posix_lock_file.
. retest with LOCKDEP enabled.
v6 patch includes:
. merge witn 5.15-rc7
. fix a bug in nfs4_check_deny_bmap that did not check for matched
nfs4_file before checking for access/deny conflict. This bug causes pynfs OPEN18 to fail since the server taking too long to release lots of un-conflict clients' state.
. enhance share reservation conflict handler to handle case where
a large number of conflict courtesy clients need to be expired. The 1st 100 clients are expired synchronously and the rest are expired in the background by the laundromat and NFS4ERR_DELAY is returned to the NFS client. This is needed to prevent the NFS client from timing out waiting got the reply.
v7 patch includes:
. Fix race condition in posix_test_lock and posix_lock_inode after
dropping spinlock.
. Enhance nfsd4_fl_expire_lock to work with with new lm_expire_lock
callback
. Always resolve share reservation conflicts asynchrously.
. Fix bug in nfs4_laundromat where spinlock is not used when
scanning cl_ownerstr_hashtbl.
. Fix bug in nfs4_laundromat where idr_get_next was called
with incorrect 'id'.
. Merge nfs4_destroy_courtesy_client into nfsd4_fl_expire_lock.
v8 patch includes:
. Fix warning in nfsd4_fl_expire_lock reported by test robot.
v9 patch includes:
. Simplify lm_expire_lock API by (1) remove the 'testonly' flag
and (2) specifying return value as true/false to indicate whether conflict was succesfully resolved.
. Rework nfsd4_fl_expire_lock to mark client with
NFSD4_DESTROY_COURTESY_CLIENT then tell the laundromat to expire the client in the background.
. Add a spinlock in nfs4_client to synchronize access to the
NFSD4_COURTESY_CLIENT and NFSD4_DESTROY_COURTESY_CLIENT flag to handle race conditions when resolving lock and share reservation conflict.
. Courtesy client that was marked as NFSD4_DESTROY_COURTESY_CLIENT
are now consisdered 'dead', waiting for the laundromat to expire it. This client is no longer allowed to use its states if it reconnects before the laundromat finishes expiring the client.
For v4.1 client, the detection is done in the processing of the SEQUENCE op and returns NFS4ERR_BAD_SESSION to force the client to re-establish new clientid and session.
For v4.0 client, the detection is done in the processing of the RENEW and state-related ops and return NFS4ERR_EXPIRE to force the client to re-establish new clientid.
v10 patch includes:
Resolve deadlock in v9 by avoiding getting cl_client and cl_cs_lock together. The laundromat needs to determine whether the expired client has any state and also has no blockers on its locks. Both of these conditions are allowed to change after the laundromat transits an expired client to courtesy client. When this happens, the laundromat will detect it on the next run and and expire the courtesy client.
Remove client persistent record before marking it as COURTESY_CLIENT and add client persistent record before clearing the COURTESY_CLIENT flag to allow the courtesy client to transist to normal client to continue to use its state.
Lock/delegation/share reversation conflict with courtesy client is resolved by marking the courtesy client as DESTROY_COURTESY_CLIENT, effectively disable it, then allow the current request to proceed immediately.
Courtesy client marked as DESTROY_COURTESY_CLIENT is not allowed to reconnect to reuse itsstate. It is expired by the laundromat asynchronously in the background.
Move processing of expired clients from nfs4_laudromat to a separate function, nfs4_get_client_reaplist, that creates the reaplist and also to process courtesy clients.
Update Documentation/filesystems/locking.rst to include new lm_lock_conflict call.
Modify leases_conflict to call lm_breaker_owns_lease only if there is real conflict. This is to allow the lock manager to resolve the delegation conflict if possible.
v11 patch includes:
Add comment for lm_lock_conflict callback.
Replace static const courtesy_client_expiry with macro.
Remove courtesy_clnt argument from find_in_sessionid_hashtbl. Callers use nfs4_client->cl_cs_client boolean to determined if it's the courtesy client and take appropriate actions.
Rename NFSD4_COURTESY_CLIENT and NFSD4_DESTROY_COURTESY_CLIENT with NFSD4_CLIENT_COURTESY and NFSD4_CLIENT_DESTROY_COURTESY.
v12 patch includes:
Remove unnecessary comment in nfs4_get_client_reaplist.
Replace nfs4_client->cl_cs_client boolean with
NFSD4_CLIENT_COURTESY_CLNT flag.
Remove courtesy_clnt argument from find_client_in_id_table and find_clp_in_name_tree. Callers use NFSD4_CLIENT_COURTESY_CLNT to determined if it's the courtesy client and take appropriate actions.
v13 patch includes:
Merge with 5.17-rc3.
Cleanup Documentation/filesystems/locking.rst: replace i_lock with flc_lock, update API's that use flc_lock.
Rename lm_lock_conflict to lm_lock_expired().
Remove comment of lm_lock_expired API in lock_manager_operations. Same information is in patch description.
Update commit messages of 4/4.
Add some comment for NFSD4_CLIENT_COURTESY_CLNT.
Add nfsd4_discard_courtesy_clnt() to eliminate duplicate code of discarding courtesy client; setting NFSD4_DESTROY_COURTESY_CLIENT.
v14 patch includes:
. merge with Chuck's public for-next branch.
. remove courtesy_client_expiry, use client's last renew time.
. simplify comment of nfs4_check_access_deny_bmap.
. add comment about race condition in nfs4_get_client_reaplist.
. add list_del when walking cslist in nfs4_get_client_reaplist.
. remove duplicate INIT_LIST_HEAD(&reaplist) from nfs4_laundromat
. Modify find_confirmed_client and find_confirmed_client_by_name
to detect courtesy client and destroy it.
. refactor lookup_clientid to use find_client_in_id_table
directly instead of find_confirmed_client.
. refactor nfsd4_setclientid to call find_clp_in_name_tree
directly instead of find_confirmed_client_by_name.
. remove comment of NFSD4_CLIENT_COURTESY.
. replace NFSD4_CLIENT_DESTROY_COURTESY with NFSD4_CLIENT_EXPIRED.
. replace NFSD4_CLIENT_COURTESY_CLNT with NFSD4_CLIENT_RECONNECTED.
Dai Ngo
reviewerRFC PATCH v14 1/10 NFSD: Add client flags, macro and spinlock to support courteous server
Add flags to track a client's lease state after the client fails to renew its lease. Add macro for courtesy client expire time. Add spinlock to synchronize access to these flags. Modify alloc_client to initialize these fields.
--- fs/nfsd/nfs4state.c | 2 ++ fs/nfsd/nfsd.h | 1 + fs/nfsd/state.h | 6 ++++++ 3 files changed, 9 insertions(+)
fs/nfsd/nfs4state.c +2 -0
@@ -2009,12 +2009,14 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)INIT_LIST_HEAD (&clp->cl_delegations);INIT_LIST_HEAD (&clp->cl_lru);INIT_LIST_HEAD (&clp->cl_revoked); +INIT_LIST_HEAD (&clp->cl_cs_list);#ifdef CONFIG_NFSD_PNFS INIT_LIST_HEAD (&clp->cl_lo_states);#endif INIT_LIST_HEAD (&clp->async_copies);spin_lock_init (&clp->async_lock);spin_lock_init (&clp->cl_lock); +spin_lock_init (&clp->cl_cs_lock);rpc_init_wait_queue (&clp->cl_cb_waitq,"Backchannel slot table" );return clp; err_no_hashtbl:
fs/nfsd/nfsd.h +1 -0
@@ -336,6 +336,7 @@ void nfsd_lockd_shutdown(void);#define COMPOUND_ERR_SLACK_SPACE 16/* OP_SETATTR */ #define NFSD_LAUNDROMAT_MINTIMEOUT 1/* seconds */ +#define NFSD_COURTESY_CLIENT_TIMEOUT (24 * 60 * 60)/* seconds */ /* * The following attributes are currently not supported by the NFSv4 server:
fs/nfsd/state.h +6 -0
@@ -343,6 +343,9 @@ struct nfs4_client {#define NFSD4_CLIENT_RECLAIM_COMPLETE (3)/* reclaim_complete done */ #define NFSD4_CLIENT_CONFIRMED (4)/* client is confirmed */ #define NFSD4_CLIENT_UPCALL_LOCK (5)/* upcall serialization */ +#define NFSD4_CLIENT_COURTESY (6) +#define NFSD4_CLIENT_EXPIRED (7) +#define NFSD4_CLIENT_RECONNECTED (8)#define NFSD4_CLIENT_CB_FLAG_MASK (1 << NFSD4_CLIENT_CB_UPDATE | \ 1 << NFSD4_CLIENT_CB_KILL)unsigned long cl_flags;
@@ -385,6 +388,9 @@ struct nfs4_client {struct list_head async_copies;/* list of async copies */ spinlock_t async_lock;/* lock for async copies */ atomic_t cl_cb_inflight;/* Outstanding callbacks */ + +spinlock_t cl_cs_lock; +struct list_head cl_cs_list; };/* struct nfs4_client_reset
Signature · 1 line
2.9.5
Dai Ngo
reviewerRFC PATCH v14 2/10 NFSD: Add lm_lock_expired call out
In reply to Dai NgoAdd callout function nfsd4_lm_lock_expired for lm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.
--- fs/nfsd/nfs4state.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
fs/nfsd/nfs4state.c +37 -0
@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl) } } +/** +* nfsd4_lm_lock_expired - check if lock conflict can be resolved. +* +* @fl: pointer to file_lock with a potential conflict +* Return values: +* %false: real conflict, lock conflict can not be resolved. +* %true: no conflict, lock conflict was resolved. +* +* Note that this function is called while the flc_lock is held. +*/ +static bool +nfsd4_lm_lock_expired (struct file_lock *fl) +{ +struct nfs4_lockowner *lo; +struct nfs4_client *clp; +bool rc = false; + +if (!fl) +return false; + lo = (struct nfs4_lockowner *)fl->fl_owner; + clp = lo->lo_owner.so_client; + +/* need to sync with courtesy client trying to reconnect */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) + rc = true; +else { +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); + rc = true; + } + } +spin_unlock (&clp->cl_cs_lock); +return rc; +} +static const struct lock_manager_operations nfsd_posix_mng_ops = { .lm_notify = nfsd4_lm_notify, .lm_get_owner = nfsd4_fl_get_owner, .lm_put_owner = nfsd4_fl_put_owner, + .lm_lock_expired = nfsd4_lm_lock_expired, }; static inline void
Signature · 1 line
2.9.5
Re: [PATCH RFC v14 02/10] NFSD: Add lm_lock_expired call out
Quoted 51 lines · Dai Ngo
Add callout function
nfsd4_lm_lock_expiredforlm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.Signed-off-by Dai Ngo <[email protected]>---
fs/nfsd/nfs4state.c| 37 +++++++++++++++++++++++++++++++++++++1 file changed, 37 insertions(+)fs/nfsd/nfs4state.c +30
-0@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl) } } +/** +* nfsd4_lm_lock_expired - check if lock conflict can be resolved. +* +* @fl: pointer to file_lock with a potential conflict +* Return values: +* %false: real conflict, lock conflict can not be resolved. +* %true: no conflict, lock conflict was resolved. +* +* Note that this function is called while the flc_lock is held. +*/ +static bool +nfsd4_lm_lock_expired (struct file_lock *fl) +{ +struct nfs4_lockowner *lo; +struct nfs4_client *clp; +bool rc = false; + +if (!fl) +return false; + lo = (struct nfs4_lockowner *)fl->fl_owner; + clp = lo->lo_owner.so_client; + +/* need to sync with courtesy client trying to reconnect */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) + rc = true; +else { +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); + rc = true;
fs/nfsd/nfs4state.c nfsd4_lm_notify(struct file_lock *fl) · line 6620
Quoted 10 lines
fs/nfsd/nfs4state.c +6
-0+ } + } +spin_unlock (&clp->cl_cs_lock); +return rc; +} +static const struct lock_manager_operations nfsd_posix_mng_ops = { .lm_notify = nfsd4_lm_notify, .lm_get_owner = nfsd4_fl_get_owner, .lm_put_owner = nfsd4_fl_put_owner,
I started to pull this series, but quickly realized it isn't based on the current contents of the NFSD for-next topic branch, which changes the names of nfsd4_fl_{put,get}_owner as we discussed during v13 review.
The upshot is that I would have to apply each of these patches by hand, which is tedious and error prone for more than one or two patches.
Please pull the latest from my for-next branch, rebase this series, and then post again. Thanks!
Quoted 7 lines
fs/nfsd/nfs4state.c +1
-0+ .lm_lock_expired = nfsd4_lm_lock_expired, };static inline void
Signature · 1 line
2.9.5
--
Chuck Lever
[email protected]
reviewerRe: [PATCH RFC v14 02/10] NFSD: Add lm_lock_expired call out
Quoted 65 lines · Chuck Lever III
Quoted 50 lines · Dai Ngo
Add callout function
nfsd4_lm_lock_expiredforlm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.Signed-off-by Dai Ngo <[email protected]>---
fs/nfsd/nfs4state.c| 37 +++++++++++++++++++++++++++++++++++++1 file changed, 37 insertions(+)fs/nfsd/nfs4state.c +30
-0@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl) } } +/** +* nfsd4_lm_lock_expired - check if lock conflict can be resolved. +* +* @fl: pointer to file_lock with a potential conflict +* Return values: +* %false: real conflict, lock conflict can not be resolved. +* %true: no conflict, lock conflict was resolved. +* +* Note that this function is called while the flc_lock is held. +*/ +static bool +nfsd4_lm_lock_expired (struct file_lock *fl) +{ +struct nfs4_lockowner *lo; +struct nfs4_client *clp; +bool rc = false; + +if (!fl) +return false; + lo = (struct nfs4_lockowner *)fl->fl_owner; + clp = lo->lo_owner.so_client; + +/* need to sync with courtesy client trying to reconnect */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) + rc = true; +else { +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); + rc = true;fs/nfsd/nfs4state.c nfsd4_lm_notify(struct file_lock *fl) · line 6620
Quoted 10 lines
fs/nfsd/nfs4state.c +6
-0+ } + } +spin_unlock (&clp->cl_cs_lock); +return rc; +} +static const struct lock_manager_operations nfsd_posix_mng_ops = { .lm_notify = nfsd4_lm_notify, .lm_get_owner = nfsd4_fl_get_owner, .lm_put_owner = nfsd4_fl_put_owner,I started to pull this series, but quickly realized it isn't based on the current contents of the NFSD for-next topic branch, which changes the names of nfsd4_fl_{put,get}_owner as we discussed during v13 review.
Quoted 6 lines
The upshot is that I would have to apply each of these patches by hand, which is tedious and error prone for more than one or two patches.
Please pull the latest from my for-next branch, rebase this series, and then post again. Thanks!'
will do in v15.
-Dai
Quoted 14 lines
Quoted 9 lines
fs/nfsd/nfs4state.c +1
-0+ .lm_lock_expired = nfsd4_lm_lock_expired, };static inline void
Signature · 1 line
2.9.5--
Chuck Lever
Dai Ngo
reviewerRFC PATCH v14 3/10 NFSD: Update nfsd_breaker_owns_lease() to handle courtesy clients
In reply to Dai NgoUpdate nfsd_breaker_owns_lease() to handle delegation conflict with courtesy clients. If conflict was caused courtesy client then discard the courtesy client by setting CLIENT_EXPIRED and return conflict resolved. Client with CLIENT_EXPIRED is expired by the laundromat.
--- fs/nfsd/nfs4state.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
fs/nfsd/nfs4state.c +18 -0
@@ -4727,6 +4727,24 @@ static bool nfsd_breaker_owns_lease(struct file_lock *fl)struct svc_rqst *rqst;struct nfs4_client *clp; + clp = dl->dl_stid.sc_client; +/* +* need to sync with courtesy client trying to reconnect using +* the cl_cs_lock, nn->client_lock can not be used since this +* function is called with the fl_lck held. +*/ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +return true; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); +spin_unlock (&clp->cl_cs_lock); +return true; + } +spin_unlock (&clp->cl_cs_lock); +if (!i_am_nfsd ())return false; rqst =kthread_data (current);
Signature · 1 line
2.9.5
Dai Ngo
reviewerRFC PATCH v14 4/10 NFSD: Update nfs4_get_vfs_file() to handle courtesy clients
In reply to Dai NgoUpdate nfs4_get_vfs_file() to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then set CLIENT_EXPIRED flag to expire the courtesy clients and allow nfs4_get_vfs_file to continue. Client with CLIENT_EXPIRED is expired by the laundromat.
--- fs/nfsd/nfs4state.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +99 -7
@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,return nfsd_setattr (rqstp, fh, &iattr,0 , (time64_t )0 ); } +static bool +nfs4_check_access_deny_bmap (struct nfs4_ol_stateid *stp,u32 access, +bool share_access) +{ +if (share_access) { +if (!stp->st_deny_bmap) +return false; + +if ((stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_BOTH )) || + (access &NFS4_SHARE_ACCESS_READ && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_READ )) || + (access &NFS4_SHARE_ACCESS_WRITE && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_WRITE ))) { +return true; + } +return false; + } +if ((access &NFS4_SHARE_DENY_BOTH ) || + (access &NFS4_SHARE_DENY_READ && + stp->st_access_bmap & (1 <<NFS4_SHARE_ACCESS_READ )) || + (access &NFS4_SHARE_DENY_WRITE && + stp->st_access_bmap & (1 <<NFS4_SHARE_ACCESS_WRITE ))) { +return true; + } +return false; +} + +/* +* Check whether nfserr_share_denied should be returned. +* +* access: is op_share_access if share_access is true. +* Check if access mode, op_share_access, would conflict with +* the current deny mode of the file 'fp'. +* access: is op_share_deny if share_access is false. +* Check if the deny mode, op_share_deny, would conflict with +* current access of the file 'fp'. +* stp: skip checking this entry. +* new_stp: normal open, not open upgrade. +* +* Function returns: +* true - access/deny mode conflict with normal client. +* false - no conflict or conflict with courtesy client(s) is resolved. +*/ +static bool +nfs4_conflict_clients (struct nfs4_file *fp,bool new_stp, +struct nfs4_ol_stateid *stp,u32 access,bool share_access) +{ +struct nfs4_ol_stateid *st; +struct nfs4_client *cl; +bool conflict = false; + +lockdep_assert_held (&fp->fi_lock); +list_for_each_entry (st, &fp->fi_stateids, st_perfile) { +if (st->st_openstp || (st == stp && new_stp) || + (!nfs4_check_access_deny_bmap (st, + access, share_access))) +continue ; + +/* need to sync with courtesy client trying to reconnect */ + cl = st->st_stid.sc_client; +spin_lock (&cl->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags)) { +spin_unlock (&cl->cl_cs_lock); +continue ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &cl->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags); +spin_unlock (&cl->cl_cs_lock); +continue ; + } +/* conflict not caused by courtesy client */ +spin_unlock (&cl->cl_cs_lock); + conflict = true; +break ; + } +return conflict; +} +static __be32 nfs4_get_vfs_file (struct svc_rqst *rqstp,struct nfs4_file *fp,struct svc_fh *cur_fh,struct nfs4_ol_stateid *stp, -struct nfsd4_open *open) +struct nfsd4_open *open,bool new_stp) {struct nfsd_file *nf =NULL ;__be32 status;
@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,*/ status =nfs4_file_check_deny (fp, open->op_share_deny);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock); - goto out; +if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out; + } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_deny, false)) { +spin_unlock (&fp->fi_lock); + goto out; + } }/* set access to the file */ status =nfs4_file_get_access (fp, open->op_share_access);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock); - goto out; +if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out; + } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_access, true)) { +spin_unlock (&fp->fi_lock); + goto out; + } }/* Set access bits in stateid */
@@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *cunsigned char old_deny_bmap = stp->st_deny_bmap;if (!test_access (open->op_share_access, stp)) -return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open); +return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open, false);/* test and set deny mode */ spin_lock (&fp->fi_lock);
@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf goto out; } }else { - status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open); + status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open, true);if (status) { stp->st_stid.sc_type =NFS4_CLOSED_STID ;release_open_stateid (stp);
Signature · 1 line
2.9.5
Re: [PATCH RFC v14 04/10] NFSD: Update nfs4_get_vfs_file() to handle courtesy clients
Quoted 12 lines · Dai Ngo
Update
nfs4_get_vfs_file()to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then setCLIENT_EXPIREDflag to expire the courtesy clients and allownfs4_get_vfs_fileto continue. Client withCLIENT_EXPIREDis expired by the laundromat.Signed-off-by Dai Ngo <[email protected]>
I'm still getting my head around this, but there are some items that can be addressed now, below.
Quoted 31 lines
---
fs/nfsd/nfs4state.c| 106 ++++++++++++++++++++++++++++++++++++++++++++++++----1 file changed, 99 insertions(+), 7 deletions(-)fs/nfsd/nfs4state.c +19
-0@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,return nfsd_setattr (rqstp, fh, &iattr,0 , (time64_t )0 ); } +static bool +nfs4_check_access_deny_bmap (struct nfs4_ol_stateid *stp,u32 access, +bool share_access) +{ +if (share_access) { +if (!stp->st_deny_bmap) +return false; + +if ((stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_BOTH )) || + (access &NFS4_SHARE_ACCESS_READ && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_READ )) || + (access &NFS4_SHARE_ACCESS_WRITE && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_WRITE ))) { +return true; + } +return false; + } + if ((access & NFS4_SHARE_DENY_BOTH) || + (access & NFS4_SHARE_DENY_READ &&
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 4996
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+ stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_READ)) || + (access & NFS4_SHARE_DENY_WRITE && + stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_WRITE))) { +return true; + } +return false; +} + +/* +* Check whether nfserr_share_denied should be returned.
This function is doing more than adjusting a return code, now that I'm reading it more carefully. How about "Check whether courtesy clients have conflicting access."
Quoted 7 lines
fs/nfsd/nfs4state.c +7
-0+* +* access: is op_share_access if share_access is true. +* Check if access mode, op_share_access, would conflict with +* the current deny mode of the file 'fp'. +* access: is op_share_deny if share_access is false. +* Check if the deny mode, op_share_deny, would conflict with +* current access of the file 'fp'.
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5013
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+* stp: skip checking this entry. +* new_stp: normal open, not open upgrade. +* +* Function returns: +* true - access/deny mode conflict with normal client. +* false - no conflict or conflict with courtesy client(s) is resolved. +*/ +static bool +nfs4_conflict_clients (struct nfs4_file *fp,bool new_stp, +struct nfs4_ol_stateid *stp,u32 access,bool share_access)
Functions that are called with locks held are usually suffixed with "_locked" -- this one should be too.
A better name might be "nfs4_resolve_deny_conflicts_locked".
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5016
Quoted 3 lines
fs/nfsd/nfs4state.c +3
-0+{ +struct nfs4_ol_stateid *st; +struct nfs4_client *cl;
Use "clp" to be consistent with other areas of the code.
Quoted 13 lines
fs/nfsd/nfs4state.c +13
-0+bool conflict = false; + +lockdep_assert_held (&fp->fi_lock); +list_for_each_entry (st, &fp->fi_stateids, st_perfile) { +if (st->st_openstp || (st == stp && new_stp) || + (!nfs4_check_access_deny_bmap (st, + access, share_access))) +continue ; + +/* need to sync with courtesy client trying to reconnect */ + cl = st->st_stid.sc_client; +spin_lock (&cl->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags)) {
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5039
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+spin_unlock (&cl->cl_cs_lock); +continue ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &cl->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags); +spin_unlock (&cl->cl_cs_lock); +continue ; + } +/* conflict not caused by courtesy client */ +spin_unlock (&cl->cl_cs_lock);
I think I'm seeing similar code as this in some of the other patches. Whereever you can, please deduplicate by creating a helper function and moving the common code there.
Quoted 19 lines
fs/nfsd/nfs4state.c +7
-3+ conflict = true; +break ; + } +return conflict; +} +static __be32 nfs4_get_vfs_file (struct svc_rqst *rqstp,struct nfs4_file *fp,struct svc_fh *cur_fh,struct nfs4_ol_stateid *stp, -struct nfsd4_open *open) +struct nfsd4_open *open,bool new_stp) {struct nfsd_file *nf =NULL ;__be32 status;@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,*/ status =nfs4_file_check_deny (fp, open->op_share_deny);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock); - goto out;
fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5073
Quoted 10 lines
fs/nfsd/nfs4state.c +9
-0+if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out; + } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_deny, false)) { +spin_unlock (&fp->fi_lock); + goto out; + } }
Doesn't nfs4_upgrade_open() need to perform the same check?
Quoted 5 lines
fs/nfsd/nfs4state.c +0
-1/* set access to the file */ status =nfs4_file_get_access (fp, open->op_share_access);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock);
fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5086
Quoted 10 lines
fs/nfsd/nfs4state.c +9
-1- goto out; +if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out; + } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_access, true)) { +spin_unlock (&fp->fi_lock); + goto out; + }
This is nfs4_file_get_access()'s only caller. Should the call to nfs4_conflict_clients() be moved into nfs4_file_get_access() ?
Quoted 24 lines
fs/nfsd/nfs4state.c +2
-2}/* Set access bits in stateid */ @@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *cunsigned char old_deny_bmap = stp->st_deny_bmap;if (!test_access (open->op_share_access, stp)) -return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open); +return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open, false);/* test and set deny mode */ spin_lock (&fp->fi_lock);@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf goto out; } }else { - status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open); + status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open, true);if (status) { stp->st_stid.sc_type =NFS4_CLOSED_STID ;release_open_stateid (stp);Signature · 1 line
2.9.5
--
Chuck Lever
[email protected]
reviewerRe: [PATCH RFC v14 04/10] NFSD: Update nfs4_get_vfs_file() to handle courtesy clients
Quoted 51 lines · Chuck Lever III
Quoted 11 lines · Dai Ngo
Update
nfs4_get_vfs_file()to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then setCLIENT_EXPIREDflag to expire the courtesy clients and allownfs4_get_vfs_fileto continue. Client withCLIENT_EXPIREDis expired by the laundromat.Signed-off-by Dai Ngo <[email protected]>I'm still getting my head around this, but there are some items that can be addressed now, below.
Quoted 36 lines
---
fs/nfsd/nfs4state.c| 106 ++++++++++++++++++++++++++++++++++++++++++++++++----1 file changed, 99 insertions(+), 7 deletions(-)fs/nfsd/nfs4state.c +24
-0@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,return nfsd_setattr (rqstp, fh, &iattr,0 , (time64_t )0 ); } +static bool +nfs4_check_access_deny_bmap (struct nfs4_ol_stateid *stp,u32 access, +bool share_access) +{ +if (share_access) { +if (!stp->st_deny_bmap) +return false; + +if ((stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_BOTH )) || + (access &NFS4_SHARE_ACCESS_READ && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_READ )) || + (access &NFS4_SHARE_ACCESS_WRITE && + stp->st_deny_bmap & (1 <<NFS4_SHARE_DENY_WRITE ))) { +return true; + } +return false; + } +if ((access &NFS4_SHARE_DENY_BOTH ) || + (access &NFS4_SHARE_DENY_READ && + stp->st_access_bmap & (1 <<NFS4_SHARE_ACCESS_READ )) || + (access &NFS4_SHARE_DENY_WRITE && + stp->st_access_bmap & (1 <<NFS4_SHARE_ACCESS_WRITE ))) { +return true; + }
Quoted 8 lines
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 4996
Quoted 5 lines
fs/nfsd/nfs4state.c +5
-0+return false; +} + +/* +* Check whether nfserr_share_denied should be returned. This function is doing more than adjusting a return code, now that I'm reading it more carefully. How about "Check whether courtesy clients have conflicting access."
fix in v15.
Quoted 22 lines
Quoted 9 lines
fs/nfsd/nfs4state.c +7
-0+* +* access: is op_share_access if share_access is true. +* Check if access mode, op_share_access, would conflict with +* the current deny mode of the file 'fp'. +* access: is op_share_deny if share_access is false. +* Check if the deny mode, op_share_deny, would conflict with +* current access of the file 'fp'. fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5013
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+* stp: skip checking this entry. +* new_stp: normal open, not open upgrade. +* +* Function returns: +* true - access/deny mode conflict with normal client. +* false - no conflict or conflict with courtesy client(s) is resolved. +*/ +static bool +nfs4_conflict_clients (struct nfs4_file *fp,bool new_stp, +struct nfs4_ol_stateid *stp,u32 access,bool share_access)Functions that are called with locks held are usually suffixed with "
_locked" -- this one should be too.
Quoted 1 line
A better name might be "
nfs4_resolve_deny_conflicts_locked".
Fix in v15.
Quoted 6 lines
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5016
Quoted 5 lines
fs/nfsd/nfs4state.c +3
-0+{ +struct nfs4_ol_stateid *st; +struct nfs4_client *cl;Use "clp" to be consistent with other areas of the code.
Fix in v15.
Quoted 21 lines
Quoted 21 lines
fs/nfsd/nfs4state.c +19
-0+bool conflict = false; + +lockdep_assert_held (&fp->fi_lock); +list_for_each_entry (st, &fp->fi_stateids, st_perfile) { +if (st->st_openstp || (st == stp && new_stp) || + (!nfs4_check_access_deny_bmap (st, + access, share_access))) +continue ; + +/* need to sync with courtesy client trying to reconnect */ + cl = st->st_stid.sc_client; +spin_lock (&cl->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags)) { +spin_unlock (&cl->cl_cs_lock); +continue ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &cl->cl_flags)) { +set_bit (NFSD4_CLIENT_EXPIRED , &cl->cl_flags); +spin_unlock (&cl->cl_cs_lock);
Quoted 8 lines
fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5039
Quoted 4 lines
fs/nfsd/nfs4state.c +4
-0+continue ; + } +/* conflict not caused by courtesy client */ +spin_unlock (&cl->cl_cs_lock);I think I'm seeing similar code as this in some of the other patches. Whereever you can, please deduplicate by creating a helper function and moving the common code there.
Fix in v15.
Quoted 24 lines
Quoted 24 lines
fs/nfsd/nfs4state.c +10
-3+ conflict = true; +break ; + } +return conflict; +} +static __be32 nfs4_get_vfs_file (struct svc_rqst *rqstp,struct nfs4_file *fp,struct svc_fh *cur_fh,struct nfs4_ol_stateid *stp, -struct nfsd4_open *open) +struct nfsd4_open *open,bool new_stp) {struct nfsd_file *nf =NULL ;__be32 status;@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,*/ status =nfs4_file_check_deny (fp, open->op_share_deny);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock); - goto out; +if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out;
Quoted 8 lines
fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5073
Quoted 7 lines
fs/nfsd/nfs4state.c +6
-0+ } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_deny, false)) { +spin_unlock (&fp->fi_lock); + goto out; + } }Doesn't
nfs4_upgrade_open()need to perform the same check?
Yes, we need to do the same check here. Fix in v15.
Quoted 18 lines
Quoted 6 lines
fs/nfsd/nfs4state.c +0
-1/* set access to the file */ status =nfs4_file_get_access (fp, open->op_share_access);if (status != nfs_ok) { -spin_unlock (&fp->fi_lock);fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5086
Quoted 10 lines
fs/nfsd/nfs4state.c +9
-1- goto out; +if (status != nfserr_share_denied) { +spin_unlock (&fp->fi_lock); + goto out; + } +if (nfs4_conflict_clients (fp, new_stp, stp, + open->op_share_access, true)) { +spin_unlock (&fp->fi_lock); + goto out; + }This is nfs4_file_get_access()'s only caller. Should the call to
nfs4_conflict_clients()be moved intonfs4_file_get_access()?
We could, but I think it's better to keep the call sign of nfs4_file_get_access as is for speed since we don't have to pass additional parameters for nfs4_resolve_deny_conflicts_locked which should rarely need to run. Also, I think it's easier to understand to code when both access and deny check are in the same place.
-Dai
Quoted 31 lines
Quoted 26 lines
fs/nfsd/nfs4state.c +2
-2}/* Set access bits in stateid */ @@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *cunsigned char old_deny_bmap = stp->st_deny_bmap;if (!test_access (open->op_share_access, stp)) -return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open); +return nfs4_get_vfs_file (rqstp, fp, cur_fh, stp, open, false);/* test and set deny mode */ spin_lock (&fp->fi_lock);@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf goto out; } }else { - status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open); + status =nfs4_get_vfs_file (rqstp, fp, current_fh, stp, open, true);if (status) { stp->st_stid.sc_type =NFS4_CLOSED_STID ;release_open_stateid (stp);Signature · 1 line
2.9.5--
Chuck Lever
Dai Ngo
reviewerRFC PATCH v14 5/10 NFSD: Update find_clp_in_name_tree() to handle courtesy clients
In reply to Dai NgoUpdate find_clp_in_name_tree:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.
Update find_confirmed_client_by_name to discard the courtesy client; set CLIENT_EXPIRED.
Update nfsd4_setclientid to expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the the conf_id_hashtbl list.
--- fs/nfsd/nfs4state.c | 45 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +42 -3
@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) node = node->rb_left; else if (cmp < 0) node = node->rb_right; - else + else { +clear_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); +/* sync with thread resolving lock/deleg conflict */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +return NULL ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +clear_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +set_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); + } +spin_unlock (&clp->cl_cs_lock);return clp; + } }return NULL ; }
@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)return NULL ; } +static void +nfsd4_discard_courtesy_clnt (struct nfs4_client *clp) +{ +spin_lock (&clp->cl_cs_lock); +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); +spin_unlock (&clp->cl_cs_lock); +} +static struct nfs4_client *find_confirmed_client (clientid_t *clid,bool sessions,struct nfsd_net *nn) {
@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)static struct nfs4_client *find_confirmed_client_by_name (struct xdr_netobj *name,struct nfsd_net *nn) { +struct nfs4_client *clp; +lockdep_assert_held (&nn->client_lock); -return find_clp_in_name_tree (name, &nn->conf_name_tree); + clp =find_clp_in_name_tree (name, &nn->conf_name_tree); +if (clp &&test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +nfsd4_discard_courtesy_clnt (clp); + clp =NULL ; + } +return clp; } static struct nfs4_client *
@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,struct nfs4_client *unconf =NULL ;__be32 status;struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_client *cclient =NULL ; new =create_client (clname, rqstp, &clverifier);if (new ==NULL )return nfserr_jukebox;spin_lock (&nn->client_lock); - conf =find_confirmed_client_by_name (&clname, nn); +/* find confirmed client by name */ + conf =find_clp_in_name_tree (&clname, &nn->conf_name_tree); +if (conf &&test_bit (NFSD4_CLIENT_RECONNECTED , &conf->cl_flags)) { + cclient = conf; + conf =NULL ; + } +if (conf &&client_has_state (conf)) { status = nfserr_clid_inuse;if (clp_used_exchangeid (conf))
@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new =NULL ; status = nfs_ok; out: +if (cclient) +unhash_client_locked (cclient);spin_unlock (&nn->client_lock); +if (cclient) +expire_client (cclient);if (new)free_client (new);if (unconf) {
Signature · 1 line
2.9.5
Re: [PATCH RFC v14 05/10] NFSD: Update find_clp_in_name_tree() to handle courtesy clients
Good morning, Dai!
Quoted 30 lines · Dai Ngo
Update
find_clp_in_name_tree:
. skip client withCLIENT_EXPIREDflag; discarded courtesy client. . if courtesy client was found then clearCLIENT_COURTESYand
setCLIENT_RECONNECTEDso callers can take appropriate action.Update
find_confirmed_client_by_nameto discard the courtesy client; setCLIENT_EXPIRED.Update
nfsd4_setclientidto expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the theconf_id_hashtbllist.Signed-off-by Dai Ngo <[email protected]>---
fs/nfsd/nfs4state.c| 45 ++++++++++++++++++++++++++++++++++++++++++---1 file changed, 42 insertions(+), 3 deletions(-)fs/nfsd/nfs4state.c +2
-1@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) node = node->rb_left; else if (cmp < 0) node = node->rb_right; - else + else { +clear_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags);
fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2848
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+/* sync with thread resolving lock/deleg conflict */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +return NULL ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +clear_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +set_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); + }
This should be written
if (test_and_clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2851
Quoted 3 lines
fs/nfsd/nfs4state.c +2
-0+spin_unlock (&clp->cl_cs_lock);return clp; + }
And since this is the same logic that is used in 6/10 and 7/10, please refactor this into a separate function and call it from the find_yada_ functions as we discussed previously. Thanks!
Quoted 65 lines
fs/nfsd/nfs4state.c +8
-0}return NULL ; }@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)return NULL ; } +static void +nfsd4_discard_courtesy_clnt (struct nfs4_client *clp) +{ +spin_lock (&clp->cl_cs_lock); +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); +spin_unlock (&clp->cl_cs_lock); +} +static struct
nfs4_client*find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn){fs/nfsd/nfs4state.c +0
-0@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)static struct
nfs4_client*find_confirmed_client_by_name(struct xdr_netobj *name, struct nfsd_net *nn){fs/nfsd/nfs4state.c +2
-0+struct nfs4_client *clp; +lockdep_assert_held(&nn->client_lock);fs/nfsd/nfs4state.c +6
-1-return find_clp_in_name_tree (name, &nn->conf_name_tree); + clp =find_clp_in_name_tree (name, &nn->conf_name_tree); +if (clp &&test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +nfsd4_discard_courtesy_clnt (clp); + clp =NULL ; + } +return clp;}
static struct
nfs4_client*fs/nfsd/nfs4state.c +8
-1@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,struct nfs4_client *unconf =NULL ;__be32 status;struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_client *cclient =NULL ; new =create_client (clname, rqstp, &clverifier);if (new ==NULL )return nfserr_jukebox;spin_lock (&nn->client_lock); - conf =find_confirmed_client_by_name (&clname, nn); +/* find confirmed client by name */ + conf =find_clp_in_name_tree (&clname, &nn->conf_name_tree); +if (conf &&test_bit (NFSD4_CLIENT_RECONNECTED , &conf->cl_flags)) { + cclient = conf; + conf =NULL ; + } +if (conf &&client_has_state (conf)) { status = nfserr_clid_inuse;if (clp_used_exchangeid (conf))@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new =NULL ; status = nfs_ok;out:
fs/nfsd/nfs4state.c +2
-0+if (cclient) +unhash_client_locked (cclient);spin_unlock(&nn->client_lock);fs/nfsd/nfs4state.c +2
-0+if (cclient) +expire_client (cclient);
Quoted 6 lines
if (new) free_client(new); if (unconf) {Signature · 1 line
2.9.5
--
Chuck Lever
[email protected]
reviewerRe: [PATCH RFC v14 05/10] NFSD: Update find_clp_in_name_tree() to handle courtesy clients
Quoted 45 lines · Chuck Lever III
Good morning, Dai!
Quoted 30 lines · Dai Ngo
Update
find_clp_in_name_tree:
. skip client withCLIENT_EXPIREDflag; discarded courtesy client. . if courtesy client was found then clearCLIENT_COURTESYand
setCLIENT_RECONNECTEDso callers can take appropriate action.Update
find_confirmed_client_by_nameto discard the courtesy client; setCLIENT_EXPIRED.Update
nfsd4_setclientidto expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the theconf_id_hashtbllist.Signed-off-by Dai Ngo <[email protected]>---
fs/nfsd/nfs4state.c| 45 ++++++++++++++++++++++++++++++++++++++++++---1 file changed, 42 insertions(+), 3 deletions(-)fs/nfsd/nfs4state.c +2
-1@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) node = node->rb_left; else if (cmp < 0) node = node->rb_right; - else + else { +clear_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags);fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2848
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+/* sync with thread resolving lock/deleg conflict */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +return NULL ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +clear_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +set_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); + }This should be written
Quoted 2 lines
if (test_and_clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
Fix in v15.
Quoted 8 lines
fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2851
Quoted 5 lines
fs/nfsd/nfs4state.c +2
-0+spin_unlock (&clp->cl_cs_lock);return clp; + }And since this is the same logic that is used in 6/10 and 7/10, please refactor this into a separate function and call it from the
find_yada_functions as we discussed previously. Thanks!
Fix in v15.
-Dai
Quoted 78 lines
Quoted 67 lines
fs/nfsd/nfs4state.c +8
-0}return NULL ; }@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)return NULL ; } +static void +nfsd4_discard_courtesy_clnt (struct nfs4_client *clp) +{ +spin_lock (&clp->cl_cs_lock); +set_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags); +spin_unlock (&clp->cl_cs_lock); +} +static struct
nfs4_client*find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn){fs/nfsd/nfs4state.c +0
-0@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)static struct
nfs4_client*find_confirmed_client_by_name(struct xdr_netobj *name, struct nfsd_net *nn){fs/nfsd/nfs4state.c +2
-0+struct nfs4_client *clp; +lockdep_assert_held(&nn->client_lock);fs/nfsd/nfs4state.c +6
-1-return find_clp_in_name_tree (name, &nn->conf_name_tree); + clp =find_clp_in_name_tree (name, &nn->conf_name_tree); +if (clp &&test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +nfsd4_discard_courtesy_clnt (clp); + clp =NULL ; + } +return clp;}
static struct
nfs4_client*fs/nfsd/nfs4state.c +8
-1@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,struct nfs4_client *unconf =NULL ;__be32 status;struct nfsd_net *nn =net_generic (SVC_NET (rqstp), nfsd_net_id); +struct nfs4_client *cclient =NULL ; new =create_client (clname, rqstp, &clverifier);if (new ==NULL )return nfserr_jukebox;spin_lock (&nn->client_lock); - conf =find_confirmed_client_by_name (&clname, nn); +/* find confirmed client by name */ + conf =find_clp_in_name_tree (&clname, &nn->conf_name_tree); +if (conf &&test_bit (NFSD4_CLIENT_RECONNECTED , &conf->cl_flags)) { + cclient = conf; + conf =NULL ; + } +if (conf &&client_has_state (conf)) { status = nfserr_clid_inuse;if (clp_used_exchangeid (conf))@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, new =NULL ; status = nfs_ok;out:
fs/nfsd/nfs4state.c +2
-0+if (cclient) +unhash_client_locked (cclient);spin_unlock(&nn->client_lock);fs/nfsd/nfs4state.c +2
-0+if (cclient) +expire_client (cclient);Quoted 6 lines
if (new) free_client(new); if (unconf) {Signature · 1 line
2.9.5--
Chuck Lever
Dai Ngo
reviewerRFC PATCH v14 6/10 NFSD: Update find_in_sessionid_hashtbl() to handle courtesy clients
In reply to Dai NgoUpdate find_in_sessionid_hashtbl to:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.
Update nfsd4_sequence and nfsd4_bind_conn_to_session to create client record for client with CLIENT_RECONNECTED set.
Update nfsd4_destroy_session to discard client with CLIENT_RECONNECTED set.
--- fs/nfsd/nfs4state.c | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-)
fs/nfsd/nfs4state.c +43 -1
@@ -1935,10 +1935,31 @@ find_in_sessionid_hashtbl(struct nfs4_sessionid *sessionid, struct net *net, {struct nfsd4_session *session;__be32 status = nfserr_badsession; +struct nfs4_client *clp; session =__find_in_sessionid_hashtbl (sessionid, net);if (!session) goto out; + clp = session->se_client; +if (clp) { +clear_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); +/* need to sync with thread resolving lock/deleg conflict */ +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); + session =NULL ; + goto out; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +/* +* clear CLIENT_COURTESY flag to prevent it from being +* destroyed by thread trying to resolve conflicts. +*/ +clear_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +set_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); + } +spin_unlock (&clp->cl_cs_lock); + } status =nfsd4_get_session_locked (session);if (status) session =NULL ;
@@ -3661,6 +3682,7 @@ __be32 nfsd4_bind_conn_to_session(struct svc_rqst *rqstp,struct nfsd4_session *session;struct net *net =SVC_NET (rqstp);struct nfsd_net *nn =net_generic (net, nfsd_net_id); +struct nfs4_client *clp;if (!nfsd4_last_compound_op (rqstp))return nfserr_not_only_op;
@@ -3693,6 +3715,13 @@ __be32 nfsd4_bind_conn_to_session(struct svc_rqst *rqstp,nfsd4_init_conn (rqstp, conn, session); status = nfs_ok; out: + clp = session->se_client; +if (test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +if (status == nfs_ok) +nfsd4_client_record_create (clp); +else +nfsd4_discard_courtesy_clnt (clp); + }nfsd4_put_session (session); out_no_session:return status;
@@ -3715,6 +3744,7 @@ nfsd4_destroy_session(struct svc_rqst *r, struct nfsd4_compound_state *cstate,int ref_held_by_me =0 ;struct net *net =SVC_NET (r);struct nfsd_net *nn =net_generic (net, nfsd_net_id); +struct nfs4_client *clp; status = nfserr_not_only_op;if (nfsd4_compound_in_session (cstate, sessionid)) {
@@ -3727,6 +3757,12 @@ nfsd4_destroy_session(struct svc_rqst *r, struct nfsd4_compound_state *cstate, ses =find_in_sessionid_hashtbl (sessionid, net, &status);if (!ses) goto out_client_lock; + clp = ses->se_client; +if (test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { + status = nfserr_badsession; +nfsd4_discard_courtesy_clnt (clp); + goto out_put_session; + } status = nfserr_wrong_cred;if (!nfsd4_mach_creds_match (ses->se_client, r)) goto out_put_session;
@@ -3831,7 +3867,7 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,struct nfsd4_compoundres *resp = rqstp->rq_resp;struct xdr_stream *xdr = resp->xdr;struct nfsd4_session *session; -struct nfs4_client *clp; +struct nfs4_client *clp =NULL ;struct nfsd4_slot *slot;struct nfsd4_conn *conn;__be32 status;
@@ -3941,6 +3977,12 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,if (conn)free_conn (conn);spin_unlock (&nn->client_lock); +if (clp &&test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +if (status == nfs_ok) +nfsd4_client_record_create (clp); +else +nfsd4_discard_courtesy_clnt (clp); + }return status; out_put_session:nfsd4_put_session_locked (session);
Signature · 1 line
2.9.5
Dai Ngo
reviewerRFC PATCH v14 7/10 NFSD: Update find_client_in_id_table() to handle courtesy clients
In reply to Dai NgoUpdate find_client_in_id_table to:
. skip client with CLIENT_EXPIRED; discarded courtesy client . if courtesy client was found then clear CLIENT_COURTESY and set
CLIENT_RECONNECTED flag so callers can take appropriate action.
Update find_confirmed_client to discard courtesy client. Update lookup_clientid to call find_client_in_id_table directly. Update set_client to create client record for courtesy client. Update find_cpntf_state to discard courtesy client.
--- fs/nfsd/nfs4state.c | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-)
fs/nfsd/nfs4state.c +32 -2
@@ -2915,6 +2915,19 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)if (same_clid (&clp->cl_clientid, clid)) {if ((bool )clp->cl_minorversion != sessions)return NULL ; + +/* need to sync with thread resolving lock/deleg conflict */ +clear_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); +spin_lock (&clp->cl_cs_lock); +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +continue ; + } +if (test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +clear_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +set_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags); + } +spin_unlock (&clp->cl_cs_lock);renew_client_locked (clp);return clp; }
@@ -2922,6 +2935,7 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)return NULL ; } +static void nfsd4_discard_courtesy_clnt (struct nfs4_client *clp) {
@@ -2934,9 +2948,15 @@ static struct nfs4_client * find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn) {struct list_head *tbl = nn->conf_id_hashtbl; +struct nfs4_client *clp;lockdep_assert_held (&nn->client_lock); -return find_client_in_id_table (tbl, clid, sessions); + clp =find_client_in_id_table (tbl, clid, sessions); +if (clp &&test_bit (NFSD4_CLIENT_RECONNECTED , &clp->cl_flags)) { +nfsd4_discard_courtesy_clnt (clp); + clp =NULL ; + } +return clp; } static struct nfs4_client *
@@ -4867,9 +4887,10 @@ static struct nfs4_client *lookup_clientid(clientid_t *clid, bool sessions, struct nfsd_net *nn) {struct nfs4_client *found; +struct list_head *tbl = nn->conf_id_hashtbl;spin_lock (&nn->client_lock); - found =find_confirmed_client (clid, sessions, nn); + found =find_client_in_id_table (tbl, clid, sessions);if (found)atomic_inc (&found->cl_rpc_users);spin_unlock (&nn->client_lock);
@@ -4894,6 +4915,8 @@ static __be32 set_client(clientid_t *clid, cstate->clp =lookup_clientid (clid, false, nn);if (!cstate->clp)return nfserr_expired; +if (test_bit (NFSD4_CLIENT_RECONNECTED , &cstate->clp->cl_flags)) +nfsd4_client_record_create (cstate->clp);return nfs_ok; }
@@ -6222,6 +6245,13 @@ static __be32 find_cpntf_state(struct nfsd_net *nn, stateid_t *st, found =lookup_clientid (&cps->cp_p_clid, true, nn);if (!found) goto out; +if (test_bit (NFSD4_CLIENT_RECONNECTED , &found->cl_flags)) { +nfsd4_discard_courtesy_clnt (found); +if (atomic_dec_and_lock (&found->cl_rpc_users, + &nn->client_lock)) +spin_unlock (&nn->client_lock); + goto out; + } *stid =find_stateid_by_type (found, &cps->cp_p_stateid,NFS4_DELEG_STID |NFS4_OPEN_STID |NFS4_LOCK_STID );
Signature · 1 line
2.9.5
Dai Ngo
reviewerRFC PATCH v14 8/10 NFSD: Refactor nfsd4_laundromat()
In reply to Dai NgoExtract a bit of logic that is about to be expanded to handle courtesy clients.
--- fs/nfsd/nfs4state.c | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-)
fs/nfsd/nfs4state.c +21 -12
@@ -5821,6 +5821,26 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) }#endif +static void +nfs4_get_client_reaplist (struct nfsd_net *nn,struct list_head *reaplist, +struct laundry_time *lt) +{ +struct list_head *pos, *next; +struct nfs4_client *clp; + +INIT_LIST_HEAD (reaplist); +spin_lock (&nn->client_lock); +list_for_each_safe (pos, next, &nn->client_lru) { + clp = list_entry(pos, struct nfs4_client, cl_lru); +if (!state_expired (lt, clp->cl_time)) +break ; +if (mark_client_expired_locked (clp)) +continue ; +list_add (&clp->cl_lru, reaplist); + } +spin_unlock (&nn->client_lock); +} +static time64_t nfs4_laundromat (struct nfsd_net *nn) {
@@ -5843,7 +5863,6 @@ nfs4_laundromat(struct nfsd_net *nn) goto out; }nfsd4_end_grace (nn); -INIT_LIST_HEAD (&reaplist);spin_lock (&nn->s2s_cp_lock);idr_for_each_entry (&nn->s2s_cp_stateids, cps_t, i) {
@@ -5853,17 +5872,7 @@ nfs4_laundromat(struct nfsd_net *nn)_free_cpntf_state_locked (nn, cps); }spin_unlock (&nn->s2s_cp_lock); - -spin_lock (&nn->client_lock); -list_for_each_safe (pos, next, &nn->client_lru) { - clp = list_entry(pos, struct nfs4_client, cl_lru); -if (!state_expired (<, clp->cl_time)) -break ; -if (mark_client_expired_locked (clp)) -continue ; -list_add (&clp->cl_lru, &reaplist); - } -spin_unlock (&nn->client_lock); +nfs4_get_client_reaplist (nn, &reaplist, <);list_for_each_safe (pos, next, &reaplist) { clp = list_entry(pos, struct nfs4_client, cl_lru);trace_nfsd_clid_purged (&clp->cl_clientid);
Signature · 1 line
2.9.5
Dai Ngo
reviewerRFC PATCH v14 9/10 NFSD: Update laundromat to handle courtesy clients
In reply to Dai NgoAdd nfs4_anylock_blocker to check if an expired client has any blockers.
Update nfs4_get_client_reaplist to:
. add discarded courtesy client; client marked with CLIENT_EXPIRED,
to reaplist.
. detect if expired client still has state and no blockers then
transit it to courtesy client by setting CLIENT_COURTESY flag and removing the client record.
--- fs/nfsd/nfs4state.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 2 deletions(-)
fs/nfsd/nfs4state.c +98 -2
@@ -5821,24 +5821,120 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) }#endif +static bool +nfs4_anylock_blocker (struct nfs4_client *clp) +{ +int i; +struct nfs4_stateowner *so, *tmp; +struct nfs4_lockowner *lo; +struct nfs4_ol_stateid *stp; +struct nfs4_file *nf; +struct inode *ino; +struct file_lock_context *ctx; +struct file_lock *fl; + +spin_lock (&clp->cl_lock); +for (i =0 ; i <OWNER_HASH_SIZE ; i++) { +/* scan each lock owner */ +list_for_each_entry_safe (so, tmp, &clp->cl_ownerstr_hashtbl[i], + so_strhash) { +if (so->so_is_open_owner) +continue ; + +/* scan lock states of this lock owner */ + lo =lockowner (so); +list_for_each_entry (stp, &lo->lo_owner.so_stateids, + st_perstateowner) { + nf = stp->st_stid.sc_file; + ino = nf->fi_inode; + ctx = ino->i_flctx; +if (!ctx) +continue ; +/* check each lock belongs to this lock state */ +list_for_each_entry (fl, &ctx->flc_posix, fl_list) { +if (fl->fl_owner != lo) +continue ; +if (!list_empty (&fl->fl_blocked_requests)) { +spin_unlock (&clp->cl_lock); +return true; + } + } + } + } + } +spin_unlock (&clp->cl_lock); +return false; +} +static void nfs4_get_client_reaplist (struct nfsd_net *nn,struct list_head *reaplist,struct laundry_time *lt) {struct list_head *pos, *next;struct nfs4_client *clp; +bool cour; +struct list_head cslist;INIT_LIST_HEAD (reaplist); +INIT_LIST_HEAD (&cslist);spin_lock (&nn->client_lock);list_for_each_safe (pos, next, &nn->client_lru) { clp = list_entry(pos, struct nfs4_client, cl_lru);if (!state_expired (lt, clp->cl_time))break ; -if (mark_client_expired_locked (clp)) + +/* client expired */ +if (!client_has_state (clp)) { +if (mark_client_expired_locked (clp)) +continue ; +list_add (&clp->cl_lru, reaplist); +continue ; + } + +/* expired client has state */ +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) + goto exp_client; + cour =test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +if (cour &&ktime_get_boottime_seconds () >= + (clp->cl_time +NFSD_COURTESY_CLIENT_TIMEOUT )) + goto exp_client; +if (nfs4_anylock_blocker (clp)) { +/* expired client has state and has blocker. */ +exp_client: +if (mark_client_expired_locked (clp)) +continue ; +list_add (&clp->cl_lru, reaplist);continue ; -list_add (&clp->cl_lru, reaplist); + } +/* +* Client expired and has state and has no blockers. +* If there is race condition with blockers, next time +* the laundromat runs it will catch it and expires +* the client. +*/ +if (!cour) { +set_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +list_add (&clp->cl_cs_list, &cslist); + } }spin_unlock (&nn->client_lock); + +while (!list_empty (&cslist)) { + clp = list_first_entry(&cslist, struct nfs4_client, cl_cs_list); +list_del_init (&clp->cl_cs_list); +spin_lock (&clp->cl_cs_lock); +/* +* Client might have re-connected. Make sure it's +* still in courtesy state before removing its record. +*/ +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags) || + !test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +continue ; + } +spin_unlock (&clp->cl_cs_lock); +nfsd4_client_record_remove (clp); + } } static time64_t
Signature · 1 line
2.9.5
Re: [PATCH RFC v14 09/10] NFSD: Update laundromat to handle courtesy clients
Quoted 7 lines · Dai Ngo
Add
nfs4_anylock_blockerto check if an expired client has any blockers.
Quoted 6 lines
Update
nfs4_get_client_reaplistto:
. add discarded courtesy client; client marked withCLIENT_EXPIRED,
to reaplist.
. detect if expired client still has state and no blockers then
transit it to courtesy client by settingCLIENT_COURTESYflag and removing the client record.
Thanks for breaking up the v13 patch and adding these nice patch descriptions !
Quoted 20 lines
Signed-off-by Dai Ngo <[email protected]>---
fs/nfsd/nfs4state.c| 100 ++++++++++++++++++++++++++++++++++++++++++++++++++--1 file changed, 98 insertions(+), 2 deletions(-)fs/nfsd/nfs4state.c +7
-0@@ -5821,24 +5821,120 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) }#endif +static bool +nfs4_anylock_blocker (struct nfs4_client *clp) +{ +int i; +struct nfs4_stateowner *so, *tmp; +struct nfs4_lockowner *lo; +struct nfs4_ol_stateid *stp;
fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5840
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+struct nfs4_file *nf; +struct inode *ino; +struct file_lock_context *ctx; +struct file_lock *fl; + +spin_lock (&clp->cl_lock); +for (i =0 ; i <OWNER_HASH_SIZE ; i++) { +/* scan each lock owner */ +list_for_each_entry_safe (so, tmp, &clp->cl_ownerstr_hashtbl[i], + so_strhash) {
You're holding the cl_lock and not changing the hash bucket list inside the loop, so list_for_each_entry() is sufficient here. _safe is not needed.
Quoted 11 lines
fs/nfsd/nfs4state.c +11
-0+if (so->so_is_open_owner) +continue ; + +/* scan lock states of this lock owner */ + lo =lockowner (so); +list_for_each_entry (stp, &lo->lo_owner.so_stateids, + st_perstateowner) { + nf = stp->st_stid.sc_file; + ino = nf->fi_inode; + ctx = ino->i_flctx; +if (!ctx)
fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5861
Quoted 10 lines
fs/nfsd/nfs4state.c +10
-0+continue ; +/* check each lock belongs to this lock state */ +list_for_each_entry (fl, &ctx->flc_posix, fl_list) { +if (fl->fl_owner != lo) +continue ; +if (!list_empty (&fl->fl_blocked_requests)) { +spin_unlock (&clp->cl_lock); +return true; + } + }
IIUC fl_blocked_requests is internal to fs/locks.c, so this little piece needs to be moved there. Jeff can give a little more guidance on function naming, etc.
Have a look at Section 6 of Documentation/process/coding-style.rst for some hints about what to do with long functions. This one is long and indented far to the right: that is indication that it should be broken into smaller helpers with names that make it clear what each step does -- no comments needed.
So for legibility, "check each lock belongs..." is a function that should be moved to fs/locks.c, as I mentioned above; and "scan lock state" can be its own function that calls "check each lock belongs".
I'm not sure what the comment "check each lock belongs to this lock state" means. I thought we were looking at blocked lists, not locks, in this search?
Quoted 23 lines
fs/nfsd/nfs4state.c +7
-0+ } + } + } +spin_unlock (&clp->cl_lock); +return false; +} +static void
nfs4_get_client_reaplist(structnfsd_net*nn, structlist_head*reaplist,
structlaundry_time*lt){ struct list_head *pos, *next; struct nfs4_client *clp;fs/nfsd/nfs4state.c +2
-0+bool cour; +struct list_head cslist;INIT_LIST_HEAD(reaplist);fs/nfsd/nfs4state.c +1
-0+INIT_LIST_HEAD (&cslist);spin_lock(&nn->client_lock); list_for_each_safe(pos, next, &nn->client_lru) { clp = list_entry(pos, struct nfs4_client, cl_lru); if (!state_expired(lt, clp->cl_time)) break;
fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5891
Quoted 8 lines
fs/nfsd/nfs4state.c +7
-1-if (mark_client_expired_locked (clp)) + +/* client expired */ +if (!client_has_state (clp)) { +if (mark_client_expired_locked (clp)) +continue ; +list_add (&clp->cl_lru, reaplist); +continue ;
The above looks like the same thing that is done below at the "exp_client:" label. Should you replace this with
if (!client_has_state(clp)) goto exp_client;
?
Then I think the one-line comments would be unnecessary. The code would be self-explanatory.
Quoted 16 lines
fs/nfsd/nfs4state.c +15
-0+ } + +/* expired client has state */ +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags)) + goto exp_client; + cour =test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +if (cour &&ktime_get_boottime_seconds () >= + (clp->cl_time +NFSD_COURTESY_CLIENT_TIMEOUT )) + goto exp_client; +if (nfs4_anylock_blocker (clp)) { +/* expired client has state and has blocker. */ +exp_client: +if (mark_client_expired_locked (clp)) +continue ; +list_add (&clp->cl_lru, reaplist);continue;
fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5913
Quoted 7 lines
fs/nfsd/nfs4state.c +6
-1-list_add (&clp->cl_lru, reaplist); + } +/* +* Client expired and has state and has no blockers. +* If there is race condition with blockers, next time +* the laundromat runs it will catch it and expires +* the client.
This comment is still not clear. "Client expired ... it will catch and expires the client." Do you mean "destroys the client" ?
fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5918
Quoted 5 lines
fs/nfsd/nfs4state.c +5
-0+*/ +if (!cour) { +set_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags); +list_add (&clp->cl_cs_list, &cslist); + }
cl_cs_lock is taken in the loop below, but don't you also need to hold that lock when manipulating the courtesy state bits in the code above?
Quoted 21 lines
} spin_unlock(&nn->client_lock);fs/nfsd/nfs4state.c +17
-0+ +while (!list_empty (&cslist)) { + clp = list_first_entry(&cslist, struct nfs4_client, cl_cs_list); +list_del_init (&clp->cl_cs_list); +spin_lock (&clp->cl_cs_lock); +/* +* Client might have re-connected. Make sure it's +* still in courtesy state before removing its record. +*/ +if (test_bit (NFSD4_CLIENT_EXPIRED , &clp->cl_flags) || + !test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags)) { +spin_unlock (&clp->cl_cs_lock); +continue ; + } +spin_unlock (&clp->cl_cs_lock); +nfsd4_client_record_remove (clp); + }}
Quoted 4 lines
static
time64_tSignature · 1 line
2.9.5
--
Chuck Lever
Dai Ngo
reviewerRFC PATCH v14 10/10 NFSD: Show state of courtesy clients in client info
In reply to Dai NgoUpdate client_info_show to show state of courtesy client and number of seconds since last renew.
--- fs/nfsd/nfs4state.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/nfsd/nfs4state.c +4 -0
@@ -2435,6 +2435,10 @@ static int client_info_show(struct seq_file *m, void *v)seq_puts (m,"status: confirmed\n" ); else seq_puts(m, "status: unconfirmed\n"); +seq_printf (m,"courtesy client: %s\n" , +test_bit (NFSD4_CLIENT_COURTESY , &clp->cl_flags) ?"yes" :"no" ); +seq_printf (m,"seconds from last renew: %lld\n" , +ktime_get_boottime_seconds () - clp->cl_time);seq_printf (m,"name: " );seq_quote_mem (m, clp->cl_name.data, clp->cl_name.len);seq_printf (m,"\nminor version: %d\n" , clp->cl_minorversion);
Signature · 1 line
2.9.5