Skip to contentSkip to conversation

NFSD: Initial implementation of NFSv4 Courteous Server

RFC PATCH Superseded Dai Ngo 2 participants 18 messages Feb 23, 2022 – Feb 28, 2022

Subsystems (2)

v14 of 25 Superseded Latest is v25 →

lore
Share & tools
lore

Dai Ngo

author reviewer
#0 lore raw

v14 cover NFSD: Initial implementation of NFSv4 Courteous Server

Hi Chuck, Bruce

This series of patches implement the NFSv4 Courteous Server.

A server which does not immediately expunge the state on lease expiration is known as a Courteous Server. A Courteous Server continues to recognize previously generated state tokens as valid until conflict arises between the expired state and the requests from another client, or the server reboots.

v2 patch includes:

. add new callback, lm_expire_lock, to lock_manager_operations to
allow the lock manager to take appropriate action with conflict lock.

. handle conflicts of NFSv4 locks with NFSv3/NLM and local locks.

. expire courtesy client after 24hr if client has not reconnected.

. do not allow expired client to become courtesy client if there are
waiters for client's locks.

. modify client_info_show to show courtesy client and seconds from
last renew.

. fix a problem with NFSv4.1 server where the it keeps returning
SEQ4_STATUS_CB_PATH_DOWN in the successful SEQUENCE reply, after the courtesy client reconnects, causing the client to keep sending BCTS requests to server.

v3 patch includes:

. modified posix_test_lock to check and resolve conflict locks
to handle NLM TEST and NFSv4 LOCKT requests.

. separate out fix for back channel stuck in SEQ4_STATUS_CB_PATH_DOWN.

v4 patch includes:

. rework nfsd_check_courtesy to avoid dead lock of fl_lock and client_lock
by asking the laudromat thread to destroy the courtesy client.

. handle NFSv4 share reservation conflicts with courtesy client. This
includes conflicts between access mode and deny mode and vice versa.

. drop the patch for back channel stuck in SEQ4_STATUS_CB_PATH_DOWN.

v5 patch includes:

. fix recursive locking of file_rwsem from posix_lock_file.

. retest with LOCKDEP enabled.

v6 patch includes:

. merge witn 5.15-rc7

. fix a bug in nfs4_check_deny_bmap that did not check for matched
nfs4_file before checking for access/deny conflict. This bug causes pynfs OPEN18 to fail since the server taking too long to release lots of un-conflict clients' state.

. enhance share reservation conflict handler to handle case where
a large number of conflict courtesy clients need to be expired. The 1st 100 clients are expired synchronously and the rest are expired in the background by the laundromat and NFS4ERR_DELAY is returned to the NFS client. This is needed to prevent the NFS client from timing out waiting got the reply.

v7 patch includes:

. Fix race condition in posix_test_lock and posix_lock_inode after
dropping spinlock.

. Enhance nfsd4_fl_expire_lock to work with with new lm_expire_lock
callback

. Always resolve share reservation conflicts asynchrously.

. Fix bug in nfs4_laundromat where spinlock is not used when
scanning cl_ownerstr_hashtbl.

. Fix bug in nfs4_laundromat where idr_get_next was called
with incorrect 'id'.

. Merge nfs4_destroy_courtesy_client into nfsd4_fl_expire_lock.

v8 patch includes:

. Fix warning in nfsd4_fl_expire_lock reported by test robot.

v9 patch includes:

. Simplify lm_expire_lock API by (1) remove the 'testonly' flag
and (2) specifying return value as true/false to indicate whether conflict was succesfully resolved.

. Rework nfsd4_fl_expire_lock to mark client with
NFSD4_DESTROY_COURTESY_CLIENT then tell the laundromat to expire the client in the background.

. Add a spinlock in nfs4_client to synchronize access to the
NFSD4_COURTESY_CLIENT and NFSD4_DESTROY_COURTESY_CLIENT flag to handle race conditions when resolving lock and share reservation conflict.

. Courtesy client that was marked as NFSD4_DESTROY_COURTESY_CLIENT
are now consisdered 'dead', waiting for the laundromat to expire it. This client is no longer allowed to use its states if it reconnects before the laundromat finishes expiring the client.

For v4.1 client, the detection is done in the processing of the SEQUENCE op and returns NFS4ERR_BAD_SESSION to force the client to re-establish new clientid and session.
For v4.0 client, the detection is done in the processing of the RENEW and state-related ops and return NFS4ERR_EXPIRE to force the client to re-establish new clientid.

v10 patch includes:

Resolve deadlock in v9 by avoiding getting cl_client and cl_cs_lock together. The laundromat needs to determine whether the expired client has any state and also has no blockers on its locks. Both of these conditions are allowed to change after the laundromat transits an expired client to courtesy client. When this happens, the laundromat will detect it on the next run and and expire the courtesy client.

Remove client persistent record before marking it as COURTESY_CLIENT and add client persistent record before clearing the COURTESY_CLIENT flag to allow the courtesy client to transist to normal client to continue to use its state.

Lock/delegation/share reversation conflict with courtesy client is resolved by marking the courtesy client as DESTROY_COURTESY_CLIENT, effectively disable it, then allow the current request to proceed immediately.

Courtesy client marked as DESTROY_COURTESY_CLIENT is not allowed to reconnect to reuse itsstate. It is expired by the laundromat asynchronously in the background.

Move processing of expired clients from nfs4_laudromat to a separate function, nfs4_get_client_reaplist, that creates the reaplist and also to process courtesy clients.

Update Documentation/filesystems/locking.rst to include new lm_lock_conflict call.

Modify leases_conflict to call lm_breaker_owns_lease only if there is real conflict. This is to allow the lock manager to resolve the delegation conflict if possible.

v11 patch includes:

Add comment for lm_lock_conflict callback.

Replace static const courtesy_client_expiry with macro.

Remove courtesy_clnt argument from find_in_sessionid_hashtbl. Callers use nfs4_client->cl_cs_client boolean to determined if it's the courtesy client and take appropriate actions.

Rename NFSD4_COURTESY_CLIENT and NFSD4_DESTROY_COURTESY_CLIENT with NFSD4_CLIENT_COURTESY and NFSD4_CLIENT_DESTROY_COURTESY.

v12 patch includes:

Remove unnecessary comment in nfs4_get_client_reaplist.

Replace nfs4_client->cl_cs_client boolean with
NFSD4_CLIENT_COURTESY_CLNT flag.

Remove courtesy_clnt argument from find_client_in_id_table and find_clp_in_name_tree. Callers use NFSD4_CLIENT_COURTESY_CLNT to determined if it's the courtesy client and take appropriate actions.

v13 patch includes:

Merge with 5.17-rc3.

Cleanup Documentation/filesystems/locking.rst: replace i_lock with flc_lock, update API's that use flc_lock.

Rename lm_lock_conflict to lm_lock_expired().

Remove comment of lm_lock_expired API in lock_manager_operations. Same information is in patch description.

Update commit messages of 4/4.

Add some comment for NFSD4_CLIENT_COURTESY_CLNT.

Add nfsd4_discard_courtesy_clnt() to eliminate duplicate code of discarding courtesy client; setting NFSD4_DESTROY_COURTESY_CLIENT.

v14 patch includes:

. merge with Chuck's public for-next branch.

. remove courtesy_client_expiry, use client's last renew time.

. simplify comment of nfs4_check_access_deny_bmap.

. add comment about race condition in nfs4_get_client_reaplist.

. add list_del when walking cslist in nfs4_get_client_reaplist.

. remove duplicate INIT_LIST_HEAD(&reaplist) from nfs4_laundromat

. Modify find_confirmed_client and find_confirmed_client_by_name
to detect courtesy client and destroy it.

. refactor lookup_clientid to use find_client_in_id_table
directly instead of find_confirmed_client.

. refactor nfsd4_setclientid to call find_clp_in_name_tree
directly instead of find_confirmed_client_by_name.

. remove comment of NFSD4_CLIENT_COURTESY.

. replace NFSD4_CLIENT_DESTROY_COURTESY with NFSD4_CLIENT_EXPIRED.

. replace NFSD4_CLIENT_COURTESY_CLNT with NFSD4_CLIENT_RECONNECTED.

author reviewer
#1 lore raw

RFC PATCH v14 1/10 NFSD: Add client flags, macro and spinlock to support courteous server

Add flags to track a client's lease state after the client fails to renew its lease. Add macro for courtesy client expire time. Add spinlock to synchronize access to these flags. Modify alloc_client to initialize these fields.

---
 fs/nfsd/nfs4state.c | 2 ++
 fs/nfsd/nfsd.h      | 1 +
 fs/nfsd/state.h     | 6 ++++++
 3 files changed, 9 insertions(+)
fs/nfsd/nfs4state.c +2 -0
index f3b71fd1d134..6a10f089ef4c 100644
@@ -2009,12 +2009,14 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)
 	INIT_LIST_HEAD(&clp->cl_delegations);
 	INIT_LIST_HEAD(&clp->cl_lru);
 	INIT_LIST_HEAD(&clp->cl_revoked);
+	INIT_LIST_HEAD(&clp->cl_cs_list);
 #ifdef CONFIG_NFSD_PNFS
 	INIT_LIST_HEAD(&clp->cl_lo_states);
 #endif
 	INIT_LIST_HEAD(&clp->async_copies);
 	spin_lock_init(&clp->async_lock);
 	spin_lock_init(&clp->cl_lock);
+	spin_lock_init(&clp->cl_cs_lock);
 	rpc_init_wait_queue(&clp->cl_cb_waitq, "Backchannel slot table");
 	return clp;
 err_no_hashtbl:
fs/nfsd/nfsd.h +1 -0
index 4fc1fd639527..23996c6ca75e 100644
@@ -336,6 +336,7 @@ void		nfsd_lockd_shutdown(void);
 #define COMPOUND_ERR_SLACK_SPACE	16     /* OP_SETATTR */
 
 #define NFSD_LAUNDROMAT_MINTIMEOUT      1   /* seconds */
+#define	NFSD_COURTESY_CLIENT_TIMEOUT	(24 * 60 * 60)	/* seconds */
 
 /*
  * The following attributes are currently not supported by the NFSv4 server:
fs/nfsd/state.h +6 -0
index 95457cfd37fc..8cbd3b69a584 100644
@@ -343,6 +343,9 @@ struct nfs4_client {
 #define NFSD4_CLIENT_RECLAIM_COMPLETE	(3)	/* reclaim_complete done */
 #define NFSD4_CLIENT_CONFIRMED		(4)	/* client is confirmed */
 #define NFSD4_CLIENT_UPCALL_LOCK	(5)	/* upcall serialization */
+#define	NFSD4_CLIENT_COURTESY		(6)
+#define	NFSD4_CLIENT_EXPIRED		(7)
+#define	NFSD4_CLIENT_RECONNECTED	(8)
 #define NFSD4_CLIENT_CB_FLAG_MASK	(1 << NFSD4_CLIENT_CB_UPDATE | \
 					 1 << NFSD4_CLIENT_CB_KILL)
 	unsigned long		cl_flags;
@@ -385,6 +388,9 @@ struct nfs4_client {
 	struct list_head	async_copies;	/* list of async copies */
 	spinlock_t		async_lock;	/* lock for async copies */
 	atomic_t		cl_cb_inflight;	/* Outstanding callbacks */
+
+	spinlock_t		cl_cs_lock;
+	struct list_head	cl_cs_list;
 };
 
 /* struct nfs4_client_reset
Signature · 1 line
2.9.5
author reviewer
#2 lore raw

RFC PATCH v14 2/10 NFSD: Add lm_lock_expired call out

In reply to Dai Ngo

Add callout function nfsd4_lm_lock_expired for lm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.

---
 fs/nfsd/nfs4state.c | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
fs/nfsd/nfs4state.c +37 -0
index 6a10f089ef4c..6bca727978ea 100644
@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl)
 	}
 }
 
+/**
+ * nfsd4_lm_lock_expired - check if lock conflict can be resolved.
+ *
+ * @fl: pointer to file_lock with a potential conflict
+ * Return values:
+ *   %false: real conflict, lock conflict can not be resolved.
+ *   %true: no conflict, lock conflict was resolved.
+ *
+ * Note that this function is called while the flc_lock is held.
+ */
+static bool
+nfsd4_lm_lock_expired(struct file_lock *fl)
+{
+	struct nfs4_lockowner *lo;
+	struct nfs4_client *clp;
+	bool rc = false;
+
+	if (!fl)
+		return false;
+	lo = (struct nfs4_lockowner *)fl->fl_owner;
+	clp = lo->lo_owner.so_client;
+
+	/* need to sync with courtesy client trying to reconnect */
+	spin_lock(&clp->cl_cs_lock);
+	if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags))
+		rc = true;
+	else {
+		if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+			rc =  true;
+		}
+	}
+	spin_unlock(&clp->cl_cs_lock);
+	return rc;
+}
+
 static const struct lock_manager_operations nfsd_posix_mng_ops  = {
 	.lm_notify = nfsd4_lm_notify,
 	.lm_get_owner = nfsd4_fl_get_owner,
 	.lm_put_owner = nfsd4_fl_put_owner,
+	.lm_lock_expired = nfsd4_lm_lock_expired,
 };
 
 static inline void
Signature · 1 line
2.9.5
#3 lore raw

Re: [PATCH RFC v14 02/10] NFSD: Add lm_lock_expired call out

Quoted 51 lines · Dai Ngo

Add callout function nfsd4_lm_lock_expired for lm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.

---
fs/nfsd/nfs4state.c | 37 +++++++++++++++++++++++++++++++++++++

1 file changed, 37 insertions(+)
fs/nfsd/nfs4state.c +30 -0
index 6a10f089ef4c..6bca727978ea 100644
@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl)
 	}
 }
 
+/**
+ * nfsd4_lm_lock_expired - check if lock conflict can be resolved.
+ *
+ * @fl: pointer to file_lock with a potential conflict
+ * Return values:
+ *   %false: real conflict, lock conflict can not be resolved.
+ *   %true: no conflict, lock conflict was resolved.
+ *
+ * Note that this function is called while the flc_lock is held.
+ */
+static bool
+nfsd4_lm_lock_expired(struct file_lock *fl)
+{
+	struct nfs4_lockowner *lo;
+	struct nfs4_client *clp;
+	bool rc = false;
+
+	if (!fl)
+		return false;
+	lo = (struct nfs4_lockowner *)fl->fl_owner;
+	clp = lo->lo_owner.so_client;
+
+	/* need to sync with courtesy client trying to reconnect */
+	spin_lock(&clp->cl_cs_lock);
+	if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags))
+		rc = true;
+	else {
+		if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+			rc =  true;

fs/nfsd/nfs4state.c nfsd4_lm_notify(struct file_lock *fl) · line 6620

Quoted 10 lines
fs/nfsd/nfs4state.c +6 -0
+		}
+	}
+	spin_unlock(&clp->cl_cs_lock);
+	return rc;
+}
+
 static const struct lock_manager_operations nfsd_posix_mng_ops  = {
 	.lm_notify = nfsd4_lm_notify,
 	.lm_get_owner = nfsd4_fl_get_owner,
 	.lm_put_owner = nfsd4_fl_put_owner,

I started to pull this series, but quickly realized it isn't based on the current contents of the NFSD for-next topic branch, which changes the names of nfsd4_fl_{put,get}_owner as we discussed during v13 review.

The upshot is that I would have to apply each of these patches by hand, which is tedious and error prone for more than one or two patches.

Please pull the latest from my for-next branch, rebase this series, and then post again. Thanks!

Quoted 7 lines
fs/nfsd/nfs4state.c +1 -0
+	.lm_lock_expired = nfsd4_lm_lock_expired,
 };
 

static inline void

Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#4 lore raw

Re: [PATCH RFC v14 02/10] NFSD: Add lm_lock_expired call out

Quoted 65 lines · Chuck Lever III
Quoted 50 lines · Dai Ngo

Add callout function nfsd4_lm_lock_expired for lm_lock_expired. If lock request has conflict with courtesy client then expire the courtesy client and return no conflict to caller.

---
fs/nfsd/nfs4state.c | 37 +++++++++++++++++++++++++++++++++++++

1 file changed, 37 insertions(+)
fs/nfsd/nfs4state.c +30 -0
index 6a10f089ef4c..6bca727978ea 100644
@@ -6578,10 +6578,47 @@ nfsd4_lm_notify(struct file_lock *fl)
 	}
 }
 
+/**
+ * nfsd4_lm_lock_expired - check if lock conflict can be resolved.
+ *
+ * @fl: pointer to file_lock with a potential conflict
+ * Return values:
+ *   %false: real conflict, lock conflict can not be resolved.
+ *   %true: no conflict, lock conflict was resolved.
+ *
+ * Note that this function is called while the flc_lock is held.
+ */
+static bool
+nfsd4_lm_lock_expired(struct file_lock *fl)
+{
+	struct nfs4_lockowner *lo;
+	struct nfs4_client *clp;
+	bool rc = false;
+
+	if (!fl)
+		return false;
+	lo = (struct nfs4_lockowner *)fl->fl_owner;
+	clp = lo->lo_owner.so_client;
+
+	/* need to sync with courtesy client trying to reconnect */
+	spin_lock(&clp->cl_cs_lock);
+	if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags))
+		rc = true;
+	else {
+		if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+			rc =  true;

fs/nfsd/nfs4state.c nfsd4_lm_notify(struct file_lock *fl) · line 6620

Quoted 10 lines
fs/nfsd/nfs4state.c +6 -0
+		}
+	}
+	spin_unlock(&clp->cl_cs_lock);
+	return rc;
+}
+
 static const struct lock_manager_operations nfsd_posix_mng_ops  = {
 	.lm_notify = nfsd4_lm_notify,
 	.lm_get_owner = nfsd4_fl_get_owner,
 	.lm_put_owner = nfsd4_fl_put_owner,

I started to pull this series, but quickly realized it isn't based on the current contents of the NFSD for-next topic branch, which changes the names of nfsd4_fl_{put,get}_owner as we discussed during v13 review.

Quoted 6 lines

The upshot is that I would have to apply each of these patches by hand, which is tedious and error prone for more than one or two patches.

Please pull the latest from my for-next branch, rebase this series, and then post again. Thanks!'

will do in v15.

-Dai

Quoted 14 lines
Quoted 9 lines
fs/nfsd/nfs4state.c +1 -0
+	.lm_lock_expired = nfsd4_lm_lock_expired,
 };
 

static inline void

Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#5 lore raw

RFC PATCH v14 3/10 NFSD: Update nfsd_breaker_owns_lease() to handle courtesy clients

In reply to Dai Ngo

Update nfsd_breaker_owns_lease() to handle delegation conflict with courtesy clients. If conflict was caused courtesy client then discard the courtesy client by setting CLIENT_EXPIRED and return conflict resolved. Client with CLIENT_EXPIRED is expired by the laundromat.

---
 fs/nfsd/nfs4state.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
fs/nfsd/nfs4state.c +18 -0
index 6bca727978ea..542a13676c91 100644
@@ -4727,6 +4727,24 @@ static bool nfsd_breaker_owns_lease(struct file_lock *fl)
 	struct svc_rqst *rqst;
 	struct nfs4_client *clp;
 
+	clp = dl->dl_stid.sc_client;
+	/*
+	 * need to sync with courtesy client trying to reconnect using
+	 * the cl_cs_lock, nn->client_lock can not be used since this
+	 * function is called with the fl_lck held.
+	 */
+	spin_lock(&clp->cl_cs_lock);
+	if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+		spin_unlock(&clp->cl_cs_lock);
+		return true;
+	}
+	if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+		set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+		spin_unlock(&clp->cl_cs_lock);
+		return true;
+	}
+	spin_unlock(&clp->cl_cs_lock);
+
 	if (!i_am_nfsd())
 		return false;
 	rqst = kthread_data(current);
Signature · 1 line
2.9.5
author reviewer
#6 lore raw

RFC PATCH v14 4/10 NFSD: Update nfs4_get_vfs_file() to handle courtesy clients

In reply to Dai Ngo

Update nfs4_get_vfs_file() to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then set CLIENT_EXPIRED flag to expire the courtesy clients and allow nfs4_get_vfs_file to continue. Client with CLIENT_EXPIRED is expired by the laundromat.

---
 fs/nfsd/nfs4state.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 99 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +99 -7
index 542a13676c91..1ffe7bafe90b 100644
@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,
 	return nfsd_setattr(rqstp, fh, &iattr, 0, (time64_t)0);
 }
 
+static bool
+nfs4_check_access_deny_bmap(struct nfs4_ol_stateid *stp, u32 access,
+			bool share_access)
+{
+	if (share_access) {
+		if (!stp->st_deny_bmap)
+			return false;
+
+		if ((stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_BOTH)) ||
+			(access & NFS4_SHARE_ACCESS_READ &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_READ)) ||
+			(access & NFS4_SHARE_ACCESS_WRITE &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_WRITE))) {
+			return true;
+		}
+		return false;
+	}
+	if ((access & NFS4_SHARE_DENY_BOTH) ||
+		(access & NFS4_SHARE_DENY_READ &&
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_READ)) ||
+		(access & NFS4_SHARE_DENY_WRITE &&
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_WRITE))) {
+		return true;
+	}
+	return false;
+}
+
+/*
+ * Check whether nfserr_share_denied should be returned.
+ *
+ * access:  is op_share_access if share_access is true.
+ *	    Check if access mode, op_share_access, would conflict with
+ *	    the current deny mode of the file 'fp'.
+ * access:  is op_share_deny if share_access is false.
+ *	    Check if the deny mode, op_share_deny, would conflict with
+ *	    current access of the file 'fp'.
+ * stp:     skip checking this entry.
+ * new_stp: normal open, not open upgrade.
+ *
+ * Function returns:
+ *	true   - access/deny mode conflict with normal client.
+ *	false  - no conflict or conflict with courtesy client(s) is resolved.
+ */
+static bool
+nfs4_conflict_clients(struct nfs4_file *fp, bool new_stp,
+		struct nfs4_ol_stateid *stp, u32 access, bool share_access)
+{
+	struct nfs4_ol_stateid *st;
+	struct nfs4_client *cl;
+	bool conflict = false;
+
+	lockdep_assert_held(&fp->fi_lock);
+	list_for_each_entry(st, &fp->fi_stateids, st_perfile) {
+		if (st->st_openstp || (st == stp && new_stp) ||
+			(!nfs4_check_access_deny_bmap(st,
+					access, share_access)))
+			continue;
+
+		/* need to sync with courtesy client trying to reconnect */
+		cl = st->st_stid.sc_client;
+		spin_lock(&cl->cl_cs_lock);
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags)) {
+			spin_unlock(&cl->cl_cs_lock);
+			continue;
+		}
+		if (test_bit(NFSD4_CLIENT_COURTESY, &cl->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags);
+			spin_unlock(&cl->cl_cs_lock);
+			continue;
+		}
+		/* conflict not caused by courtesy client */
+		spin_unlock(&cl->cl_cs_lock);
+		conflict = true;
+		break;
+	}
+	return conflict;
+}
+
 static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 		struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp,
-		struct nfsd4_open *open)
+		struct nfsd4_open *open, bool new_stp)
 {
 	struct nfsd_file *nf = NULL;
 	__be32 status;
@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 	 */
 	status = nfs4_file_check_deny(fp, open->op_share_deny);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);
-		goto out;
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_deny, false)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
 	}
 
 	/* set access to the file */
 	status = nfs4_file_get_access(fp, open->op_share_access);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);
-		goto out;
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_access, true)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
 	}
 
 	/* Set access bits in stateid */
@@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *c
 	unsigned char old_deny_bmap = stp->st_deny_bmap;
 
 	if (!test_access(open->op_share_access, stp))
-		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open);
+		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open, false);
 
 	/* test and set deny mode */
 	spin_lock(&fp->fi_lock);
@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 			goto out;
 		}
 	} else {
-		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open);
+		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open, true);
 		if (status) {
 			stp->st_stid.sc_type = NFS4_CLOSED_STID;
 			release_open_stateid(stp);
Signature · 1 line
2.9.5
#7 lore raw

Re: [PATCH RFC v14 04/10] NFSD: Update nfs4_get_vfs_file() to handle courtesy clients

Quoted 12 lines · Dai Ngo

Update nfs4_get_vfs_file() to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then set CLIENT_EXPIRED flag to expire the courtesy clients and allow nfs4_get_vfs_file to continue. Client with CLIENT_EXPIRED is expired by the laundromat.

I'm still getting my head around this, but there are some items that can be addressed now, below.

Quoted 31 lines

---
fs/nfsd/nfs4state.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++----

1 file changed, 99 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +19 -0
index 542a13676c91..1ffe7bafe90b 100644
@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,
 	return nfsd_setattr(rqstp, fh, &iattr, 0, (time64_t)0);
 }
 
+static bool
+nfs4_check_access_deny_bmap(struct nfs4_ol_stateid *stp, u32 access,
+			bool share_access)
+{
+	if (share_access) {
+		if (!stp->st_deny_bmap)
+			return false;
+
+		if ((stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_BOTH)) ||
+			(access & NFS4_SHARE_ACCESS_READ &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_READ)) ||
+			(access & NFS4_SHARE_ACCESS_WRITE &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_WRITE))) {
+			return true;
+		}
+		return false;
+	}
+	if ((access & NFS4_SHARE_DENY_BOTH) ||
+		(access & NFS4_SHARE_DENY_READ &&

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 4996

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_READ)) ||
+		(access & NFS4_SHARE_DENY_WRITE &&
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_WRITE))) {
+		return true;
+	}
+	return false;
+}
+
+/*
+ * Check whether nfserr_share_denied should be returned.

This function is doing more than adjusting a return code, now that I'm reading it more carefully. How about "Check whether courtesy clients have conflicting access."

Quoted 7 lines
fs/nfsd/nfs4state.c +7 -0
+ *
+ * access:  is op_share_access if share_access is true.
+ *	    Check if access mode, op_share_access, would conflict with
+ *	    the current deny mode of the file 'fp'.
+ * access:  is op_share_deny if share_access is false.
+ *	    Check if the deny mode, op_share_deny, would conflict with
+ *	    current access of the file 'fp'.

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5013

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+ * stp:     skip checking this entry.
+ * new_stp: normal open, not open upgrade.
+ *
+ * Function returns:
+ *	true   - access/deny mode conflict with normal client.
+ *	false  - no conflict or conflict with courtesy client(s) is resolved.
+ */
+static bool
+nfs4_conflict_clients(struct nfs4_file *fp, bool new_stp,
+		struct nfs4_ol_stateid *stp, u32 access, bool share_access)

Functions that are called with locks held are usually suffixed with "_locked" -- this one should be too.

A better name might be "nfs4_resolve_deny_conflicts_locked".

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5016

Quoted 3 lines
fs/nfsd/nfs4state.c +3 -0
+{
+	struct nfs4_ol_stateid *st;
+	struct nfs4_client *cl;

Use "clp" to be consistent with other areas of the code.

Quoted 13 lines
fs/nfsd/nfs4state.c +13 -0
+	bool conflict = false;
+
+	lockdep_assert_held(&fp->fi_lock);
+	list_for_each_entry(st, &fp->fi_stateids, st_perfile) {
+		if (st->st_openstp || (st == stp && new_stp) ||
+			(!nfs4_check_access_deny_bmap(st,
+					access, share_access)))
+			continue;
+
+		/* need to sync with courtesy client trying to reconnect */
+		cl = st->st_stid.sc_client;
+		spin_lock(&cl->cl_cs_lock);
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags)) {

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5039

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+			spin_unlock(&cl->cl_cs_lock);
+			continue;
+		}
+		if (test_bit(NFSD4_CLIENT_COURTESY, &cl->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags);
+			spin_unlock(&cl->cl_cs_lock);
+			continue;
+		}
+		/* conflict not caused by courtesy client */
+		spin_unlock(&cl->cl_cs_lock);

I think I'm seeing similar code as this in some of the other patches. Whereever you can, please deduplicate by creating a helper function and moving the common code there.

Quoted 19 lines
fs/nfsd/nfs4state.c +7 -3
+		conflict = true;
+		break;
+	}
+	return conflict;
+}
+
 static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 		struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp,
-		struct nfsd4_open *open)
+		struct nfsd4_open *open, bool new_stp)
 {
 	struct nfsd_file *nf = NULL;
 	__be32 status;
@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 	 */
 	status = nfs4_file_check_deny(fp, open->op_share_deny);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);
-		goto out;

fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5073

Quoted 10 lines
fs/nfsd/nfs4state.c +9 -0
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_deny, false)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
 	}

Doesn't nfs4_upgrade_open() need to perform the same check?

Quoted 5 lines
fs/nfsd/nfs4state.c +0 -1
 	/* set access to the file */
 	status = nfs4_file_get_access(fp, open->op_share_access);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);

fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5086

Quoted 10 lines
fs/nfsd/nfs4state.c +9 -1
-		goto out;
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_access, true)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}

This is nfs4_file_get_access()'s only caller. Should the call to nfs4_conflict_clients() be moved into nfs4_file_get_access() ?

Quoted 24 lines
fs/nfsd/nfs4state.c +2 -2
 	}
 
 	/* Set access bits in stateid */
@@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *c
 	unsigned char old_deny_bmap = stp->st_deny_bmap;
 
 	if (!test_access(open->op_share_access, stp))
-		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open);
+		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open, false);
 
 	/* test and set deny mode */
 	spin_lock(&fp->fi_lock);
@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 			goto out;
 		}
 	} else {
-		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open);
+		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open, true);
 		if (status) {
 			stp->st_stid.sc_type = NFS4_CLOSED_STID;
 			release_open_stateid(stp);
Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#8 lore raw

Re: [PATCH RFC v14 04/10] NFSD: Update nfs4_get_vfs_file() to handle courtesy clients

Quoted 51 lines · Chuck Lever III
Quoted 11 lines · Dai Ngo

Update nfs4_get_vfs_file() to handle share reservation conflict with courtesy client. If share/access check fails with share denied then check if the conflict was caused by courtesy clients. If that's the case then set CLIENT_EXPIRED flag to expire the courtesy clients and allow nfs4_get_vfs_file to continue. Client with CLIENT_EXPIRED is expired by the laundromat.

I'm still getting my head around this, but there are some items that can be addressed now, below.

Quoted 36 lines

---
fs/nfsd/nfs4state.c | 106 ++++++++++++++++++++++++++++++++++++++++++++++++----

1 file changed, 99 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +24 -0
index 542a13676c91..1ffe7bafe90b 100644
@@ -4965,9 +4965,87 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,
 	return nfsd_setattr(rqstp, fh, &iattr, 0, (time64_t)0);
 }
 
+static bool
+nfs4_check_access_deny_bmap(struct nfs4_ol_stateid *stp, u32 access,
+			bool share_access)
+{
+	if (share_access) {
+		if (!stp->st_deny_bmap)
+			return false;
+
+		if ((stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_BOTH)) ||
+			(access & NFS4_SHARE_ACCESS_READ &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_READ)) ||
+			(access & NFS4_SHARE_ACCESS_WRITE &&
+				stp->st_deny_bmap & (1 << NFS4_SHARE_DENY_WRITE))) {
+			return true;
+		}
+		return false;
+	}
+	if ((access & NFS4_SHARE_DENY_BOTH) ||
+		(access & NFS4_SHARE_DENY_READ &&
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_READ)) ||
+		(access & NFS4_SHARE_DENY_WRITE &&
+			stp->st_access_bmap & (1 << NFS4_SHARE_ACCESS_WRITE))) {
+		return true;
+	}
Quoted 8 lines

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 4996

Quoted 5 lines
fs/nfsd/nfs4state.c +5 -0
+	return false;
+}
+
+/*
+ * Check whether nfserr_share_denied should be returned.

This function is doing more than adjusting a return code, now that I'm reading it more carefully. How about "Check whether courtesy clients have conflicting access."

fix in v15.

Quoted 22 lines
Quoted 9 lines
fs/nfsd/nfs4state.c +7 -0
+ *
+ * access:  is op_share_access if share_access is true.
+ *	    Check if access mode, op_share_access, would conflict with
+ *	    the current deny mode of the file 'fp'.
+ * access:  is op_share_deny if share_access is false.
+ *	    Check if the deny mode, op_share_deny, would conflict with
+ *	    current access of the file 'fp'.

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5013

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+ * stp:     skip checking this entry.
+ * new_stp: normal open, not open upgrade.
+ *
+ * Function returns:
+ *	true   - access/deny mode conflict with normal client.
+ *	false  - no conflict or conflict with courtesy client(s) is resolved.
+ */
+static bool
+nfs4_conflict_clients(struct nfs4_file *fp, bool new_stp,
+		struct nfs4_ol_stateid *stp, u32 access, bool share_access)

Functions that are called with locks held are usually suffixed with "_locked" -- this one should be too.

Quoted 1 line

A better name might be "nfs4_resolve_deny_conflicts_locked".

Fix in v15.

Quoted 6 lines

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5016

Quoted 5 lines
fs/nfsd/nfs4state.c +3 -0
+{
+	struct nfs4_ol_stateid *st;
+	struct nfs4_client *cl;

Use "clp" to be consistent with other areas of the code.

Fix in v15.

Quoted 21 lines
Quoted 21 lines
fs/nfsd/nfs4state.c +19 -0
+	bool conflict = false;
+
+	lockdep_assert_held(&fp->fi_lock);
+	list_for_each_entry(st, &fp->fi_stateids, st_perfile) {
+		if (st->st_openstp || (st == stp && new_stp) ||
+			(!nfs4_check_access_deny_bmap(st,
+					access, share_access)))
+			continue;
+
+		/* need to sync with courtesy client trying to reconnect */
+		cl = st->st_stid.sc_client;
+		spin_lock(&cl->cl_cs_lock);
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags)) {
+			spin_unlock(&cl->cl_cs_lock);
+			continue;
+		}
+		if (test_bit(NFSD4_CLIENT_COURTESY, &cl->cl_flags)) {
+			set_bit(NFSD4_CLIENT_EXPIRED, &cl->cl_flags);
+			spin_unlock(&cl->cl_cs_lock);
Quoted 8 lines

fs/nfsd/nfs4state.c nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh, · line 5039

Quoted 4 lines
fs/nfsd/nfs4state.c +4 -0
+			continue;
+		}
+		/* conflict not caused by courtesy client */
+		spin_unlock(&cl->cl_cs_lock);

I think I'm seeing similar code as this in some of the other patches. Whereever you can, please deduplicate by creating a helper function and moving the common code there.

Fix in v15.

Quoted 24 lines
Quoted 24 lines
fs/nfsd/nfs4state.c +10 -3
+		conflict = true;
+		break;
+	}
+	return conflict;
+}
+
 static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 		struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp,
-		struct nfsd4_open *open)
+		struct nfsd4_open *open, bool new_stp)
 {
 	struct nfsd_file *nf = NULL;
 	__be32 status;
@@ -4983,15 +5061,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
 	 */
 	status = nfs4_file_check_deny(fp, open->op_share_deny);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);
-		goto out;
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
Quoted 8 lines

fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5073

Quoted 7 lines
fs/nfsd/nfs4state.c +6 -0
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_deny, false)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
 	}

Doesn't nfs4_upgrade_open() need to perform the same check?

Yes, we need to do the same check here. Fix in v15.

Quoted 18 lines
Quoted 6 lines
fs/nfsd/nfs4state.c +0 -1
 	/* set access to the file */
 	status = nfs4_file_get_access(fp, open->op_share_access);
 	if (status != nfs_ok) {
-		spin_unlock(&fp->fi_lock);

fs/nfsd/nfs4state.c static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp, · line 5086

Quoted 10 lines
fs/nfsd/nfs4state.c +9 -1
-		goto out;
+		if (status != nfserr_share_denied) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}
+		if (nfs4_conflict_clients(fp, new_stp, stp,
+				open->op_share_access, true)) {
+			spin_unlock(&fp->fi_lock);
+			goto out;
+		}

This is nfs4_file_get_access()'s only caller. Should the call to nfs4_conflict_clients() be moved into nfs4_file_get_access() ?

We could, but I think it's better to keep the call sign of nfs4_file_get_access as is for speed since we don't have to pass additional parameters for nfs4_resolve_deny_conflicts_locked which should rarely need to run. Also, I think it's easier to understand to code when both access and deny check are in the same place.

-Dai

Quoted 31 lines
Quoted 26 lines
fs/nfsd/nfs4state.c +2 -2
 	}
 
 	/* Set access bits in stateid */
@@ -5042,7 +5134,7 @@ nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *c
 	unsigned char old_deny_bmap = stp->st_deny_bmap;
 
 	if (!test_access(open->op_share_access, stp))
-		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open);
+		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open, false);
 
 	/* test and set deny mode */
 	spin_lock(&fp->fi_lock);
@@ -5391,7 +5483,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 			goto out;
 		}
 	} else {
-		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open);
+		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open, true);
 		if (status) {
 			stp->st_stid.sc_type = NFS4_CLOSED_STID;
 			release_open_stateid(stp);
Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#9 lore raw

RFC PATCH v14 5/10 NFSD: Update find_clp_in_name_tree() to handle courtesy clients

In reply to Dai Ngo

Update find_clp_in_name_tree:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.

Update find_confirmed_client_by_name to discard the courtesy client; set CLIENT_EXPIRED.

Update nfsd4_setclientid to expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the the conf_id_hashtbl list.

---
 fs/nfsd/nfs4state.c | 45 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +42 -3
index 1ffe7bafe90b..4990553180f8 100644
@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root)
 			node = node->rb_left;
 		else if (cmp < 0)
 			node = node->rb_right;
-		else
+		else {
+			clear_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			/* sync with thread resolving lock/deleg conflict */
+			spin_lock(&clp->cl_cs_lock);
+			if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+				spin_unlock(&clp->cl_cs_lock);
+				return NULL;
+			}
+			if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+				clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+				set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			}
+			spin_unlock(&clp->cl_cs_lock);
 			return clp;
+		}
 	}
 	return NULL;
 }
@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)
 	return NULL;
 }
 
+static void
+nfsd4_discard_courtesy_clnt(struct nfs4_client *clp)
+{
+	spin_lock(&clp->cl_cs_lock);
+	set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+	spin_unlock(&clp->cl_cs_lock);
+}
+
 static struct nfs4_client *
 find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn)
 {
@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)
 static struct nfs4_client *
 find_confirmed_client_by_name(struct xdr_netobj *name, struct nfsd_net *nn)
 {
+	struct nfs4_client *clp;
+
 	lockdep_assert_held(&nn->client_lock);
-	return find_clp_in_name_tree(name, &nn->conf_name_tree);
+	clp = find_clp_in_name_tree(name, &nn->conf_name_tree);
+	if (clp && test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		nfsd4_discard_courtesy_clnt(clp);
+		clp = NULL;
+	}
+	return clp;
 }
 
 static struct nfs4_client *
@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfs4_client	*unconf = NULL;
 	__be32 			status;
 	struct nfsd_net		*nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	struct nfs4_client	*cclient = NULL;
 
 	new = create_client(clname, rqstp, &clverifier);
 	if (new == NULL)
 		return nfserr_jukebox;
 	spin_lock(&nn->client_lock);
-	conf = find_confirmed_client_by_name(&clname, nn);
+	/* find confirmed client by name */
+	conf = find_clp_in_name_tree(&clname, &nn->conf_name_tree);
+	if (conf && test_bit(NFSD4_CLIENT_RECONNECTED, &conf->cl_flags)) {
+		cclient = conf;
+		conf = NULL;
+	}
+
 	if (conf && client_has_state(conf)) {
 		status = nfserr_clid_inuse;
 		if (clp_used_exchangeid(conf))
@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	new = NULL;
 	status = nfs_ok;
 out:
+	if (cclient)
+		unhash_client_locked(cclient);
 	spin_unlock(&nn->client_lock);
+	if (cclient)
+		expire_client(cclient);
 	if (new)
 		free_client(new);
 	if (unconf) {
Signature · 1 line
2.9.5
#10 lore raw

Re: [PATCH RFC v14 05/10] NFSD: Update find_clp_in_name_tree() to handle courtesy clients

Good morning, Dai!

Quoted 30 lines · Dai Ngo

Update find_clp_in_name_tree:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.

Update find_confirmed_client_by_name to discard the courtesy client; set CLIENT_EXPIRED.

Update nfsd4_setclientid to expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the the conf_id_hashtbl list.

---
fs/nfsd/nfs4state.c | 45 ++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 42 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +2 -1
index 1ffe7bafe90b..4990553180f8 100644
@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root)
 			node = node->rb_left;
 		else if (cmp < 0)
 			node = node->rb_right;
-		else
+		else {
+			clear_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);

fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2848

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+			/* sync with thread resolving lock/deleg conflict */
+			spin_lock(&clp->cl_cs_lock);
+			if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+				spin_unlock(&clp->cl_cs_lock);
+				return NULL;
+			}
+			if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+				clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+				set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			}

This should be written

	if (test_and_clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags))
		set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);

fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2851

Quoted 3 lines
fs/nfsd/nfs4state.c +2 -0
+			spin_unlock(&clp->cl_cs_lock);
 			return clp;
+		}

And since this is the same logic that is used in 6/10 and 7/10, please refactor this into a separate function and call it from the find_yada_ functions as we discussed previously. Thanks!

Quoted 65 lines
fs/nfsd/nfs4state.c +8 -0
 	}
 	return NULL;
 }
@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)
 	return NULL;
 }
 
+static void
+nfsd4_discard_courtesy_clnt(struct nfs4_client *clp)
+{
+	spin_lock(&clp->cl_cs_lock);
+	set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+	spin_unlock(&clp->cl_cs_lock);
+}
+

static struct nfs4_client *
find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn) {

fs/nfsd/nfs4state.c +0 -0
@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)

static struct nfs4_client * find_confirmed_client_by_name(struct xdr_netobj *name, struct nfsd_net *nn) {

fs/nfsd/nfs4state.c +2 -0
+	struct nfs4_client *clp;
+
	lockdep_assert_held(&nn->client_lock);
fs/nfsd/nfs4state.c +6 -1
-	return find_clp_in_name_tree(name, &nn->conf_name_tree);
+	clp = find_clp_in_name_tree(name, &nn->conf_name_tree);
+	if (clp && test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		nfsd4_discard_courtesy_clnt(clp);
+		clp = NULL;
+	}
+	return clp;

}

static struct nfs4_client *

fs/nfsd/nfs4state.c +8 -1
@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfs4_client	*unconf = NULL;
 	__be32 			status;
 	struct nfsd_net		*nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	struct nfs4_client	*cclient = NULL;
 
 	new = create_client(clname, rqstp, &clverifier);
 	if (new == NULL)
 		return nfserr_jukebox;
 	spin_lock(&nn->client_lock);
-	conf = find_confirmed_client_by_name(&clname, nn);
+	/* find confirmed client by name */
+	conf = find_clp_in_name_tree(&clname, &nn->conf_name_tree);
+	if (conf && test_bit(NFSD4_CLIENT_RECONNECTED, &conf->cl_flags)) {
+		cclient = conf;
+		conf = NULL;
+	}
+
 	if (conf && client_has_state(conf)) {
 		status = nfserr_clid_inuse;
 		if (clp_used_exchangeid(conf))
@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	new = NULL;
 	status = nfs_ok;

out:

fs/nfsd/nfs4state.c +2 -0
+	if (cclient)
+		unhash_client_locked(cclient);
	spin_unlock(&nn->client_lock);
fs/nfsd/nfs4state.c +2 -0
+	if (cclient)
+		expire_client(cclient);
Quoted 6 lines
	if (new)
		free_client(new);
	if (unconf) {
Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#11 lore raw

Re: [PATCH RFC v14 05/10] NFSD: Update find_clp_in_name_tree() to handle courtesy clients

Quoted 45 lines · Chuck Lever III

Good morning, Dai!

Quoted 30 lines · Dai Ngo

Update find_clp_in_name_tree:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.

Update find_confirmed_client_by_name to discard the courtesy client; set CLIENT_EXPIRED.

Update nfsd4_setclientid to expire the confirmed courtesy client to prevent multiple confirmed clients with the same name on the the conf_id_hashtbl list.

---
fs/nfsd/nfs4state.c | 45 ++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 42 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +2 -1
index 1ffe7bafe90b..4990553180f8 100644
@@ -2834,8 +2834,21 @@ find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root)
 			node = node->rb_left;
 		else if (cmp < 0)
 			node = node->rb_right;
-		else
+		else {
+			clear_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);

fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2848

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+			/* sync with thread resolving lock/deleg conflict */
+			spin_lock(&clp->cl_cs_lock);
+			if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+				spin_unlock(&clp->cl_cs_lock);
+				return NULL;
+			}
+			if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+				clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+				set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			}

This should be written

Quoted 2 lines
	if (test_and_clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags))
		set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);

Fix in v15.

Quoted 8 lines

fs/nfsd/nfs4state.c find_clp_in_name_tree(struct xdr_netobj *name, struct rb_root *root) · line 2851

Quoted 5 lines
fs/nfsd/nfs4state.c +2 -0
+			spin_unlock(&clp->cl_cs_lock);
 			return clp;
+		}

And since this is the same logic that is used in 6/10 and 7/10, please refactor this into a separate function and call it from the find_yada_ functions as we discussed previously. Thanks!

Fix in v15.

-Dai

Quoted 78 lines
Quoted 67 lines
fs/nfsd/nfs4state.c +8 -0
 	}
 	return NULL;
 }
@@ -2888,6 +2901,14 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)
 	return NULL;
 }
 
+static void
+nfsd4_discard_courtesy_clnt(struct nfs4_client *clp)
+{
+	spin_lock(&clp->cl_cs_lock);
+	set_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags);
+	spin_unlock(&clp->cl_cs_lock);
+}
+

static struct nfs4_client *
find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn) {

fs/nfsd/nfs4state.c +0 -0
@@ -2914,8 +2935,15 @@ static bool clp_used_exchangeid(struct nfs4_client *clp)

static struct nfs4_client * find_confirmed_client_by_name(struct xdr_netobj *name, struct nfsd_net *nn) {

fs/nfsd/nfs4state.c +2 -0
+	struct nfs4_client *clp;
+
	lockdep_assert_held(&nn->client_lock);
fs/nfsd/nfs4state.c +6 -1
-	return find_clp_in_name_tree(name, &nn->conf_name_tree);
+	clp = find_clp_in_name_tree(name, &nn->conf_name_tree);
+	if (clp && test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		nfsd4_discard_courtesy_clnt(clp);
+		clp = NULL;
+	}
+	return clp;

}

static struct nfs4_client *

fs/nfsd/nfs4state.c +8 -1
@@ -4032,12 +4060,19 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfs4_client	*unconf = NULL;
 	__be32 			status;
 	struct nfsd_net		*nn = net_generic(SVC_NET(rqstp), nfsd_net_id);
+	struct nfs4_client	*cclient = NULL;
 
 	new = create_client(clname, rqstp, &clverifier);
 	if (new == NULL)
 		return nfserr_jukebox;
 	spin_lock(&nn->client_lock);
-	conf = find_confirmed_client_by_name(&clname, nn);
+	/* find confirmed client by name */
+	conf = find_clp_in_name_tree(&clname, &nn->conf_name_tree);
+	if (conf && test_bit(NFSD4_CLIENT_RECONNECTED, &conf->cl_flags)) {
+		cclient = conf;
+		conf = NULL;
+	}
+
 	if (conf && client_has_state(conf)) {
 		status = nfserr_clid_inuse;
 		if (clp_used_exchangeid(conf))
@@ -4068,7 +4103,11 @@ nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	new = NULL;
 	status = nfs_ok;

out:

fs/nfsd/nfs4state.c +2 -0
+	if (cclient)
+		unhash_client_locked(cclient);
	spin_unlock(&nn->client_lock);
fs/nfsd/nfs4state.c +2 -0
+	if (cclient)
+		expire_client(cclient);
Quoted 6 lines
	if (new)
		free_client(new);
	if (unconf) {
Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#12 lore raw

RFC PATCH v14 6/10 NFSD: Update find_in_sessionid_hashtbl() to handle courtesy clients

In reply to Dai Ngo

Update find_in_sessionid_hashtbl to:
. skip client with CLIENT_EXPIRED flag; discarded courtesy client. . if courtesy client was found then clear CLIENT_COURTESY and
set CLIENT_RECONNECTED so callers can take appropriate action.

Update nfsd4_sequence and nfsd4_bind_conn_to_session to create client record for client with CLIENT_RECONNECTED set.

Update nfsd4_destroy_session to discard client with CLIENT_RECONNECTED set.

---
 fs/nfsd/nfs4state.c | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)
fs/nfsd/nfs4state.c +43 -1
index 4990553180f8..5cef1a78cc28 100644
@@ -1935,10 +1935,31 @@ find_in_sessionid_hashtbl(struct nfs4_sessionid *sessionid, struct net *net,
 {
 	struct nfsd4_session *session;
 	__be32 status = nfserr_badsession;
+	struct nfs4_client *clp;
 
 	session = __find_in_sessionid_hashtbl(sessionid, net);
 	if (!session)
 		goto out;
+	clp = session->se_client;
+	if (clp) {
+		clear_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+		/* need to sync with thread resolving lock/deleg conflict */
+		spin_lock(&clp->cl_cs_lock);
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+			spin_unlock(&clp->cl_cs_lock);
+			session = NULL;
+			goto out;
+		}
+		if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			/*
+			 * clear CLIENT_COURTESY flag to prevent it from being
+			 * destroyed by thread trying to resolve conflicts.
+			 */
+			clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+			set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+		}
+		spin_unlock(&clp->cl_cs_lock);
+	}
 	status = nfsd4_get_session_locked(session);
 	if (status)
 		session = NULL;
@@ -3661,6 +3682,7 @@ __be32 nfsd4_bind_conn_to_session(struct svc_rqst *rqstp,
 	struct nfsd4_session *session;
 	struct net *net = SVC_NET(rqstp);
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
+	struct nfs4_client *clp;
 
 	if (!nfsd4_last_compound_op(rqstp))
 		return nfserr_not_only_op;
@@ -3693,6 +3715,13 @@ __be32 nfsd4_bind_conn_to_session(struct svc_rqst *rqstp,
 	nfsd4_init_conn(rqstp, conn, session);
 	status = nfs_ok;
 out:
+	clp = session->se_client;
+	if (test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		if (status == nfs_ok)
+			nfsd4_client_record_create(clp);
+		else
+			nfsd4_discard_courtesy_clnt(clp);
+	}
 	nfsd4_put_session(session);
 out_no_session:
 	return status;
@@ -3715,6 +3744,7 @@ nfsd4_destroy_session(struct svc_rqst *r, struct nfsd4_compound_state *cstate,
 	int ref_held_by_me = 0;
 	struct net *net = SVC_NET(r);
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
+	struct nfs4_client *clp;
 
 	status = nfserr_not_only_op;
 	if (nfsd4_compound_in_session(cstate, sessionid)) {
@@ -3727,6 +3757,12 @@ nfsd4_destroy_session(struct svc_rqst *r, struct nfsd4_compound_state *cstate,
 	ses = find_in_sessionid_hashtbl(sessionid, net, &status);
 	if (!ses)
 		goto out_client_lock;
+	clp = ses->se_client;
+	if (test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		status = nfserr_badsession;
+		nfsd4_discard_courtesy_clnt(clp);
+		goto out_put_session;
+	}
 	status = nfserr_wrong_cred;
 	if (!nfsd4_mach_creds_match(ses->se_client, r))
 		goto out_put_session;
@@ -3831,7 +3867,7 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	struct nfsd4_compoundres *resp = rqstp->rq_resp;
 	struct xdr_stream *xdr = resp->xdr;
 	struct nfsd4_session *session;
-	struct nfs4_client *clp;
+	struct nfs4_client *clp = NULL;
 	struct nfsd4_slot *slot;
 	struct nfsd4_conn *conn;
 	__be32 status;
@@ -3941,6 +3977,12 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	if (conn)
 		free_conn(conn);
 	spin_unlock(&nn->client_lock);
+	if (clp && test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		if (status == nfs_ok)
+			nfsd4_client_record_create(clp);
+		else
+			nfsd4_discard_courtesy_clnt(clp);
+	}
 	return status;
 out_put_session:
 	nfsd4_put_session_locked(session);
Signature · 1 line
2.9.5
author reviewer
#13 lore raw

RFC PATCH v14 7/10 NFSD: Update find_client_in_id_table() to handle courtesy clients

In reply to Dai Ngo

Update find_client_in_id_table to:
. skip client with CLIENT_EXPIRED; discarded courtesy client . if courtesy client was found then clear CLIENT_COURTESY and set
CLIENT_RECONNECTED flag so callers can take appropriate action.

Update find_confirmed_client to discard courtesy client. Update lookup_clientid to call find_client_in_id_table directly. Update set_client to create client record for courtesy client. Update find_cpntf_state to discard courtesy client.

---
 fs/nfsd/nfs4state.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)
fs/nfsd/nfs4state.c +32 -2
index 5cef1a78cc28..6e3ca0ea4f28 100644
@@ -2915,6 +2915,19 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)
 		if (same_clid(&clp->cl_clientid, clid)) {
 			if ((bool)clp->cl_minorversion != sessions)
 				return NULL;
+
+			/* need to sync with thread resolving lock/deleg conflict */
+			clear_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			spin_lock(&clp->cl_cs_lock);
+			if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags)) {
+				spin_unlock(&clp->cl_cs_lock);
+				continue;
+			}
+			if (test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+				clear_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+				set_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags);
+			}
+			spin_unlock(&clp->cl_cs_lock);
 			renew_client_locked(clp);
 			return clp;
 		}
@@ -2922,6 +2935,7 @@ find_client_in_id_table(struct list_head *tbl, clientid_t *clid, bool sessions)
 	return NULL;
 }
 
+
 static void
 nfsd4_discard_courtesy_clnt(struct nfs4_client *clp)
 {
@@ -2934,9 +2948,15 @@ static struct nfs4_client *
 find_confirmed_client(clientid_t *clid, bool sessions, struct nfsd_net *nn)
 {
 	struct list_head *tbl = nn->conf_id_hashtbl;
+	struct nfs4_client *clp;
 
 	lockdep_assert_held(&nn->client_lock);
-	return find_client_in_id_table(tbl, clid, sessions);
+	clp = find_client_in_id_table(tbl, clid, sessions);
+	if (clp && test_bit(NFSD4_CLIENT_RECONNECTED, &clp->cl_flags)) {
+		nfsd4_discard_courtesy_clnt(clp);
+		clp = NULL;
+	}
+	return clp;
 }
 
 static struct nfs4_client *
@@ -4867,9 +4887,10 @@ static struct nfs4_client *lookup_clientid(clientid_t *clid, bool sessions,
 						struct nfsd_net *nn)
 {
 	struct nfs4_client *found;
+	struct list_head *tbl = nn->conf_id_hashtbl;
 
 	spin_lock(&nn->client_lock);
-	found = find_confirmed_client(clid, sessions, nn);
+	found = find_client_in_id_table(tbl, clid, sessions);
 	if (found)
 		atomic_inc(&found->cl_rpc_users);
 	spin_unlock(&nn->client_lock);
@@ -4894,6 +4915,8 @@ static __be32 set_client(clientid_t *clid,
 	cstate->clp = lookup_clientid(clid, false, nn);
 	if (!cstate->clp)
 		return nfserr_expired;
+	if (test_bit(NFSD4_CLIENT_RECONNECTED, &cstate->clp->cl_flags))
+		nfsd4_client_record_create(cstate->clp);
 	return nfs_ok;
 }
 
@@ -6222,6 +6245,13 @@ static __be32 find_cpntf_state(struct nfsd_net *nn, stateid_t *st,
 	found = lookup_clientid(&cps->cp_p_clid, true, nn);
 	if (!found)
 		goto out;
+	if (test_bit(NFSD4_CLIENT_RECONNECTED, &found->cl_flags)) {
+		nfsd4_discard_courtesy_clnt(found);
+		if (atomic_dec_and_lock(&found->cl_rpc_users,
+				&nn->client_lock))
+			spin_unlock(&nn->client_lock);
+		goto out;
+	}
 
 	*stid = find_stateid_by_type(found, &cps->cp_p_stateid,
 			NFS4_DELEG_STID|NFS4_OPEN_STID|NFS4_LOCK_STID);
Signature · 1 line
2.9.5
author reviewer
#14 lore raw

RFC PATCH v14 8/10 NFSD: Refactor nfsd4_laundromat()

In reply to Dai Ngo

Extract a bit of logic that is about to be expanded to handle courtesy clients.

---
 fs/nfsd/nfs4state.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
fs/nfsd/nfs4state.c +21 -12
index 6e3ca0ea4f28..01c51adf4873 100644
@@ -5821,6 +5821,26 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
 }
 #endif
 
+static void
+nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
+				struct laundry_time *lt)
+{
+	struct list_head *pos, *next;
+	struct nfs4_client *clp;
+
+	INIT_LIST_HEAD(reaplist);
+	spin_lock(&nn->client_lock);
+	list_for_each_safe(pos, next, &nn->client_lru) {
+		clp = list_entry(pos, struct nfs4_client, cl_lru);
+		if (!state_expired(lt, clp->cl_time))
+			break;
+		if (mark_client_expired_locked(clp))
+			continue;
+		list_add(&clp->cl_lru, reaplist);
+	}
+	spin_unlock(&nn->client_lock);
+}
+
 static time64_t
 nfs4_laundromat(struct nfsd_net *nn)
 {
@@ -5843,7 +5863,6 @@ nfs4_laundromat(struct nfsd_net *nn)
 		goto out;
 	}
 	nfsd4_end_grace(nn);
-	INIT_LIST_HEAD(&reaplist);
 
 	spin_lock(&nn->s2s_cp_lock);
 	idr_for_each_entry(&nn->s2s_cp_stateids, cps_t, i) {
@@ -5853,17 +5872,7 @@ nfs4_laundromat(struct nfsd_net *nn)
 			_free_cpntf_state_locked(nn, cps);
 	}
 	spin_unlock(&nn->s2s_cp_lock);
-
-	spin_lock(&nn->client_lock);
-	list_for_each_safe(pos, next, &nn->client_lru) {
-		clp = list_entry(pos, struct nfs4_client, cl_lru);
-		if (!state_expired(&lt, clp->cl_time))
-			break;
-		if (mark_client_expired_locked(clp))
-			continue;
-		list_add(&clp->cl_lru, &reaplist);
-	}
-	spin_unlock(&nn->client_lock);
+	nfs4_get_client_reaplist(nn, &reaplist, &lt);
 	list_for_each_safe(pos, next, &reaplist) {
 		clp = list_entry(pos, struct nfs4_client, cl_lru);
 		trace_nfsd_clid_purged(&clp->cl_clientid);
Signature · 1 line
2.9.5
author reviewer
#15 lore raw

RFC PATCH v14 9/10 NFSD: Update laundromat to handle courtesy clients

In reply to Dai Ngo

Add nfs4_anylock_blocker to check if an expired client has any blockers.

Update nfs4_get_client_reaplist to:
. add discarded courtesy client; client marked with CLIENT_EXPIRED,
to reaplist.
. detect if expired client still has state and no blockers then
transit it to courtesy client by setting CLIENT_COURTESY flag and removing the client record.

---
 fs/nfsd/nfs4state.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 98 insertions(+), 2 deletions(-)
fs/nfsd/nfs4state.c +98 -2
index 01c51adf4873..282b8f040540 100644
@@ -5821,24 +5821,120 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
 }
 #endif
 
+static bool
+nfs4_anylock_blocker(struct nfs4_client *clp)
+{
+	int i;
+	struct nfs4_stateowner *so, *tmp;
+	struct nfs4_lockowner *lo;
+	struct nfs4_ol_stateid *stp;
+	struct nfs4_file *nf;
+	struct inode *ino;
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	spin_lock(&clp->cl_lock);
+	for (i = 0; i < OWNER_HASH_SIZE; i++) {
+		/* scan each lock owner */
+		list_for_each_entry_safe(so, tmp, &clp->cl_ownerstr_hashtbl[i],
+				so_strhash) {
+			if (so->so_is_open_owner)
+				continue;
+
+			/* scan lock states of this lock owner */
+			lo = lockowner(so);
+			list_for_each_entry(stp, &lo->lo_owner.so_stateids,
+					st_perstateowner) {
+				nf = stp->st_stid.sc_file;
+				ino = nf->fi_inode;
+				ctx = ino->i_flctx;
+				if (!ctx)
+					continue;
+				/* check each lock belongs to this lock state */
+				list_for_each_entry(fl, &ctx->flc_posix, fl_list) {
+					if (fl->fl_owner != lo)
+						continue;
+					if (!list_empty(&fl->fl_blocked_requests)) {
+						spin_unlock(&clp->cl_lock);
+						return true;
+					}
+				}
+			}
+		}
+	}
+	spin_unlock(&clp->cl_lock);
+	return false;
+}
+
 static void
 nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
 				struct laundry_time *lt)
 {
 	struct list_head *pos, *next;
 	struct nfs4_client *clp;
+	bool cour;
+	struct list_head cslist;
 
 	INIT_LIST_HEAD(reaplist);
+	INIT_LIST_HEAD(&cslist);
 	spin_lock(&nn->client_lock);
 	list_for_each_safe(pos, next, &nn->client_lru) {
 		clp = list_entry(pos, struct nfs4_client, cl_lru);
 		if (!state_expired(lt, clp->cl_time))
 			break;
-		if (mark_client_expired_locked(clp))
+
+		/* client expired */
+		if (!client_has_state(clp)) {
+			if (mark_client_expired_locked(clp))
+				continue;
+			list_add(&clp->cl_lru, reaplist);
+			continue;
+		}
+
+		/* expired client has state */
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags))
+			goto exp_client;
+		cour = test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+		if (cour && ktime_get_boottime_seconds() >=
+				(clp->cl_time + NFSD_COURTESY_CLIENT_TIMEOUT))
+			goto exp_client;
+		if (nfs4_anylock_blocker(clp)) {
+			/* expired client has state and has blocker. */
+exp_client:
+			if (mark_client_expired_locked(clp))
+				continue;
+			list_add(&clp->cl_lru, reaplist);
 			continue;
-		list_add(&clp->cl_lru, reaplist);
+		}
+		/*
+		 * Client expired and has state and has no blockers.
+		 * If there is race condition with blockers, next time
+		 * the laundromat runs it will catch it and expires
+		 * the client.
+		 */
+		if (!cour) {
+			set_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+			list_add(&clp->cl_cs_list, &cslist);
+		}
 	}
 	spin_unlock(&nn->client_lock);
+
+	while (!list_empty(&cslist)) {
+		clp = list_first_entry(&cslist, struct nfs4_client, cl_cs_list);
+		list_del_init(&clp->cl_cs_list);
+		spin_lock(&clp->cl_cs_lock);
+		/*
+		 * Client might have re-connected. Make sure it's
+		 * still in courtesy state before removing its record.
+		 */
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags) ||
+			!test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			spin_unlock(&clp->cl_cs_lock);
+			continue;
+		}
+		spin_unlock(&clp->cl_cs_lock);
+		nfsd4_client_record_remove(clp);
+	}
 }
 
 static time64_t
Signature · 1 line
2.9.5
#16 lore raw

Re: [PATCH RFC v14 09/10] NFSD: Update laundromat to handle courtesy clients

Quoted 7 lines · Dai Ngo

Add nfs4_anylock_blocker to check if an expired client has any blockers.

Quoted 6 lines

Update nfs4_get_client_reaplist to:
. add discarded courtesy client; client marked with CLIENT_EXPIRED,
to reaplist.
. detect if expired client still has state and no blockers then
transit it to courtesy client by setting CLIENT_COURTESY flag and removing the client record.

Thanks for breaking up the v13 patch and adding these nice patch descriptions !

Quoted 20 lines

---
fs/nfsd/nfs4state.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++--

1 file changed, 98 insertions(+), 2 deletions(-)
fs/nfsd/nfs4state.c +7 -0
index 01c51adf4873..282b8f040540 100644
@@ -5821,24 +5821,120 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
 }
 #endif
 
+static bool
+nfs4_anylock_blocker(struct nfs4_client *clp)
+{
+	int i;
+	struct nfs4_stateowner *so, *tmp;
+	struct nfs4_lockowner *lo;
+	struct nfs4_ol_stateid *stp;

fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5840

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+	struct nfs4_file *nf;
+	struct inode *ino;
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	spin_lock(&clp->cl_lock);
+	for (i = 0; i < OWNER_HASH_SIZE; i++) {
+		/* scan each lock owner */
+		list_for_each_entry_safe(so, tmp, &clp->cl_ownerstr_hashtbl[i],
+				so_strhash) {

You're holding the cl_lock and not changing the hash bucket list inside the loop, so list_for_each_entry() is sufficient here. _safe is not needed.

Quoted 11 lines
fs/nfsd/nfs4state.c +11 -0
+			if (so->so_is_open_owner)
+				continue;
+
+			/* scan lock states of this lock owner */
+			lo = lockowner(so);
+			list_for_each_entry(stp, &lo->lo_owner.so_stateids,
+					st_perstateowner) {
+				nf = stp->st_stid.sc_file;
+				ino = nf->fi_inode;
+				ctx = ino->i_flctx;
+				if (!ctx)

fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5861

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+					continue;
+				/* check each lock belongs to this lock state */
+				list_for_each_entry(fl, &ctx->flc_posix, fl_list) {
+					if (fl->fl_owner != lo)
+						continue;
+					if (!list_empty(&fl->fl_blocked_requests)) {
+						spin_unlock(&clp->cl_lock);
+						return true;
+					}
+				}

IIUC fl_blocked_requests is internal to fs/locks.c, so this little piece needs to be moved there. Jeff can give a little more guidance on function naming, etc.

Have a look at Section 6 of Documentation/process/coding-style.rst for some hints about what to do with long functions. This one is long and indented far to the right: that is indication that it should be broken into smaller helpers with names that make it clear what each step does -- no comments needed.

So for legibility, "check each lock belongs..." is a function that should be moved to fs/locks.c, as I mentioned above; and "scan lock state" can be its own function that calls "check each lock belongs".

I'm not sure what the comment "check each lock belongs to this lock state" means. I thought we were looking at blocked lists, not locks, in this search?

Quoted 23 lines
fs/nfsd/nfs4state.c +7 -0
+			}
+		}
+	}
+	spin_unlock(&clp->cl_lock);
+	return false;
+}
+

static void
nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
struct laundry_time *lt)

{
	struct list_head *pos, *next;
	struct nfs4_client *clp;
fs/nfsd/nfs4state.c +2 -0
+	bool cour;
+	struct list_head cslist;
	INIT_LIST_HEAD(reaplist);
fs/nfsd/nfs4state.c +1 -0
+	INIT_LIST_HEAD(&cslist);
	spin_lock(&nn->client_lock);
	list_for_each_safe(pos, next, &nn->client_lru) {
		clp = list_entry(pos, struct nfs4_client, cl_lru);
		if (!state_expired(lt, clp->cl_time))
			break;

fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5891

Quoted 8 lines
fs/nfsd/nfs4state.c +7 -1
-		if (mark_client_expired_locked(clp))
+
+		/* client expired */
+		if (!client_has_state(clp)) {
+			if (mark_client_expired_locked(clp))
+				continue;
+			list_add(&clp->cl_lru, reaplist);
+			continue;

The above looks like the same thing that is done below at the "exp_client:" label. Should you replace this with

	if (!client_has_state(clp))
		goto exp_client;

?

Then I think the one-line comments would be unnecessary. The code would be self-explanatory.

Quoted 16 lines
fs/nfsd/nfs4state.c +15 -0
+		}
+
+		/* expired client has state */
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags))
+			goto exp_client;
+		cour = test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+		if (cour && ktime_get_boottime_seconds() >=
+				(clp->cl_time + NFSD_COURTESY_CLIENT_TIMEOUT))
+			goto exp_client;
+		if (nfs4_anylock_blocker(clp)) {
+			/* expired client has state and has blocker. */
+exp_client:
+			if (mark_client_expired_locked(clp))
+				continue;
+			list_add(&clp->cl_lru, reaplist);
			continue;

fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5913

Quoted 7 lines
fs/nfsd/nfs4state.c +6 -1
-		list_add(&clp->cl_lru, reaplist);
+		}
+		/*
+		 * Client expired and has state and has no blockers.
+		 * If there is race condition with blockers, next time
+		 * the laundromat runs it will catch it and expires
+		 * the client.

This comment is still not clear. "Client expired ... it will catch and expires the client." Do you mean "destroys the client" ?

fs/nfsd/nfs4state.c static void nfsd4_ssc_expire_umount(struct nfsd_net *nn) · line 5918

Quoted 5 lines
fs/nfsd/nfs4state.c +5 -0
+		 */
+		if (!cour) {
+			set_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags);
+			list_add(&clp->cl_cs_list, &cslist);
+		}

cl_cs_lock is taken in the loop below, but don't you also need to hold that lock when manipulating the courtesy state bits in the code above?

Quoted 21 lines
	}
	spin_unlock(&nn->client_lock);
fs/nfsd/nfs4state.c +17 -0
+
+	while (!list_empty(&cslist)) {
+		clp = list_first_entry(&cslist, struct nfs4_client, cl_cs_list);
+		list_del_init(&clp->cl_cs_list);
+		spin_lock(&clp->cl_cs_lock);
+		/*
+		 * Client might have re-connected. Make sure it's
+		 * still in courtesy state before removing its record.
+		 */
+		if (test_bit(NFSD4_CLIENT_EXPIRED, &clp->cl_flags) ||
+			!test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags)) {
+			spin_unlock(&clp->cl_cs_lock);
+			continue;
+		}
+		spin_unlock(&clp->cl_cs_lock);
+		nfsd4_client_record_remove(clp);
+	}

}

Quoted 4 lines

static time64_t

Signature · 1 line
2.9.5

--
Chuck Lever

author reviewer
#17 lore raw

RFC PATCH v14 10/10 NFSD: Show state of courtesy clients in client info

In reply to Dai Ngo

Update client_info_show to show state of courtesy client and number of seconds since last renew.

---
 fs/nfsd/nfs4state.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/nfsd/nfs4state.c +4 -0
index 282b8f040540..b2909ae816a8 100644
@@ -2435,6 +2435,10 @@ static int client_info_show(struct seq_file *m, void *v)
 		seq_puts(m, "status: confirmed\n");
 	else
 		seq_puts(m, "status: unconfirmed\n");
+	seq_printf(m, "courtesy client: %s\n",
+		test_bit(NFSD4_CLIENT_COURTESY, &clp->cl_flags) ? "yes" : "no");
+	seq_printf(m, "seconds from last renew: %lld\n",
+		ktime_get_boottime_seconds() - clp->cl_time);
 	seq_printf(m, "name: ");
 	seq_quote_mem(m, clp->cl_name.data, clp->cl_name.len);
 	seq_printf(m, "\nminor version: %d\n", clp->cl_minorversion);
Signature · 1 line
2.9.5