Skip to contentSkip to conversation

NFSD: add support for NFSv4 write delegation

PATCH Superseded Dai Ngo 4 participants 17 messages May 20, 2023 – May 22, 2023

Subsystems (4)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Dai Ngo

author reviewer
#0 lore raw

v4 cover NFSD: add support for NFSv4 write delegation

NFSD: add support for NFSv4 write delegation

The NFSv4 server currently supports read delegation using VFS lease which is implemented using file_lock.

This patch series add write delegation support for NFSv4 server by:

. remove the check for F_WRLCK in generic_add_lease to allow
file_lock to be used for write delegation.

. grant write delegation for OPEN with NFS4_SHARE_ACCESS_WRITE
if there is no conflict with other OPENs.

Write delegation conflict with another OPEN, REMOVE, RENAME and SETATTR are handled the same as read delegation using notify_change, try_break_deleg.

Changes since v1:

[PATCH 3/4] NFSD: add supports for CB_GETATTR callback
- remove WARN_ON_ONCE from encode_bitmap4
- replace decode_bitmap4 with xdr_stream_decode_uint32_array
- replace xdr_inline_decode and xdr_decode_hyper in decode_cb_getattr with xdr_stream_decode_u64. Also remove the un-needed likely().
- modify signature of encode_cb_getattr4args to take pointer to nfs4_cb_fattr
- replace decode_attr_length with xdr_stream_decode_u32
- rename decode_cb_getattr to decode_cb_fattr4
- fold the initialization of cb_cinfo and cb_fsize into decode_cb_fattr4
- rename ncf_cb_cinfo to ncf_cb_change to avoid confusion of cindo usage in fs/nfsd/nfs4xdr.c
- correct NFS4_dec_cb_getattr_sz and update size description

[PATCH 4/4] NFSD: handle GETATTR conflict with write delegation
- change nfs4_handle_wrdeleg_conflict returns __be32 to fix test robot
- change ncf_cb_cinfo to ncf_cb_change to avoid confusion of cindo usage in fs/nfsd/nfs4xdr.c

Changes since v2:

[PATCH 2/4] NFSD: enable support for write delegation
- rename 'deleg' to 'dl_type' in nfs4_set_delegation
- remove 'wdeleg' in nfs4_open_delegation

- drop [PATCH 3/4] NFSD: add supports for CB_GETATTR callback and [PATCH 4/4] NFSD: handle GETATTR conflict with write delegation for futher clarification of the benefits of these patches

Changes since v3:

- recall write delegation when there is GETATTR from 2nd client
- add trace point to track when write delegation is granted

author reviewer
#1 lore raw

PATCH v4 1/4 locks: allow support for write delegation

Remove the check for F_WRLCK in generic_add_lease to allow file_lock to be used for write delegation.

First consumer is NFSD.

---
 fs/locks.c | 7 -------
 1 file changed, 7 deletions(-)
fs/locks.c +0 -7
index df8b26a42524..08fb0b4fd4f8 100644
@@ -1729,13 +1729,6 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr
 	if (is_deleg && !inode_trylock(inode))
 		return -EAGAIN;
 
-	if (is_deleg && arg == F_WRLCK) {
-		/* Write delegations are not currently supported: */
-		inode_unlock(inode);
-		WARN_ON_ONCE(1);
-		return -EINVAL;
-	}
-
 	percpu_down_read(&file_rwsem);
 	spin_lock(&ctx->flc_lock);
 	time_out_leases(inode, &dispose);
Signature · 1 line
2.9.5
maintainer Reviewed-by on v4 1/4
#2 lore raw

Re: [PATCH v4 1/4] locks: allow support for write delegation

Quoted 18 lines · Dai Ngo

Remove the check for F_WRLCK in generic_add_lease to allow file_lock to be used for write delegation.

First consumer is NFSD.

---
 fs/locks.c | 7 -------
 1 file changed, 7 deletions(-)
fs/locks.c +0 -0
index df8b26a42524..08fb0b4fd4f8 100644
@@ -1729,13 +1729,6 @@ generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr
 	if (is_deleg && !inode_trylock(inode))
 		return -EAGAIN;
 

fs/locks.c generic_add_lease(struct file *filp, long arg, struct file_lock **flp, void **pr · line 1734

Quoted 10 lines
fs/locks.c +0 -7
-	if (is_deleg && arg == F_WRLCK) {
-		/* Write delegations are not currently supported: */
-		inode_unlock(inode);
-		WARN_ON_ONCE(1);
-		return -EINVAL;
-	}
-
 	percpu_down_read(&file_rwsem);
 	spin_lock(&ctx->flc_lock);
 	time_out_leases(inode, &dispose);
author reviewer
#3 lore raw

PATCH v4 2/4 NFSD: enable support for write delegation

In reply to Dai Ngo

This patch grants write delegation for OPEN with NFS4_SHARE_ACCESS_WRITE if there is no conflict with other OPENs.

Write delegation conflict with another OPEN, REMOVE, RENAME and SETATTR are handled the same as read delegation using notify_change, try_break_deleg.

---
 fs/nfsd/nfs4state.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
fs/nfsd/nfs4state.c +16 -8
index 6e61fa3acaf1..3f98b7485c72 100644
@@ -1144,7 +1144,7 @@ static void block_delegations(struct knfsd_fh *fh)
 
 static struct nfs4_delegation *
 alloc_init_deleg(struct nfs4_client *clp, struct nfs4_file *fp,
-		 struct nfs4_clnt_odstate *odstate)
+		struct nfs4_clnt_odstate *odstate, u32 dl_type)
 {
 	struct nfs4_delegation *dp;
 	long n;
@@ -1170,7 +1170,7 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_file *fp,
 	INIT_LIST_HEAD(&dp->dl_recall_lru);
 	dp->dl_clnt_odstate = odstate;
 	get_clnt_odstate(odstate);
-	dp->dl_type = NFS4_OPEN_DELEGATE_READ;
+	dp->dl_type = dl_type;
 	dp->dl_retries = 1;
 	dp->dl_recalled = false;
 	nfsd4_init_cb(&dp->dl_recall, dp->dl_stid.sc_client,
@@ -5451,6 +5451,7 @@ nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 	struct nfs4_delegation *dp;
 	struct nfsd_file *nf;
 	struct file_lock *fl;
+	u32 dl_type;
 
 	/*
 	 * The fi_had_conflict and nfs_get_existing_delegation checks
@@ -5460,7 +5461,13 @@ nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 	if (fp->fi_had_conflict)
 		return ERR_PTR(-EAGAIN);
 
-	nf = find_readable_file(fp);
+	if (open->op_share_access & NFS4_SHARE_ACCESS_WRITE) {
+		nf = find_writeable_file(fp);
+		dl_type = NFS4_OPEN_DELEGATE_WRITE;
+	} else {
+		nf = find_readable_file(fp);
+		dl_type = NFS4_OPEN_DELEGATE_READ;
+	}
 	if (!nf) {
 		/*
 		 * We probably could attempt another open and get a read
@@ -5491,11 +5498,11 @@ nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 		return ERR_PTR(status);
 
 	status = -ENOMEM;
-	dp = alloc_init_deleg(clp, fp, odstate);
+	dp = alloc_init_deleg(clp, fp, odstate, dl_type);
 	if (!dp)
 		goto out_delegees;
 
-	fl = nfs4_alloc_init_lease(dp, NFS4_OPEN_DELEGATE_READ);
+	fl = nfs4_alloc_init_lease(dp, dl_type);
 	if (!fl)
 		goto out_clnt_odstate;
 
@@ -5590,8 +5597,6 @@ nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 		case NFS4_OPEN_CLAIM_PREVIOUS:
 			if (!cb_up)
 				open->op_recall = 1;
-			if (open->op_delegate_type != NFS4_OPEN_DELEGATE_READ)
-				goto out_no_deleg;
 			break;
 		case NFS4_OPEN_CLAIM_NULL:
 			parent = currentfh;
@@ -5617,7 +5622,10 @@ nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 	memcpy(&open->op_delegate_stateid, &dp->dl_stid.sc_stateid, sizeof(dp->dl_stid.sc_stateid));
 
 	trace_nfsd_deleg_read(&dp->dl_stid.sc_stateid);
-	open->op_delegate_type = NFS4_OPEN_DELEGATE_READ;
+	if (open->op_share_access & NFS4_SHARE_ACCESS_WRITE)
+		open->op_delegate_type = NFS4_OPEN_DELEGATE_WRITE;
+	else
+		open->op_delegate_type = NFS4_OPEN_DELEGATE_READ;
 	nfs4_put_stid(&dp->dl_stid);
 	return;
 out_no_deleg:
Signature · 1 line
2.9.5
author reviewer
#4 lore raw

PATCH v4 3/4 NFSD: handle GETATTR conflict with write delegation

In reply to Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
 fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +45 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
 	return nfserr_resource;
 }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
+	dp = fl->fl_owner;
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
 /*
  * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
  * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
 		if (status)
 			goto out;
 	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
 	err = vfs_getattr(&path, &stat,
 			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
Signature · 1 line
2.9.5
#5 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 52 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +30 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
 return nfserr_resource;
 }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+ struct file_lock_context *ctx;
+ struct file_lock *fl;
+
+ ctx = locks_inode_context(inode);
+ if (!ctx)
+ return NULL;
+ spin_lock(&ctx->flc_lock);
+ if (!list_empty(&ctx->flc_lease)) {
+ fl = list_first_entry(&ctx->flc_lease,
+ struct file_lock, fl_list);
+ if (fl->fl_type == F_WRLCK) {
+ spin_unlock(&ctx->flc_lock);
+ return fl;
+ }
+ }
+ spin_unlock(&ctx->flc_lock);
+ return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+ __be32 status;
+ struct file_lock *fl;
+ struct nfs4_delegation *dp;
+
+ fl = nfs4_wrdeleg_filelock(rqstp, inode);

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2962

Quoted 10 lines
fs/nfsd/nfs4xdr.c +10 -0
+ if (!fl)
+ return 0;
+ dp = fl->fl_owner;
+ if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+ return 0;
+ refcount_inc(&dp->dl_stid.sc_count);
+ status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+ return status;
+}
+

fs/nfsd/nfs4state.c seems more appropriate for these. I'll move them as I apply this patch.

Quoted 18 lines
fs/nfsd/nfs4xdr.c +5 -0
 /*
 * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
 * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
 if (status)
 goto out;
 }
+ if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+ status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+ if (status)
+ goto out;
+ }
 
 err = vfs_getattr(&path, &stat,
 STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
Signature · 1 line
2.9.5

--
Chuck Lever

maintainer
#6 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

In reply to Dai Ngo
Quoted 41 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
 fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +23 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
 	return nfserr_resource;
 }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2955

Quoted 10 lines
fs/nfsd/nfs4xdr.c +10 -0
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
+	dp = fl->fl_owner;

One problem here is that you don't know whether the owner was set by nfsd. This owner could be a struct file from a userland lease holder, and that that point it's not safe to dereference it below like you are.

The q&d way we check for this in other places is to validate that the fl_lmops is correct. In this case you'd want to make sure it's set to &nfsd_lease_mng_ops.

Beyond that, you also don't know whether that owner or file_lock still _exists_ after you drop the flc_lock. You need to either do these checks while holding that lock, or take a reference to the owner before you start dereferencing things.

Probably, you're better off here just doing it all under the flc_lock.

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 3 lines
fs/nfsd/nfs4xdr.c +3 -0
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);

Another problem: the sc_count might already have gone to zero here. You don't already hold a reference to dl_stid at this point, so you can't just increment it without taking the cl_lock for that client.

You may be able to do this safely with refcount_inc_not_zero, and just ignore the delegation if it's already at zero.

Quoted 19 lines
fs/nfsd/nfs4xdr.c +9 -0
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
 /*
  * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
  * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
 		if (status)
 			goto out;
 	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
 	err = vfs_getattr(&path, &stat,
 			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
Signature · 1 line
Jeff Layton <[email protected]>
author reviewer
#7 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 56 lines · Jeff Layton
Quoted 41 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
  fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +23 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
  	return nfserr_resource;
  }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2955

Quoted 10 lines
fs/nfsd/nfs4xdr.c +10 -0
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
+	dp = fl->fl_owner;

One problem here is that you don't know whether the owner was set by nfsd. This owner could be a struct file from a userland lease holder, and that that point it's not safe to dereference it below like you are.

Quoted 3 lines

The q&d way we check for this in other places is to validate that the fl_lmops is correct. In this case you'd want to make sure it's set to &nfsd_lease_mng_ops.

Thanks Jeff, fix in v5.

Quoted 7 lines

Beyond that, you also don't know whether that owner or file_lock still _exists_ after you drop the flc_lock. You need to either do these checks while holding that lock, or take a reference to the owner before you start dereferencing things.

Probably, you're better off here just doing it all under the flc_lock.

fix in v5.

Quoted 10 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 4 lines
fs/nfsd/nfs4xdr.c +3 -0
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);

Another problem: the sc_count might already have gone to zero here. You don't already hold a reference to dl_stid at this point, so you can't just increment it without taking the cl_lock for that client.

You may be able to do this safely with refcount_inc_not_zero, and just ignore the delegation if it's already at zero.

Fix in v5.

Chuck, I can do v5 to address feedback from you and Jeff.

Thanks,
-Dai

Quoted 20 lines
Quoted 20 lines
fs/nfsd/nfs4xdr.c +9 -0
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
  /*
   * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
   * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
  		if (status)
  			goto out;
  	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
  	err = vfs_getattr(&path, &stat,
  			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
maintainer
#8 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

In reply to Dai Ngo
Quoted 44 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
 fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +26 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
 	return nfserr_resource;
 }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 10 lines
fs/nfsd/nfs4xdr.c +10 -0
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
+	dp = fl->fl_owner;
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all? AFAICT, you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

Quoted 19 lines
fs/nfsd/nfs4xdr.c +9 -0
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
 /*
  * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
  * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
 		if (status)
 			goto out;
 	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
 	err = vfs_getattr(&path, &stat,
 			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#9 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 51 lines · Jeff Layton
Quoted 50 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
 fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +32 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
 	return nfserr_resource;
 }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
Quoted 8 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 4 lines
fs/nfsd/nfs4xdr.c +4 -0
+	dp = fl->fl_owner;
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all? AFAICT, you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

I had the same thought, but I assumed I was missing something.

Quoted 22 lines
Quoted 19 lines
fs/nfsd/nfs4xdr.c +9 -0
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
 /*
  * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
  * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
 		if (status)
 			goto out;
 	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
 	err = vfs_getattr(&path, &stat,
 			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
Signature · 1 line
Jeff Layton <[email protected]>
author reviewer
#10 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

In reply to Jeff Layton
Quoted 48 lines · Jeff Layton
Quoted 47 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
  fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +29 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
  	return nfserr_resource;
  }
 
+static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+	struct file_lock_context *ctx;
+	struct file_lock *fl;
+
+	ctx = locks_inode_context(inode);
+	if (!ctx)
+		return NULL;
+	spin_lock(&ctx->flc_lock);
+	if (!list_empty(&ctx->flc_lease)) {
+		fl = list_first_entry(&ctx->flc_lease,
+					struct file_lock, fl_list);
+		if (fl->fl_type == F_WRLCK) {
+			spin_unlock(&ctx->flc_lock);
+			return fl;
+		}
+	}
+	spin_unlock(&ctx->flc_lock);
+	return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+	__be32 status;
+	struct file_lock *fl;
+	struct nfs4_delegation *dp;
+
Quoted 8 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 7 lines
fs/nfsd/nfs4xdr.c +7 -0
+	fl = nfs4_wrdeleg_filelock(rqstp, inode);
+	if (!fl)
+		return 0;
+	dp = fl->fl_owner;
+	if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+		return 0;
+	refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all?

This is same as in nfsd_break_one_deleg and revoke_delegation. I think it is to prevent the delegation to be freed while delegation is being recalled.

Quoted 3 lines

AFAICT,
you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

In v2, the sc_count is decrement by nfs4_put_stid. I forgot to do that in V4. I'll add it back in v5.

Thanks,
-Dai

Quoted 20 lines
Quoted 20 lines
fs/nfsd/nfs4xdr.c +9 -0
+	status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+	return status;
+}
+
  /*
   * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
   * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
  		if (status)
  			goto out;
  	}
+	if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+		status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+		if (status)
+			goto out;
+	}
 
  	err = vfs_getattr(&path, &stat,
  			  STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
author reviewer
#11 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 64 lines · On 5/21/23 7:56 PM, [email protected] wrote:
Quoted 49 lines · Jeff Layton
Quoted 48 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
  fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +0 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, 
 u32 bmval0, u32 bmval1, u32 bmval2)
       return nfserr_resource;
   }

  +static struct file_lock *

fs/nfsd/nfs4xdr.c +23 -0
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+    struct file_lock_context *ctx;
+    struct file_lock *fl;
+
+    ctx = locks_inode_context(inode);
+    if (!ctx)
+        return NULL;
+    spin_lock(&ctx->flc_lock);
+    if (!list_empty(&ctx->flc_lease)) {
+        fl = list_first_entry(&ctx->flc_lease,
+                    struct file_lock, fl_list);
+        if (fl->fl_type == F_WRLCK) {
+            spin_unlock(&ctx->flc_lock);
+            return fl;
+        }
+    }
+    spin_unlock(&ctx->flc_lock);
+    return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode 

*inode)

fs/nfsd/nfs4xdr.c +5 -0
+{
+    __be32 status;
+    struct file_lock *fl;
+    struct nfs4_delegation *dp;
+
Quoted 8 lines
Quoted 7 lines

+    fl = nfs4_wrdeleg_filelock(rqstp, inode);
+    if (!fl)
+        return 0;
+    dp = fl->fl_owner;
+    if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker)) +        return 0;
+    refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all?

This is same as in nfsd_break_one_deleg and revoke_delegation. I think it is to prevent the delegation to be freed while delegation is being recalled.

Quoted 6 lines
Quoted 3 lines

  AFAICT,
you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

In v2, the sc_count is decrement by nfs4_put_stid. I forgot to do that in V4. I'll add it back in v5.

Actually the refcount is decremented after the CB_RECALL is done by nfs4_put_stid in nfsd4_cb_recall_release. So we don't have to decrement it here. This is to prevent the delegation to be free while the recall is being sent.

-Dai

Quoted 24 lines

Thanks,
-Dai

Quoted 20 lines
Quoted 20 lines

+    status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ)); +    return status;

fs/nfsd/nfs4xdr.c +2 -0
+}
+

  /*
   * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
   * ourselves.

fs/nfsd/nfs4xdr.c +5 -0
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, 
 struct svc_fh *fhp,
           if (status)
               goto out;
       }
+    if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+        status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+        if (status)
+            goto out;
+    }
         err = vfs_getattr(&path, &stat,
                 STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,
#12 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 72 lines · Dai Ngo
Quoted 66 lines · On 5/21/23 7:56 PM, [email protected] wrote:
Quoted 47 lines · Jeff Layton
Quoted 46 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
  fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +28 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2)
      return nfserr_resource;
  }
  +static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+    struct file_lock_context *ctx;
+    struct file_lock *fl;
+
+    ctx = locks_inode_context(inode);
+    if (!ctx)
+        return NULL;
+    spin_lock(&ctx->flc_lock);
+    if (!list_empty(&ctx->flc_lease)) {
+        fl = list_first_entry(&ctx->flc_lease,
+                    struct file_lock, fl_list);
+        if (fl->fl_type == F_WRLCK) {
+            spin_unlock(&ctx->flc_lock);
+            return fl;
+        }
+    }
+    spin_unlock(&ctx->flc_lock);
+    return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *inode)
+{
+    __be32 status;
+    struct file_lock *fl;
+    struct nfs4_delegation *dp;
+
Quoted 8 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 7 lines
fs/nfsd/nfs4xdr.c +7 -0
+    fl = nfs4_wrdeleg_filelock(rqstp, inode);
+    if (!fl)
+        return 0;
+    dp = fl->fl_owner;
+    if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+        return 0;
+    refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all?

This is same as in nfsd_break_one_deleg and revoke_delegation. I think it is to prevent the delegation to be freed while delegation is being recalled.

Quoted 3 lines

AFAICT,
you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

Quoted 7 lines
Quoted 2 lines

In v2, the sc_count is decrement by nfs4_put_stid. I forgot to do that in V4. I'll add it back in v5.

Actually the refcount is decremented after the CB_RECALL is done by nfs4_put_stid in nfsd4_cb_recall_release. So we don't have to decrement it here. This is to prevent the delegation to be free while the recall is being sent.

For v5, please add this good information to the documenting comment for this function.

Quoted 17 lines

-Dai

Quoted 15 lines

Thanks,
-Dai

Quoted 11 lines
Quoted 11 lines
fs/nfsd/nfs4xdr.c +4 -0
+    status = nfserrno(nfsd_open_break_lease(inode, NFSD_MAY_READ));
+    return status;
+}
+
  /*
   * Note: @fhp can be NULL; in this case, we might have to compose the filehandle
   * ourselves.
@@ -2966,6 +3006,11 @@ nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp,
          if (status)
              goto out;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp, · line 3016

Quoted 8 lines
fs/nfsd/nfs4xdr.c +5 -0
      }
+    if (bmval0 & (FATTR4_WORD0_CHANGE | FATTR4_WORD0_SIZE)) {
+        status = nfs4_handle_wrdeleg_conflict(rqstp, d_inode(dentry));
+        if (status)
+            goto out;
+    }
        err = vfs_getattr(&path, &stat,
                STATX_BASIC_STATS | STATX_BTIME | STATX_CHANGE_COOKIE,

--
Chuck Lever

maintainer
#13 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

In reply to [email protected]
Quoted 31 lines · On Sun, 2023-05-21 at 20:56 -0700, [email protected] wrote:
Quoted 30 lines · On 5/21/23 7:56 PM, [email protected] wrote:
Quoted 28 lines · Jeff Layton
Quoted 27 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
  fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +0 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr, 
 u32 bmval0, u32 bmval1, u32 bmval2)
       return nfserr_resource;
   }

  +static struct file_lock *

fs/nfsd/nfs4xdr.c +8 -0
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+    struct file_lock_context *ctx;
+    struct file_lock *fl;
+
+    ctx = locks_inode_context(inode);
+    if (!ctx)
+        return NULL;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    spin_lock(&ctx->flc_lock);
+    if (!list_empty(&ctx->flc_lease)) {
+        fl = list_first_entry(&ctx->flc_lease,
+                    struct file_lock, fl_list);
+        if (fl->fl_type == F_WRLCK) {
+            spin_unlock(&ctx->flc_lock);
+            return fl;
+        }

One more issue here too. FL_LAYOUT file_locks live on this list too. They shouldn't conflict with leases or delegations, so you probably just want to skip them.

Longer term, I wonder if we ought to add a new list in the file_lock_context for layouts? There's no reason to keep them all on the same list.

Quoted 30 lines
Quoted 30 lines
Quoted 13 lines
Quoted 13 lines

+    }
+    spin_unlock(&ctx->flc_lock);
+    return NULL;

fs/nfsd/nfs4xdr.c +4 -0
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode 

*inode)

fs/nfsd/nfs4xdr.c +5 -0
+{
+    __be32 status;
+    struct file_lock *fl;
+    struct nfs4_delegation *dp;
+
Quoted 8 lines
Quoted 7 lines

+    fl = nfs4_wrdeleg_filelock(rqstp, inode);
+    if (!fl)
+        return 0;
+    dp = fl->fl_owner;
+    if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker)) +        return 0;
+    refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all?

This is same as in nfsd_break_one_deleg and revoke_delegation. I think it is to prevent the delegation to be freed while delegation is being recalled.

Quoted 3 lines

  AFAICT,
you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

Quoted 8 lines
Quoted 2 lines

In v2, the sc_count is decrement by nfs4_put_stid. I forgot to do that in V4. I'll add it back in v5.

Actually the refcount is decremented after the CB_RECALL is done by nfs4_put_stid in nfsd4_cb_recall_release. So we don't have to decrement it here. This is to prevent the delegation to be free while the recall is being sent.

That's the put for the increment in nfsd_break_one_deleg.

You don't need to take an extra reference to a delegation to call nfsd_open_break_lease. You might not even know which delegation is being broken. There could even be more than one, after all.

I think that extra refcount_inc is likely to cause a leak.

Signature · 1 line
Jeff Layton <[email protected]>
author reviewer
#14 lore raw

Re: [PATCH v4 3/4] NFSD: handle GETATTR conflict with write delegation

Quoted 34 lines · Jeff Layton
Quoted 33 lines · On Sun, 2023-05-21 at 20:56 -0700, [email protected] wrote:
Quoted 32 lines · On 5/21/23 7:56 PM, [email protected] wrote:
Quoted 31 lines · Jeff Layton
Quoted 30 lines · Dai Ngo

If the GETATTR request on a file that has write delegation in effect and the request attributes include the change info and size attribute then the write delegation is recalled and NFS4ERR_DELAY is returned for the GETATTR.

---
   fs/nfsd/nfs4xdr.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
   1 file changed, 45 insertions(+)
fs/nfsd/nfs4xdr.c +11 -0
index 76db2fe29624..e069b970f136 100644
@@ -2920,6 +2920,46 @@ nfsd4_encode_bitmap(struct xdr_stream *xdr,
 u32 bmval0, u32 bmval1, u32 bmval2)
       return nfserr_resource;
   }
   +static struct file_lock *
+nfs4_wrdeleg_filelock(struct svc_rqst *rqstp, struct inode *inode)
+{
+    struct file_lock_context *ctx;
+    struct file_lock *fl;
+
+    ctx = locks_inode_context(inode);
+    if (!ctx)
+        return NULL;
+    spin_lock(&ctx->flc_lock);
+    if (!list_empty(&ctx->flc_lease)) {
+        fl = list_first_entry(&ctx->flc_lease,
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2939

Quoted 5 lines
fs/nfsd/nfs4xdr.c +5 -0
+                    struct file_lock, fl_list);
+        if (fl->fl_type == F_WRLCK) {
+            spin_unlock(&ctx->flc_lock);
+            return fl;
+        }

One more issue here too. FL_LAYOUT file_locks live on this list too. They shouldn't conflict with leases or delegations, so you probably just want to skip them.

oh ok, that means we have to scan the list and skip the FL_LAYOUT file_locks.

Quoted 4 lines

Longer term, I wonder if we ought to add a new list in the file_lock_context for layouts? There's no reason to keep them all on the same list.

Yes, that would be nice.

Thanks Jeff,
-Dai

Quoted 43 lines
Quoted 29 lines
Quoted 29 lines
Quoted 14 lines
Quoted 14 lines
fs/nfsd/nfs4xdr.c +12 -0
+    }
+    spin_unlock(&ctx->flc_lock);
+    return NULL;
+}
+
+static __be32
+nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode
 *inode)
+{
+    __be32 status;
+    struct file_lock *fl;
+    struct nfs4_delegation *dp;
+
Quoted 8 lines

fs/nfsd/nfs4xdr.c nfsd4_encode_bitmap(struct xdr_stream *xdr, u32 bmval0, u32 bmval1, u32 bmval2) · line 2958

Quoted 7 lines
fs/nfsd/nfs4xdr.c +7 -0
+    fl = nfs4_wrdeleg_filelock(rqstp, inode);
+    if (!fl)
+        return 0;
+    dp = fl->fl_owner;
+    if (dp->dl_recall.cb_clp == *(rqstp->rq_lease_breaker))
+        return 0;
+    refcount_inc(&dp->dl_stid.sc_count);

Another question: Why are you taking a reference here at all?

This is same as in nfsd_break_one_deleg and revoke_delegation. I think it is to prevent the delegation to be freed while delegation is being recalled.

Quoted 3 lines

  AFAICT,
you don't even look at the delegation again after that point, so it's not clear to me who's responsible for putting that reference.

Quoted 7 lines
Quoted 2 lines

In v2, the sc_count is decrement by nfs4_put_stid. I forgot to do that in V4. I'll add it back in v5.

Actually the refcount is decremented after the CB_RECALL is done by nfs4_put_stid in nfsd4_cb_recall_release. So we don't have to decrement it here. This is to prevent the delegation to be free while the recall is being sent.

That's the put for the increment in nfsd_break_one_deleg.

You don't need to take an extra reference to a delegation to call nfsd_open_break_lease. You might not even know which delegation is being broken. There could even be more than one, after all.

I think that extra refcount_inc is likely to cause a leak.

author reviewer
#15 lore raw

PATCH v4 4/4 NFSD: add trace point to track when write delegation is granted

In reply to Dai Ngo

Add trace point to track whether read or write delegation is granted.

---
 fs/nfsd/nfs4state.c | 8 +++++---
 fs/nfsd/trace.h     | 1 +
 2 files changed, 6 insertions(+), 3 deletions(-)
fs/nfsd/nfs4state.c +5 -3
index 3f98b7485c72..b90b74a5e66e 100644
@@ -5621,11 +5621,13 @@ nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
 
 	memcpy(&open->op_delegate_stateid, &dp->dl_stid.sc_stateid, sizeof(dp->dl_stid.sc_stateid));
 
-	trace_nfsd_deleg_read(&dp->dl_stid.sc_stateid);
-	if (open->op_share_access & NFS4_SHARE_ACCESS_WRITE)
+	if (open->op_share_access & NFS4_SHARE_ACCESS_WRITE) {
 		open->op_delegate_type = NFS4_OPEN_DELEGATE_WRITE;
-	else
+		trace_nfsd_deleg_write(&dp->dl_stid.sc_stateid);
+	} else {
 		open->op_delegate_type = NFS4_OPEN_DELEGATE_READ;
+		trace_nfsd_deleg_read(&dp->dl_stid.sc_stateid);
+	}
 	nfs4_put_stid(&dp->dl_stid);
 	return;
 out_no_deleg:
fs/nfsd/trace.h +1 -0
index 4183819ea082..a14cf8684255 100644
@@ -607,6 +607,7 @@ DEFINE_STATEID_EVENT(layout_recall_release);
 
 DEFINE_STATEID_EVENT(open);
 DEFINE_STATEID_EVENT(deleg_read);
+DEFINE_STATEID_EVENT(deleg_write);
 DEFINE_STATEID_EVENT(deleg_return);
 DEFINE_STATEID_EVENT(deleg_recall);
 
Signature · 1 line
2.9.5
#16 lore raw
In reply to Dai Ngo
Quoted 46 lines · Dai Ngo

NFSD: add support for NFSv4 write delegation

The NFSv4 server currently supports read delegation using VFS lease which is implemented using file_lock.

This patch series add write delegation support for NFSv4 server by:

. remove the check for F_WRLCK in generic_add_lease to allow
file_lock to be used for write delegation.

. grant write delegation for OPEN with NFS4_SHARE_ACCESS_WRITE
if there is no conflict with other OPENs.

Write delegation conflict with another OPEN, REMOVE, RENAME and SETATTR are handled the same as read delegation using notify_change, try_break_deleg.

Changes since v1:

[PATCH 3/4] NFSD: add supports for CB_GETATTR callback
- remove WARN_ON_ONCE from encode_bitmap4
- replace decode_bitmap4 with xdr_stream_decode_uint32_array
- replace xdr_inline_decode and xdr_decode_hyper in decode_cb_getattr with xdr_stream_decode_u64. Also remove the un-needed likely().
- modify signature of encode_cb_getattr4args to take pointer to nfs4_cb_fattr
- replace decode_attr_length with xdr_stream_decode_u32
- rename decode_cb_getattr to decode_cb_fattr4
- fold the initialization of cb_cinfo and cb_fsize into decode_cb_fattr4
- rename ncf_cb_cinfo to ncf_cb_change to avoid confusion of cindo usage in fs/nfsd/nfs4xdr.c
- correct NFS4_dec_cb_getattr_sz and update size description

[PATCH 4/4] NFSD: handle GETATTR conflict with write delegation
- change nfs4_handle_wrdeleg_conflict returns __be32 to fix test robot
- change ncf_cb_cinfo to ncf_cb_change to avoid confusion of cindo usage in fs/nfsd/nfs4xdr.c

Changes since v2:

[PATCH 2/4] NFSD: enable support for write delegation
- rename 'deleg' to 'dl_type' in nfs4_set_delegation
- remove 'wdeleg' in nfs4_open_delegation

Quoted 9 lines

- drop [PATCH 3/4] NFSD: add supports for CB_GETATTR callback and [PATCH 4/4] NFSD: handle GETATTR conflict with write delegation for futher clarification of the benefits of these patches

Changes since v3:

- recall write delegation when there is GETATTR from 2nd client
- add trace point to track when write delegation is granted

I'll apply this series to nfsd-next. There are a handful of changes I'd like to make (with permission):

- Squash 4/4 into 2/4
- Apply 1/4 last instead of first
- Add Jeff's Reviewed-by at least to 1/4 and 2/4

3/4 gives us a platform for measuring recalls triggered by GETATTR. It might not make any difference if we add a new counter for that -- but a tracepoint could also do it without altering the kernel/userspace API. I'm still pondering it.

--
Chuck Lever