Skip to contentSkip to conversation

nfsd: close potential race between open and delegation

PATCH Superseded Jeff Layton 3 participants 9 messages Jul 14, 2022 – Jul 25, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

cover nfsd: close potential race between open and delegation

This is a respin of the patchset that I sent earlier today. I hit a deadlock with that one because of the ambiguous locking.

This series is based on top of Neil's set entitled:

[PATCH 0/8] NFSD: clean up locking.

His patchset makes the locking in the nfsd4_open codepath much more consistent, and this becomes a lot simpler to fix. Without that set however, the state of the parent's i_rwsem is unclear after nfsd_lookup is called, and I don't see a way to determine it reliably.

Jeff Layton (2):
nfsd: drop fh argument from alloc_init_deleg
nfsd: vet the opened dentry after setting a delegation

 fs/nfsd/nfs4state.c | 54 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 45 insertions(+), 9 deletions(-)
Signature · 1 line
2.36.1
author maintainer
#1 lore raw

PATCH 1/2 nfsd: drop fh argument from alloc_init_deleg

Currently, we pass the fh of the opened file down through several functions so that alloc_init_deleg can pass it to delegation_blocked. The filehandle of the open file is available in the nfs4_file however, so there's no need to pass it in a separate argument.

Drop the argument from alloc_init_deleg, nfs4_open_delegation and nfs4_set_delegation.

---
 fs/nfsd/nfs4state.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
fs/nfsd/nfs4state.c +6 -8
index 8201d716a557..d2d21fdf5c41 100644
@@ -1131,7 +1131,6 @@ static void block_delegations(struct knfsd_fh *fh)
 
 static struct nfs4_delegation *
 alloc_init_deleg(struct nfs4_client *clp, struct nfs4_file *fp,
-		 struct svc_fh *current_fh,
 		 struct nfs4_clnt_odstate *odstate)
 {
 	struct nfs4_delegation *dp;
@@ -1141,7 +1140,7 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_file *fp,
 	n = atomic_long_inc_return(&num_delegations);
 	if (n < 0 || n > max_delegations)
 		goto out_dec;
-	if (delegation_blocked(&current_fh->fh_handle))
+	if (delegation_blocked(&fp->fi_fhandle))
 		goto out_dec;
 	dp = delegstateid(nfs4_alloc_stid(clp, deleg_slab, nfs4_free_deleg));
 	if (dp == NULL)
@@ -5269,7 +5268,7 @@ static int nfsd4_check_conflicting_opens(struct nfs4_client *clp,
 }
 
 static struct nfs4_delegation *
-nfs4_set_delegation(struct nfs4_client *clp, struct svc_fh *fh,
+nfs4_set_delegation(struct nfs4_client *clp,
 		    struct nfs4_file *fp, struct nfs4_clnt_odstate *odstate)
 {
 	int status = 0;
@@ -5314,7 +5313,7 @@ nfs4_set_delegation(struct nfs4_client *clp, struct svc_fh *fh,
 		return ERR_PTR(status);
 
 	status = -ENOMEM;
-	dp = alloc_init_deleg(clp, fp, fh, odstate);
+	dp = alloc_init_deleg(clp, fp, odstate);
 	if (!dp)
 		goto out_delegees;
 
@@ -5382,8 +5381,7 @@ static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status)
  * proper support for them.
  */
 static void
-nfs4_open_delegation(struct svc_fh *fh, struct nfsd4_open *open,
-			struct nfs4_ol_stateid *stp)
+nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp)
 {
 	struct nfs4_delegation *dp;
 	struct nfs4_openowner *oo = openowner(stp->st_stateowner);
@@ -5415,7 +5413,7 @@ nfs4_open_delegation(struct svc_fh *fh, struct nfsd4_open *open,
 		default:
 			goto out_no_deleg;
 	}
-	dp = nfs4_set_delegation(clp, fh, stp->st_stid.sc_file, stp->st_clnt_odstate);
+	dp = nfs4_set_delegation(clp, stp->st_stid.sc_file, stp->st_clnt_odstate);
 	if (IS_ERR(dp))
 		goto out_no_deleg;
 
@@ -5547,7 +5545,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 	* Attempt to hand out a delegation. No error return, because the
 	* OPEN succeeds even if we fail.
 	*/
-	nfs4_open_delegation(current_fh, open, stp);
+	nfs4_open_delegation(open, stp);
 nodeleg:
 	status = nfs_ok;
 	trace_nfsd_open(&stp->st_stid.sc_stateid);
Signature · 1 line
2.36.1
author maintainer
#2 lore raw

PATCH 2/2 nfsd: vet the opened dentry after setting a delegation

In reply to Jeff Layton

Between opening a file and setting a delegation on it, someone could rename or unlink the dentry. If this happens, we do not want to grant a delegation on the open.

On a CLAIM_NULL open, we're opening by filename, and we'll hold the i_rwsem while when attempting to set a delegation. After getting a lease, redo the lookup of the file being opened and validate that the resulting dentry matches the one in the open file description.

---
 fs/nfsd/nfs4state.c | 48 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 5 deletions(-)
fs/nfsd/nfs4state.c +43 -5
index d2d21fdf5c41..8a8d8c738950 100644
@@ -5267,11 +5267,38 @@ static int nfsd4_check_conflicting_opens(struct nfs4_client *clp,
 	return 0;
 }
 
+/*
+ * It's possible that between opening the dentry and setting the delegation,
+ * that it has been renamed or unlinked. Redo the lookup to validate that this
+ * hasn't happened.
+ */
+static int
+nfsd4_vet_deleg_dentry(struct nfsd4_open *open, struct nfs4_file *fp,
+		       struct dentry *parent)
+{
+	struct dentry *child;
+
+	lockdep_assert_held(&d_inode(parent)->i_rwsem);
+
+	child = lookup_one_len(open->op_fname, parent, open->op_fnamelen);
+	if (IS_ERR(child))
+		return PTR_ERR(child);
+	dput(child);
+
+	if (child != file_dentry(fp->fi_deleg_file->nf_file))
+		return -EAGAIN;
+
+	return 0;
+}
+
 static struct nfs4_delegation *
-nfs4_set_delegation(struct nfs4_client *clp,
-		    struct nfs4_file *fp, struct nfs4_clnt_odstate *odstate)
+nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
+		    struct dentry *parent)
 {
 	int status = 0;
+	struct nfs4_client *clp = stp->st_stid.sc_client;
+	struct nfs4_file *fp = stp->st_stid.sc_file;
+	struct nfs4_clnt_odstate *odstate = stp->st_clnt_odstate;
 	struct nfs4_delegation *dp;
 	struct nfsd_file *nf;
 	struct file_lock *fl;
@@ -5326,6 +5353,13 @@ nfs4_set_delegation(struct nfs4_client *clp,
 		locks_free_lock(fl);
 	if (status)
 		goto out_clnt_odstate;
+
+	if (parent) {
+		status = nfsd4_vet_deleg_dentry(open, fp, parent);
+		if (status)
+			goto out_unlock;
+	}
+
 	status = nfsd4_check_conflicting_opens(clp, fp);
 	if (status)
 		goto out_unlock;
@@ -5381,11 +5415,13 @@ static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status)
  * proper support for them.
  */
 static void
-nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp)
+nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp,
+		     struct dentry *cdentry)
 {
 	struct nfs4_delegation *dp;
 	struct nfs4_openowner *oo = openowner(stp->st_stateowner);
 	struct nfs4_client *clp = stp->st_stid.sc_client;
+	struct dentry *parent = NULL;
 	int cb_up;
 	int status = 0;
 
@@ -5399,6 +5435,8 @@ nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp)
 				goto out_no_deleg;
 			break;
 		case NFS4_OPEN_CLAIM_NULL:
+			parent = cdentry;
+			fallthrough;
 		case NFS4_OPEN_CLAIM_FH:
 			/*
 			 * Let's not give out any delegations till everyone's
@@ -5413,7 +5451,7 @@ nfs4_open_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp)
 		default:
 			goto out_no_deleg;
 	}
-	dp = nfs4_set_delegation(clp, stp->st_stid.sc_file, stp->st_clnt_odstate);
+	dp = nfs4_set_delegation(open, stp, parent);
 	if (IS_ERR(dp))
 		goto out_no_deleg;
 
@@ -5545,7 +5583,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 	* Attempt to hand out a delegation. No error return, because the
 	* OPEN succeeds even if we fail.
 	*/
-	nfs4_open_delegation(open, stp);
+	nfs4_open_delegation(open, stp, resp->cstate.current_fh.fh_dentry);
 nodeleg:
 	status = nfs_ok;
 	trace_nfsd_open(&stp->st_stid.sc_stateid);
Signature · 1 line
2.36.1
#3 lore raw
In reply to Jeff Layton
Quoted 11 lines · Jeff Layton

This is a respin of the patchset that I sent earlier today. I hit a deadlock with that one because of the ambiguous locking.

This series is based on top of Neil's set entitled:

[PATCH 0/8] NFSD: clean up locking.

His patchset makes the locking in the nfsd4_open codepath much more consistent, and this becomes a lot simpler to fix. Without that set however, the state of the parent's i_rwsem is unclear after nfsd_lookup is called, and I don't see a way to determine it reliably.

I haven't examined these patch very closely, but a few initial thoughts are:

1/ Before my series, you can unambiguously tell if i_rwsem is held by
checking fhp->fh_locked. In fact, just call "fh_lock()", and you can then be sure the fh is locked, whether or not it was locked before
however...
2/ Do we really need to lock the parent? If a rename or unlink happens
after the lease was taken, the lease will be broken. So
take lease.
repeat lookup (locklessly)
Check if lease has been broken
Should provide all you need.

You don't *need* to lock the directory to open an existing file and with my pending parallel-updates patch set, you only need a shared lock on the directory to create a file. So I'd rather not be locking the directory at all to get a delegation

3/ When you vet the name you only do a lookup_one_len(), while
nfsd_lookup_dentry() also calls nfsd_cross_mnt() as it is possible for a file to be mounted on.
That means that if I did bind mount one file over another and export over NFSD, the file will never offer a delegation.
This is a minor point, but I think it would be best to be as correct and consistent as possible.

Thanks for working on this!

NeilBrown

Quoted 12 lines

Jeff Layton (2):
nfsd: drop fh argument from alloc_init_deleg
nfsd: vet the opened dentry after setting a delegation

 fs/nfsd/nfs4state.c | 54 +++++++++++++++++++++++++++++++++++++--------
 1 file changed, 45 insertions(+), 9 deletions(-)
Signature · 1 line
2.36.1
author maintainer
#4 lore raw
Quoted 13 lines · NeilBrown
Quoted 11 lines · Jeff Layton

This is a respin of the patchset that I sent earlier today. I hit a deadlock with that one because of the ambiguous locking.

This series is based on top of Neil's set entitled:

[PATCH 0/8] NFSD: clean up locking.

His patchset makes the locking in the nfsd4_open codepath much more consistent, and this becomes a lot simpler to fix. Without that set however, the state of the parent's i_rwsem is unclear after nfsd_lookup is called, and I don't see a way to determine it reliably.

Quoted 6 lines

I haven't examined these patch very closely, but a few initial thoughts are:

1/ Before my series, you can unambiguously tell if i_rwsem is held by
checking fhp->fh_locked. In fact, just call "fh_lock()", and you can then be sure the fh is locked, whether or not it was locked before

Thanks, good to know. I wasn't sure how reliable that bool is. I guess though that once you have a svc_fh, then you can more or less assume that you have exclusive access to it for the life of the RPC being processed.

Quoted 8 lines

however...
2/ Do we really need to lock the parent? If a rename or unlink happens
after the lease was taken, the lease will be broken. So
take lease.
repeat lookup (locklessly)
Check if lease has been broken
Should provide all you need.

Quoted 5 lines

You don't *need* to lock the directory to open an existing file and with my pending parallel-updates patch set, you only need a shared lock on the directory to create a file. So I'd rather not be locking the directory at all to get a delegation

Yeah, we probably don't need to lock the dir. That said, after your patch series we already hold the i_rwsem on the parent at this point so lookup_one_len is fine in this instance.

Quoted 8 lines

3/ When you vet the name you only do a lookup_one_len(), while
nfsd_lookup_dentry() also calls nfsd_cross_mnt() as it is possible for a file to be mounted on.
That means that if I did bind mount one file over another and export over NFSD, the file will never offer a delegation.
This is a minor point, but I think it would be best to be as correct and consistent as possible.

Agreed, but that will take a bit more work. nfsd_lookup_dentry takes several parameters that we don't currently have access to in nfs4_set_delegation (e.g. the rqstp). Those will need to be plumbed through several functions.

Quoted 1 line

Thanks for working on this!

...and thank you for the locking cleanup! Getting rid of fh_lock/_unlock is a really nice cleanup that makes it a lot more clear how this should work.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 16 lines · Jeff Layton
Quoted 8 lines · NeilBrown

however...
2/ Do we really need to lock the parent? If a rename or unlink happens
after the lease was taken, the lease will be broken. So
take lease.
repeat lookup (locklessly)
Check if lease has been broken
Should provide all you need.

Quoted 5 lines

You don't *need* to lock the directory to open an existing file and with my pending parallel-updates patch set, you only need a shared lock on the directory to create a file. So I'd rather not be locking the directory at all to get a delegation

Quoted 3 lines

Yeah, we probably don't need to lock the dir. That said, after your patch series we already hold the i_rwsem on the parent at this point so lookup_one_len is fine in this instance.

But the only reason we hold i_rwsem at this point is to prevent renames in the "opened existing file" case. The "created new file" case holds it as well just be be consistent with the first case.

If we "vet" the dentry, then we don't need the lock any more. We can then simplify nfsd_lookup_dentry() to always assume the dir is not locked - so the "locked" arg can go, and nfsd_lookup() can lose the "lock" arg and always return with the directory unlocked.

I'm tempted to add your patch to the front of my series. The inconsistency in locking can be fix by unlocking the directory before we get even close to handing out a delegation - so the delegation never sees a locked directory.
But right now I have a cold and don't trust myself to think clearly enough to create code worth posting. Hopefully I'll be thinking more clearly later in the week.

While I'm here ... is "vet" a good word? The meaning is appropriate, but I wonder if it would cause our friends for whom English isn't their first language to stumble. There are about 5 uses in the kernel at present.

Would validate or verify be better? Even though they are longer..

Thanks,
NeilBrown

author maintainer
#6 lore raw
Quoted 30 lines · NeilBrown
Quoted 16 lines · Jeff Layton
Quoted 8 lines · NeilBrown

however...
2/ Do we really need to lock the parent? If a rename or unlink happens
after the lease was taken, the lease will be broken. So
take lease.
repeat lookup (locklessly)
Check if lease has been broken
Should provide all you need.

Quoted 5 lines

You don't *need* to lock the directory to open an existing file and with my pending parallel-updates patch set, you only need a shared lock on the directory to create a file. So I'd rather not be locking the directory at all to get a delegation

Quoted 3 lines

Yeah, we probably don't need to lock the dir. That said, after your patch series we already hold the i_rwsem on the parent at this point so lookup_one_len is fine in this instance.

But the only reason we hold i_rwsem at this point is to prevent renames in the "opened existing file" case. The "created new file" case holds it as well just be be consistent with the first case.

If we "vet" the dentry, then we don't need the lock any more. We can then simplify nfsd_lookup_dentry() to always assume the dir is not locked - so the "locked" arg can go, and nfsd_lookup() can lose the "lock" arg and always return with the directory unlocked.

Quoted 4 lines

I'm tempted to add your patch to the front of my series. The inconsistency in locking can be fix by unlocking the directory before we get even close to handing out a delegation - so the delegation never sees a locked directory.

Hmm, ok. I suppose we don't necessarily have to care whether the thing is locked before calling into nfsd_lookup_dentry. I'll take another stab at fixing this in the kernel w/o your series. That'll make Chuck happy too.

Quoted 10 lines

But right now I have a cold and don't trust myself to think clearly enough to create code worth posting. Hopefully I'll be thinking more clearly later in the week.

While I'm here ... is "vet" a good word? The meaning is appropriate, but I wonder if it would cause our friends for whom English isn't their first language to stumble. There are about 5 uses in the kernel at present.

Would validate or verify be better? Even though they are longer..

Good point. I'm all for helping out non-native English speakers. I'll plan to change it to something less esoteric.

Signature · 1 line
Jeff Layton <[email protected]>
#7 lore raw
In reply to NeilBrown
Quoted 7 lines · NeilBrown

But right now I have a cold and don't trust myself to think clearly enough to create code worth posting. Hopefully I'll be thinking more clearly later in the week.

Thanks for the update!

Here are my plans: I'd like to finalize content of the first 5.20 NFSD pull request this week. If these patches are not ready by then, I can prepare a second PR later in the 5.20 merge window with your work, which should give you another two weeks. If they are still not ready, I'll get them in first thing for the next merge window.

--
Chuck Lever

#8 lore raw
Quoted 7 lines · Chuck Lever III
Quoted 6 lines · NeilBrown

But right now I have a cold and don't trust myself to think clearly enough to create code worth posting. Hopefully I'll be thinking more clearly later in the week.

Quoted 8 lines

Thanks for the update!

Here are my plans: I'd like to finalize content of the first 5.20 NFSD pull request this week. If these patches are not ready by then, I can prepare a second PR later in the 5.20 merge window with your work, which should give you another two weeks. If they are still not ready, I'll get them in first thing for the next merge window.

FYI I've addressed the outstanding issues (I think), but have not yet done any testing or given the patches a final inspection. I hope to do that tomorrow, and will post the patches once it is done.
If you want a preview you can find them on github.com/neilbrown/linux in the "nfsd" branch.

Jeff's patches to validate delegations after getting the lease, so we don't have to hold the lock so long, comes first.

NeilBrown