Skip to contentSkip to conversation

nfsd: fix use-after-free oops in v4.0 (and some other minor cleanups)

PATCH In review Jeff Layton 4 participants 13 messages Mar 23, 2015 – Mar 25, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover nfsd: fix use-after-free oops in v4.0 (and some other minor cleanups)

Hi Bruce,

After staring at this code for a _long_ time, I think I've finally found the source of the use-after-free oops that HCH spotted. The first patch in this series should fix that. The other two patches are just cleanups that I generated while staring at the code.

The first one obviously needs to go into v4.0 (and stable) ASAP. The other two can wait for v4.1.

Jeff Layton (3):
nfsd: return correct openowner when there is a race to put one in the
hash
nfsd: remove bogus setting of status in nfsd4_process_open2
nfsd: remove unused status arg to nfsd4_cleanup_open_state

 fs/nfsd/nfs4proc.c  | 2 +-
 fs/nfsd/nfs4state.c | 5 ++---
 fs/nfsd/xdr4.h      | 2 +-
 3 files changed, 4 insertions(+), 5 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/3 nfsd: return correct openowner when there is a race to put one in the hash

alloc_init_open_stateowner can return an already freed entry if there is a race to put openowners in the hashtable.

In commit 7ffb588086e9, we changed it so that we allocate and initialize an openowner, and then check to see if a matching one got stuffed into the hashtable in the meantime. If it did, then we free the one we just allocated and take a reference on the one already there. There is a bug here though. The code will then return the pointer to the one that was allocated (and has now been freed).

This wasn't evident before as this race almost never occurred. The Linux kernel client used to serialize requests for a single openowner. That has changed now with v4.0 kernels, and this race can now easily occur.

Fixes 7ffb588086e9
Cc <[email protected]> # v3.17+
Cc Trond Myklebust <[email protected]>
---
 fs/nfsd/nfs4state.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfs4state.c +1 -1
index d2f2c37dc2db..cf29d1a698b3 100644
@@ -3221,7 +3221,7 @@ alloc_init_open_stateowner(unsigned int strhashval, struct nfsd4_open *open,
 	} else
 		nfs4_free_openowner(&oo->oo_owner);
 	spin_unlock(&clp->cl_lock);
-	return oo;
+	return ret;
 }
 
 static void init_open_stateid(struct nfs4_ol_stateid *stp, struct nfs4_file *fp, struct nfsd4_open *open) {
Signature · 1 line
2.1.0
#2 lore raw

Re: [PATCH 1/3] nfsd: return correct openowner when there is a race to put one in the hash

Quoted 2 lines · Jeff Layton

alloc_init_open_stateowner can return an already freed entry if there is a race to put openowners in the hashtable.

Looks like alloc_init_lock_stateowner has the same bug, so I'll apply something like this pending testing.

I wonder if it's actually possible to hit this one?

--b.

commit bdff3084f09f

Author: J. Bruce Fields <[email protected]>
Date:   Mon Mar 23 11:02:30 2015 -0400

nfsd: return correct lockowner when there is a race on hash insert

alloc_init_lock_stateowner can return an already freed entry if there is a race to put openowners in the hashtable.

Noticed by inspection after Jeff Layton fixed the same bug for open owners. Depending on client behavior, this one may be trickier to trigger in practice.

Fixes c58c6610ec24 "nfsd: Protect adding/removing lock owners using client_lock"
Cc Trond Myklebust <[email protected]>
Cc Jeff Layton <[email protected]>
fs/nfsd/nfs4state.c +1 -1
index d2f2c37dc2db..49ae6116992f 100644
@@ -5062,7 +5062,7 @@ alloc_init_lock_stateowner(unsigned int strhashval, struct nfs4_client *clp,
 	} else
 		nfs4_free_lockowner(&lo->lo_owner);
 	spin_unlock(&clp->cl_lock);
-	return lo;
+	return ret;
 }
 
 static void
Acked-by
#3 lore raw

Re: [PATCH 1/3] nfsd: return correct openowner when there is a race to put one in the hash

Quoted 35 lines · On Mon, 23 Mar 2015 11:36:14 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 2 lines · Jeff Layton

alloc_init_open_stateowner can return an already freed entry if there is a race to put openowners in the hashtable.

Looks like alloc_init_lock_stateowner has the same bug, so I'll apply something like this pending testing.

I wonder if it's actually possible to hit this one?

--b.

commit bdff3084f09f

Author: J. Bruce Fields <[email protected]>
Date:   Mon Mar 23 11:02:30 2015 -0400

nfsd: return correct lockowner when there is a race on hash insert

alloc_init_lock_stateowner can return an already freed entry if there is a race to put openowners in the hashtable.

Noticed by inspection after Jeff Layton fixed the same bug for open owners. Depending on client behavior, this one may be trickier to trigger in practice.

Fixes c58c6610ec24 "nfsd: Protect adding/removing lock owners using client_lock"
Cc Trond Myklebust <[email protected]>
Cc Jeff Layton <[email protected]>
fs/nfsd/nfs4state.c +0 -0
index d2f2c37dc2db..49ae6116992f 100644

fs/nfsd/nfs4state.c alloc_init_lock_stateowner(unsigned int strhashval, struct nfs4_client *clp, · line 5068

Quoted 8 lines
fs/nfsd/nfs4state.c +1 -1
@@ -5062,7 +5062,7 @@ alloc_init_lock_stateowner(unsigned int strhashval, struct nfs4_client *clp,
 	} else
 		nfs4_free_lockowner(&lo->lo_owner);
 	spin_unlock(&clp->cl_lock);
-	return lo;
+	return ret;
 }
 
 static void

Maybe. Commit b4019c0e219 allows us to do parallel LOCK/LOCKU calls for the same owner, but it also says:

"Note, however, that we still serialise on the open stateid if the lock stateid is unconfirmed."

...which I take to mean that the initial LOCK calls likely wouldn't race this way. Still, it's probably possible to craft a pynfs test or something that does that.

Either way, it's clearly a bug that should be fixed:

Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] nfsd: return correct openowner when there is a race to put one in the hash

In reply to Jeff Layton

Good catch!

But maybe it's better to write the code so that the intent is obvious:

fs/nfsd/nfs4state.c +6 -4
index f6b2a09..3c4c3d2 100644
@@ -3215,11 +3215,13 @@ alloc_init_open_stateowner(unsigned int strhashval, struct nfsd4_open *open,
 	INIT_LIST_HEAD(&oo->oo_close_lru);
 	spin_lock(&clp->cl_lock);
 	ret = find_openstateowner_str_locked(strhashval, open, clp);
-	if (ret == NULL) {
-		hash_openowner(oo, clp, strhashval);
-		ret = oo;
-	} else
+	if (ret) {
 		nfs4_free_openowner(&oo->oo_owner);
+		spin_unlock(&clp->cl_lock);
+		return ret;
+	}
+
+	hash_openowner(oo, clp, strhashval);
 	spin_unlock(&clp->cl_lock);
 	return oo;
 }
#5 lore raw

Re: [PATCH 1/3] nfsd: return correct openowner when there is a race to put one in the hash

Quoted 18 lines · On Wed, 25 Mar 2015 01:45:16 -0700 Christoph Hellwig <[email protected]> wrote:

Good catch!

But maybe it's better to write the code so that the intent is obvious:

fs/nfsd/nfs4state.c +0 -4
index f6b2a09..3c4c3d2 100644
@@ -3215,11 +3215,13 @@ alloc_init_open_stateowner(unsigned int strhashval, struct nfsd4_open *open,
 	INIT_LIST_HEAD(&oo->oo_close_lru);
 	spin_lock(&clp->cl_lock);
 	ret = find_openstateowner_str_locked(strhashval, open, clp);
-	if (ret == NULL) {
-		hash_openowner(oo, clp, strhashval);
-		ret = oo;
-	} else

fs/nfsd/nfs4state.c alloc_init_open_stateowner(unsigned int strhashval, struct nfsd4_open *open, · line 3227

Quoted 10 lines
fs/nfsd/nfs4state.c +6 -0
+	if (ret) {
 		nfs4_free_openowner(&oo->oo_owner);
+		spin_unlock(&clp->cl_lock);
+		return ret;
+	}
+
+	hash_openowner(oo, clp, strhashval);
 	spin_unlock(&clp->cl_lock);
 	return oo;
 }

Sure, that'd be fine too. Care to write up a changelog entry and send it to Bruce? I think that would likely apply just fine to stable kernels.

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw

Re: [PATCH 1/3] nfsd: return correct openowner when there is a race to put one in the hash

Let's just keep your version for simplicity for now..

#7 lore raw

PATCH 2/3 nfsd: remove bogus setting of status in nfsd4_process_open2

In reply to Jeff Layton

status is always reset after this (and it doesn't make much sense there anyway).

---
 fs/nfsd/nfs4state.c | 1 -
 1 file changed, 1 deletion(-)
fs/nfsd/nfs4state.c +0 -1
index cf29d1a698b3..a66e297dc01c 100644
@@ -4049,7 +4049,6 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 		status = nfserr_bad_stateid;
 		if (nfsd4_is_deleg_cur(open))
 			goto out;
-		status = nfserr_jukebox;
 	}
 
 	/*
Signature · 1 line
2.1.0
#9 lore raw

PATCH 3/3 nfsd: remove unused status arg to nfsd4_cleanup_open_state

In reply to Jeff Layton
---
 fs/nfsd/nfs4proc.c  | 2 +-
 fs/nfsd/nfs4state.c | 2 +-
 fs/nfsd/xdr4.h      | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
fs/nfsd/nfs4proc.c +1 -1
index d30bea8d0277..e54f18ae5b2e 100644
@@ -470,7 +470,7 @@ out:
 		fh_put(resfh);
 		kfree(resfh);
 	}
-	nfsd4_cleanup_open_state(cstate, open, status);
+	nfsd4_cleanup_open_state(cstate, open);
 	nfsd4_bump_seqid(cstate, status);
 	return status;
 }
fs/nfsd/nfs4state.c +1 -1
index a66e297dc01c..3d9b0b9b1e2c 100644
@@ -4117,7 +4117,7 @@ out:
 }
 
 void nfsd4_cleanup_open_state(struct nfsd4_compound_state *cstate,
-			      struct nfsd4_open *open, __be32 status)
+			      struct nfsd4_open *open)
 {
 	if (open->op_openowner) {
 		struct nfs4_stateowner *so = &open->op_openowner->oo_owner;
fs/nfsd/xdr4.h +1 -1
index 0bda93e58e1b..83e80d6d94a6 100644
@@ -683,7 +683,7 @@ extern __be32 nfsd4_process_open2(struct svc_rqst *rqstp,
 		struct svc_fh *current_fh, struct nfsd4_open *open);
 extern void nfsd4_cstate_clear_replay(struct nfsd4_compound_state *cstate);
 extern void nfsd4_cleanup_open_state(struct nfsd4_compound_state *cstate,
-		struct nfsd4_open *open, __be32 status);
+		struct nfsd4_open *open);
 extern __be32 nfsd4_open_confirm(struct svc_rqst *rqstp,
 		struct nfsd4_compound_state *, struct nfsd4_open_confirm *oc);
 extern __be32 nfsd4_close(struct svc_rqst *rqstp,
Signature · 1 line
2.1.0
#11 lore raw
In reply to Jeff Layton
Quoted 7 lines · Jeff Layton

After staring at this code for a _long_ time, I think I've finally found the source of the use-after-free oops that HCH spotted. The first patch in this series should fix that. The other two patches are just cleanups that I generated while staring at the code.

The first one obviously needs to go into v4.0 (and stable) ASAP. The other two can wait for v4.1.

Zowie, thanks! It's a relief to have this one found....--b.

Quoted 19 lines

Jeff Layton (3):
nfsd: return correct openowner when there is a race to put one in the
hash
nfsd: remove bogus setting of status in nfsd4_process_open2
nfsd: remove unused status arg to nfsd4_cleanup_open_state

 fs/nfsd/nfs4proc.c  | 2 +-
 fs/nfsd/nfs4state.c | 5 ++---
 fs/nfsd/xdr4.h      | 2 +-
 3 files changed, 4 insertions(+), 5 deletions(-)
Signature · 6 lines
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#12 lore raw
Quoted 11 lines · On Mon, 23 Mar 2015 11:12:57 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 7 lines · Jeff Layton

After staring at this code for a _long_ time, I think I've finally found the source of the use-after-free oops that HCH spotted. The first patch in this series should fix that. The other two patches are just cleanups that I generated while staring at the code.

The first one obviously needs to go into v4.0 (and stable) ASAP. The other two can wait for v4.1.

Zowie, thanks! It's a relief to have this one found....--b.

Definitely a relief. Just to reiterate what you spotted on IRC too, it looks like there's a similar bug in alloc_init_lock_stateowner so we'll need a patch for that as well.

Quoted 19 lines
Quoted 19 lines

Jeff Layton (3):
nfsd: return correct openowner when there is a race to put one in the
hash
nfsd: remove bogus setting of status in nfsd4_process_open2
nfsd: remove unused status arg to nfsd4_cleanup_open_state

 fs/nfsd/nfs4proc.c  | 2 +-
 fs/nfsd/nfs4state.c | 5 ++---
 fs/nfsd/xdr4.h      | 2 +-
 3 files changed, 4 insertions(+), 5 deletions(-)
Signature · 6 lines
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Jeff Layton <[email protected]>