Skip to contentSkip to conversation

nfsd: fix potential race in nfs4_find_file

PATCH In review Jeff Layton 3 participants 8 messages Jan 5, 2023 – Jan 7, 2023

Subsystems (2)
lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

PATCH 1/1 nfsd: fix potential race in nfs4_find_file

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
 fs/nfsd/nfs4state.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +10 -6
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 static struct nfsd_file *
 nfs4_find_file(struct nfs4_stid *s, int flags)
 {
+	struct nfsd_file *ret = NULL;
+
 	if (!s)
 		return NULL;
 
 	switch (s->sc_type) {
 	case NFS4_DELEG_STID:
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;
 	case NFS4_OPEN_STID:
 	case NFS4_LOCK_STID:
 		if (flags & RD_STATE)
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);
 		else
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
 	}
 
-	return NULL;
+	return ret;
 }
 
 static __be32
Signature · 1 line
2.39.0
#1 lore raw
Quoted 31 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
fs/nfsd/nfs4state.c | 16 ++++++++++------

1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +0 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,

static struct nfsd_file *
nfs4_find_file(struct nfs4_stid *s, int flags)
{

fs/nfsd/nfs4state.c +2 -0
+	struct nfsd_file *ret = NULL;
+
	if (!s)
		return NULL;

	switch (s->sc_type) {
	case NFS4_DELEG_STID:

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6428

Quoted 5 lines
fs/nfsd/nfs4state.c +2 -3
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))

You'd think this would be a really really hard race to hit.

What I'm wondering, though, is whether the WARN_ON_ONCE should be dropped by this patch. I've never seen it fire.

Quoted 17 lines
fs/nfsd/nfs4state.c +3 -0
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

case NFS4_OPEN_STID:
case NFS4_LOCK_STID:
if (flags & RD_STATE)

fs/nfsd/nfs4state.c +1 -1
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);

else

fs/nfsd/nfs4state.c +1 -1
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
	}
fs/nfsd/nfs4state.c +1 -1
-	return NULL;
+	return ret;

}

Quoted 4 lines

static __be32

Signature · 1 line
2.39.0

--
Chuck Lever

author maintainer
#2 lore raw
Quoted 38 lines · Chuck Lever III
Quoted 30 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
fs/nfsd/nfs4state.c | 16 ++++++++++------

1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +0 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,

static struct nfsd_file *
nfs4_find_file(struct nfs4_stid *s, int flags)
{

fs/nfsd/nfs4state.c +2 -0
+	struct nfsd_file *ret = NULL;
+
	if (!s)
		return NULL;

	switch (s->sc_type) {
	case NFS4_DELEG_STID:

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6428

Quoted 5 lines
fs/nfsd/nfs4state.c +2 -3
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))

You'd think this would be a really really hard race to hit.

Quoted 4 lines

What I'm wondering, though, is whether the WARN_ON_ONCE should be dropped by this patch. I've never seen it fire.

I have:

    https://bugzilla.redhat.com/show_bug.cgi?id=1997177

It's possible though that those WARNs are fallout from other bugs in the delegation handling, but it's hard to know for sure. I think we ought to keep it there for now.

Quoted 27 lines
Quoted 17 lines
fs/nfsd/nfs4state.c +3 -0
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

case NFS4_OPEN_STID:
case NFS4_LOCK_STID:
if (flags & RD_STATE)

fs/nfsd/nfs4state.c +1 -1
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);

else

fs/nfsd/nfs4state.c +1 -1
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
	}
fs/nfsd/nfs4state.c +1 -1
-	return NULL;
+	return ret;

}

Quoted 4 lines

static __be32

Signature · 1 line
2.39.0

--
Chuck Lever

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 48 lines · Jeff Layton
Quoted 38 lines · Chuck Lever III
Quoted 30 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
fs/nfsd/nfs4state.c | 16 ++++++++++------

1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +0 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,

static struct nfsd_file *
nfs4_find_file(struct nfs4_stid *s, int flags)
{

fs/nfsd/nfs4state.c +2 -0
+	struct nfsd_file *ret = NULL;
+
	if (!s)
		return NULL;

	switch (s->sc_type) {
	case NFS4_DELEG_STID:

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6428

Quoted 5 lines
fs/nfsd/nfs4state.c +2 -3
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))

You'd think this would be a really really hard race to hit.

Quoted 4 lines

What I'm wondering, though, is whether the WARN_ON_ONCE should be dropped by this patch. I've never seen it fire.

Quoted 6 lines

I have:

   https://bugzilla.redhat.com/show_bug.cgi?id=1997177

It's possible though that those WARNs are fallout from other bugs in the delegation handling, but it's hard to know for sure.

Before 2015 there were a bunch of BUG_ON's in this code that were converted to WARN after Linus complained. Before that, I think these were all debugging sentinels. (in which case I would argue they might be better recast as tracepoints, but that's for another day).

Quoted 1 line

I think we ought to keep it there for now.

The question is whether the WARN_ON is adding value for customers. Can they do something about it? If they give us this information, can we do something about it?

I can't tell from the warning whether the problem is due to a server bug or valid client behavior. Both the server and the client workload appear to survive.

So, I just don't feel like it's adding value, and firing a WARN while holding a spinlock makes me squidgy.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
fs/nfsd/nfs4state.c +3 -0
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

case NFS4_OPEN_STID:
case NFS4_LOCK_STID:
if (flags & RD_STATE)

fs/nfsd/nfs4state.c +1 -1
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);

else

fs/nfsd/nfs4state.c +1 -1
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
	}
fs/nfsd/nfs4state.c +1 -1
-	return NULL;
+	return ret;

}

Quoted 13 lines
Quoted 10 lines
Quoted 4 lines

static __be32

Signature · 1 line
2.39.0

--
Chuck Lever

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

#4 lore raw
In reply to Jeff Layton
Quoted 25 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
 fs/nfsd/nfs4state.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +2 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 static struct nfsd_file *
 nfs4_find_file(struct nfs4_stid *s, int flags)
 {
+	struct nfsd_file *ret = NULL;
+
 	if (!s)
 		return NULL;
 

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6431

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 	switch (s->sc_type) {
 	case NFS4_DELEG_STID:
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

As an nfsd_file is freed with rcu, we don't need the spinlock.

rcu_read_lock()
ret = rcu_dereference(s->sc_file->fi_deleg_file); if (ret)

	ret = nfsd_file_get(ret);

rcu_read_unlock();

You could even put the NULL test in nfsd_file_get() and have:

rcu_read_lock()l;
ret = nfsd_file_get(rcu_dereference(s->sc_file->fi_deleg_file)); rcu_read_unlock();

but that might not be a win.

I agree with Chuck that the WARNing isn't helpful.

NeilBrown

Quoted 19 lines
fs/nfsd/nfs4state.c +3 -3
 	case NFS4_OPEN_STID:
 	case NFS4_LOCK_STID:
 		if (flags & RD_STATE)
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);
 		else
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
 	}
 
-	return NULL;
+	return ret;
 }
 
 static __be32
Signature · 1 line
2.39.0
author maintainer
#5 lore raw
Quoted 47 lines · NeilBrown
Quoted 25 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
 fs/nfsd/nfs4state.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +2 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 static struct nfsd_file *
 nfs4_find_file(struct nfs4_stid *s, int flags)
 {
+	struct nfsd_file *ret = NULL;
+
 	if (!s)
 		return NULL;
 

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6431

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 	switch (s->sc_type) {
 	case NFS4_DELEG_STID:
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

As an nfsd_file is freed with rcu, we don't need the spinlock.

rcu_read_lock()
ret = rcu_dereference(s->sc_file->fi_deleg_file); if (ret)

	ret = nfsd_file_get(ret);

rcu_read_unlock();

You could even put the NULL test in nfsd_file_get() and have:

Quoted 9 lines

rcu_read_lock()l;
ret = nfsd_file_get(rcu_dereference(s->sc_file->fi_deleg_file)); rcu_read_unlock();

but that might not be a win.

I agree with Chuck that the WARNing isn't helpful.

Good point. I'll send a v2 set in a bit.

Quoted 20 lines
Quoted 20 lines
fs/nfsd/nfs4state.c +3 -3
 	case NFS4_OPEN_STID:
 	case NFS4_LOCK_STID:
 		if (flags & RD_STATE)
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);
 		else
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
 	}
 
-	return NULL;
+	return ret;
 }
 
 static __be32
Signature · 1 line
2.39.0
Signature · 1 line
Jeff Layton <[email protected]>
author maintainer
#6 lore raw
In reply to NeilBrown
Quoted 51 lines · NeilBrown
Quoted 25 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
 fs/nfsd/nfs4state.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +2 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 static struct nfsd_file *
 nfs4_find_file(struct nfs4_stid *s, int flags)
 {
+	struct nfsd_file *ret = NULL;
+
 	if (!s)
 		return NULL;
 

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6431

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 	switch (s->sc_type) {
 	case NFS4_DELEG_STID:
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

As an nfsd_file is freed with rcu, we don't need the spinlock.

rcu_read_lock()
ret = rcu_dereference(s->sc_file->fi_deleg_file); if (ret)

	ret = nfsd_file_get(ret);

rcu_read_unlock();

You could even put the NULL test in nfsd_file_get() and have:

rcu_read_lock()l;
ret = nfsd_file_get(rcu_dereference(s->sc_file->fi_deleg_file)); rcu_read_unlock();

Quoted 6 lines

but that might not be a win.

I agree with Chuck that the WARNing isn't helpful.

NeilBrown

Ok, I took a look at this.

To do it right, we'd need to annotate the fi_deleg_file field with __rcu. That means we'd need to clean up a bunch of existing fi_deleg_file accesses to properly use rcu_dereference_protected.

This is probably worthwhile stuff to do, but it's a larger patch series and will touch a bunch of unrelated delegation handling. At this point, I think I'd rather just keep the spinlocking here since that should be safe. Cleaning up delegation handling is a longer-term project that I'd rather table for now.

I will remove the WARN_ON_ONCE though, and I think allowing nfsd_file_get to accept a NULL pointer is probably a good thing too. I'll resend a new series in a bit.

Quoted 20 lines
Quoted 20 lines
fs/nfsd/nfs4state.c +3 -3
 	case NFS4_OPEN_STID:
 	case NFS4_LOCK_STID:
 		if (flags & RD_STATE)
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);
 		else
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
 	}
 
-	return NULL;
+	return ret;
 }
 
 static __be32
Signature · 1 line
2.39.0
Signature · 1 line
Jeff Layton <[email protected]>
#7 lore raw
Quoted 65 lines · Jeff Layton
Quoted 51 lines · NeilBrown
Quoted 25 lines · Jeff Layton

Even though there is a WARN_ON_ONCE check, it seems possible for nfs4_find_file to race with the destruction of an fi_deleg_file while trying to take a reference to it.

put_deleg_file is done while holding the fi_lock. Take and hold it when dealing with the fi_deleg_file in nfs4_find_file.

---
 fs/nfsd/nfs4state.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +2 -0
index b68238024e49..3df3ae84bd07 100644
@@ -6417,23 +6417,27 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 static struct nfsd_file *
 nfs4_find_file(struct nfs4_stid *s, int flags)
 {
+	struct nfsd_file *ret = NULL;
+
 	if (!s)
 		return NULL;
 

fs/nfsd/nfs4state.c nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, · line 6431

Quoted 10 lines
fs/nfsd/nfs4state.c +5 -3
 	switch (s->sc_type) {
 	case NFS4_DELEG_STID:
-		if (WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
-			return NULL;
-		return nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_lock(&s->sc_file->fi_lock);
+		if (!WARN_ON_ONCE(!s->sc_file->fi_deleg_file))
+			ret = nfsd_file_get(s->sc_file->fi_deleg_file);
+		spin_unlock(&s->sc_file->fi_lock);
+		break;

As an nfsd_file is freed with rcu, we don't need the spinlock.

rcu_read_lock()
ret = rcu_dereference(s->sc_file->fi_deleg_file); if (ret)

	ret = nfsd_file_get(ret);

rcu_read_unlock();

You could even put the NULL test in nfsd_file_get() and have:

rcu_read_lock()l;
ret = nfsd_file_get(rcu_dereference(s->sc_file->fi_deleg_file)); rcu_read_unlock();

Quoted 6 lines

but that might not be a win.

I agree with Chuck that the WARNing isn't helpful.

NeilBrown

Ok, I took a look at this.

To do it right, we'd need to annotate the fi_deleg_file field with __rcu. That means we'd need to clean up a bunch of existing fi_deleg_file accesses to properly use rcu_dereference_protected.

Quoted 5 lines

This is probably worthwhile stuff to do, but it's a larger patch series and will touch a bunch of unrelated delegation handling. At this point, I think I'd rather just keep the spinlocking here since that should be safe. Cleaning up delegation handling is a longer-term project that I'd rather table for now.

That all seems very sensible - thank for looking into it.

NeilBrown

Quoted 29 lines

I will remove the WARN_ON_ONCE though, and I think allowing nfsd_file_get to accept a NULL pointer is probably a good thing too. I'll resend a new series in a bit.

Quoted 20 lines
Quoted 20 lines
fs/nfsd/nfs4state.c +3 -3
 	case NFS4_OPEN_STID:
 	case NFS4_LOCK_STID:
 		if (flags & RD_STATE)
-			return find_readable_file(s->sc_file);
+			ret = find_readable_file(s->sc_file);
 		else
-			return find_writeable_file(s->sc_file);
+			ret = find_writeable_file(s->sc_file);
 	}
 
-	return NULL;
+	return ret;
 }
 
 static __be32
Signature · 1 line
2.39.0
Signature · 1 line
Jeff Layton <[email protected]>