Skip to contentSkip to conversation

nfsd: plug some filecache refcount leaks

PATCH In review Jeff Layton 3 participants 11 messages Jul 10, 2024 – Jul 11, 2024

Subsystems (2)
lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

cover nfsd: plug some filecache refcount leaks

Youzhong Yang sent an email to the list (along with some draft patches) that indicated some nfsd_file refcount leaks. I went crawling over the filecache code (again) and found a couple of places where we didn't put references when we should. I'm not sure if it'll fix the problem they reported, but they are bugs.

Plus, let's start counting nfsd_file allocations. The last patch adds support for this.

---
Jeff Layton (3):
nfsd: fix refcount leak when failing to hash nfsd_file
nfsd: fix refcount leak when file is unhashed after being found
nfsd: count nfsd_file allocations

 fs/nfsd/filecache.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

---
base-commit: 24decb225ed20a5ba46a79f4609e109cb0e4a359
change-id: 20240710-nfsd-next-01e2afdebb31

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>
author maintainer
#1 lore raw

PATCH 1/3 nfsd: fix refcount leak when failing to hash nfsd_file

At this point, we have a new nf that we couldn't properly insert into the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of lease break error")
---
 fs/nfsd/filecache.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -1
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (likely(ret == 0))
 		goto open_file;
 
-	if (ret == -EEXIST)
+	if (ret == -EEXIST) {
+		nfsd_file_free(nf);
 		goto retry;
+	}
 	trace_nfsd_file_insert_err(rqstp, inode, may_flags, ret);
 	status = nfserr_jukebox;
 	goto construction_err;
Signature · 1 line
2.45.2
#2 lore raw

Re: [PATCH 1/3] nfsd: fix refcount leak when failing to hash nfsd_file

Shouldn't we have fh_put(fhp) before 'retry'?

Quoted 30 lines · Jeff Layton

At this point, we have a new nf that we couldn't properly insert into the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of lease break error")
---
 fs/nfsd/filecache.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -2
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
        if (likely(ret == 0))
                goto open_file;
 
-       if (ret == -EEXIST)
+       if (ret == -EEXIST) {
+               nfsd_file_free(nf);
                goto retry;
+       }
        trace_nfsd_file_insert_err(rqstp, inode, may_flags, ret);
        status = nfserr_jukebox;
        goto construction_err;
 
--

2.45.2

author maintainer
#3 lore raw

Re: [PATCH 1/3] nfsd: fix refcount leak when failing to hash nfsd_file

Quoted 2 lines · Youzhong Yang

Shouldn't we have fh_put(fhp) before 'retry'?

A subtle question, actually...

It probably wouldn't hurt to do that, but I don't think it's required.

The main reason we call fh_put is to force a second call to nfsd_set_fh_dentry. IOW, we want to redo the lookup by filehandle and find the inode.

In the EEXIST case, presumably we have found the inode but we raced with another task in setting an nfsd_file for it in the hash. That's different from the case where the thing was unhashed or we got an EOPENSTALE.  So, I think we probably don't require refinding the inode in that case.

More pointedly, I'm not sure this particular case is actually possible. The entries are hashed on the inode pointer value, and we're searching and inserting into the hash under the i_lock.

Chuck, thoughts?

Quoted 36 lines
Quoted 34 lines · Jeff Layton

At this point, we have a new nf that we couldn't properly insert into
the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of lease

break error")

---
 fs/nfsd/filecache.c | 4 +++-

 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -1
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp,
 struct svc_fh *fhp,
         if (likely(ret == 0))
                 goto open_file;
 
-       if (ret == -EEXIST)
+       if (ret == -EEXIST) {
+               nfsd_file_free(nf);
                 goto retry;
+       }
         trace_nfsd_file_insert_err(rqstp, inode, may_flags, ret);
         status = nfserr_jukebox;

        goto construction_err;

--
2.45.2

Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw

Re: [PATCH 1/3] nfsd: fix refcount leak when failing to hash nfsd_file

Quoted 22 lines · Jeff Layton
Quoted 2 lines · Youzhong Yang

Shouldn't we have fh_put(fhp) before 'retry'?

A subtle question, actually...

It probably wouldn't hurt to do that, but I don't think it's required.

The main reason we call fh_put is to force a second call to nfsd_set_fh_dentry. IOW, we want to redo the lookup by filehandle and find the inode.

In the EEXIST case, presumably we have found the inode but we raced with another task in setting an nfsd_file for it in the hash. That's different from the case where the thing was unhashed or we got an EOPENSTALE. So, I think we probably don't require refinding the inode in that case.

Quoted 5 lines

More pointedly, I'm not sure this particular case is actually possible. The entries are hashed on the inode pointer value, and we're searching and inserting into the hash under the i_lock.

Chuck, thoughts?

Is the question whether we want to dput() the dentry that is attached to the fhp ? fh_verify's API contract says:

310 * Regardless of success or failure of fh_verify(), fh_put() should be 311 * called on @fhp when the caller is finished with the filehandle.

It looks like none of nfsd_file_acquire's callers do an fh_put() in their error flows.

But maybe I've misunderstood the issue.

Quoted 33 lines
Quoted 33 lines
Quoted 31 lines · Jeff Layton

At this point, we have a new nf that we couldn't properly insert into
the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of lease

break error")

---
 fs/nfsd/filecache.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -1
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp,
 struct svc_fh *fhp,
        if (likely(ret == 0))
                goto open_file;
 
-       if (ret == -EEXIST)
+       if (ret == -EEXIST) {
+               nfsd_file_free(nf);
                goto retry;
+       }
        trace_nfsd_file_insert_err(rqstp, inode, may_flags, ret);
        status = nfserr_jukebox;
        goto construction_err;
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

--
2.45.2

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

author maintainer
#5 lore raw

Re: [PATCH 1/3] nfsd: fix refcount leak when failing to hash nfsd_file

Quoted 43 lines · Chuck Lever III
Quoted 27 lines

On Jul 11, 2024, at 1:53 PM, Jeff Layton <[email protected]> wrote:

Quoted 2 lines · Youzhong Yang

Shouldn't we have fh_put(fhp) before 'retry'?

A subtle question, actually...

It probably wouldn't hurt to do that, but I don't think it's required.

The main reason we call fh_put is to force a second call to nfsd_set_fh_dentry. IOW, we want to redo the lookup by filehandle and
find the inode.

In the EEXIST case, presumably we have found the inode but we raced with another task in setting an nfsd_file for it in the hash. That's
different from the case where the thing was unhashed or we got an EOPENSTALE.  So, I think we probably don't require refinding the inode
in that case.

Quoted 7 lines

More pointedly, I'm not sure this particular case is actually possible.
The entries are hashed on the inode pointer value, and we're searching
and inserting into the hash under the i_lock.

Chuck, thoughts?

Is the question whether we want to dput() the dentry that is attached to the fhp ? fh_verify's API contract says:

310  * Regardless of success or failure of fh_verify(), fh_put() should be
311  * called on @fhp when the caller is finished with the filehandle.

Quoted 5 lines

It looks like none of nfsd_file_acquire's callers do an fh_put() in their error flows.

But maybe I've misunderstood the issue.

Note that this API is weird and doesn't conform to typical get/put semantics.

The fhp is instantiated before nfsd_file_do_acquire is called, and all of the callers that I can see do eventually call fh_put on it. fh_put is idempotent, so there should be no harm in calling it multiple times.

My question here though was more about this EEXIST case. Should we even bother checking for that? I don't see how it's possible.

Quoted 50 lines
Quoted 38 lines
Quoted 38 lines
Quoted 35 lines · Jeff Layton

At this point, we have a new nf that we couldn't properly insert
into
the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of

lease
break error")

---
 fs/nfsd/filecache.c | 4 +++-

 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -1
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst
 *rqstp,
 struct svc_fh *fhp,
         if (likely(ret == 0))
                 goto open_file;
 
-       if (ret == -EEXIST)
+       if (ret == -EEXIST) {
+               nfsd_file_free(nf);
                 goto retry;
+       }
         trace_nfsd_file_insert_err(rqstp, inode, may_flags,

ret);
        status = nfserr_jukebox;
        goto construction_err;

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

--
2.45.2

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

Signature · 1 line
Jeff Layton <[email protected]>
#6 lore raw

Re: [PATCH 1/3] nfsd: fix refcount leak when failing to hash nfsd_file

Quoted 57 lines · Jeff Layton
Quoted 43 lines · Chuck Lever III
Quoted 27 lines

On Jul 11, 2024, at 1:53 PM, Jeff Layton <[email protected]> wrote:

Quoted 2 lines · Youzhong Yang

Shouldn't we have fh_put(fhp) before 'retry'?

A subtle question, actually...

It probably wouldn't hurt to do that, but I don't think it's required.

The main reason we call fh_put is to force a second call to nfsd_set_fh_dentry. IOW, we want to redo the lookup by filehandle and
find the inode.

In the EEXIST case, presumably we have found the inode but we raced with another task in setting an nfsd_file for it in the hash. That's
different from the case where the thing was unhashed or we got an EOPENSTALE. So, I think we probably don't require refinding the inode
in that case.

Quoted 7 lines

More pointedly, I'm not sure this particular case is actually possible.
The entries are hashed on the inode pointer value, and we're searching
and inserting into the hash under the i_lock.

Chuck, thoughts?

Is the question whether we want to dput() the dentry that is attached to the fhp ? fh_verify's API contract says:

310 * Regardless of success or failure of fh_verify(), fh_put() should be
311 * called on @fhp when the caller is finished with the filehandle.

Quoted 5 lines

It looks like none of nfsd_file_acquire's callers do an fh_put() in their error flows.

But maybe I've misunderstood the issue.

Note that this API is weird and doesn't conform to typical get/put semantics.

Quoted 6 lines

The fhp is instantiated before nfsd_file_do_acquire is called, and all of the callers that I can see do eventually call fh_put on it. fh_put is idempotent, so there should be no harm in calling it multiple times.

My question here though was more about this EEXIST case. Should we even bother checking for that? I don't see how it's possible.

If memory serves, at one point nfsd_file_acquire() used rhtable_insert_yada(), which returns -EEXIST for certain table overflow cases. Possibly with the list version of rhtable, consumers can't get -EEXIST at all.

Quoted 45 lines
Quoted 45 lines
Quoted 43 lines
Quoted 40 lines
Quoted 38 lines · Jeff Layton

At this point, we have a new nf that we couldn't properly insert
into
the hashtable. Just free it before retrying, since it was never hashed.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of

lease
break error")

---
 fs/nfsd/filecache.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/nfsd/filecache.c +3 -1
index f84913691b78..4fb5e8546831 100644
@@ -1038,8 +1038,10 @@ nfsd_file_do_acquire(struct svc_rqst
 *rqstp,
 struct svc_fh *fhp,
        if (likely(ret == 0))
                goto open_file;
 
-       if (ret == -EEXIST)
+       if (ret == -EEXIST) {
+               nfsd_file_free(nf);
                goto retry;
+       }
        trace_nfsd_file_insert_err(rqstp, inode, may_flags,
ret);
        status = nfserr_jukebox;
        goto construction_err;

--
2.45.2

Signature · 1 line
Jeff Layton <[email protected]>
Quoted 7 lines
Quoted 4 lines

--
Chuck Lever

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

author maintainer
#7 lore raw

PATCH 2/3 nfsd: fix refcount leak when file is unhashed after being found

In reply to Jeff Layton

If we wait_for_construction and find that the file is no longer hashed, and we're going to retry the open, the old nfsd_file reference is currently leaked. Put the reference before retrying.

Fixes c6593366c0bf ("nfsd: don't kill nfsd_files because of lease break error")
---
 fs/nfsd/filecache.c | 1 +
 1 file changed, 1 insertion(+)
fs/nfsd/filecache.c +1 -0
index 4fb5e8546831..52063f2cf0df 100644
@@ -1056,6 +1056,7 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			status = nfserr_jukebox;
 			goto construction_err;
 		}
+		nfsd_file_put(nf);
 		open_retry = false;
 		fh_put(fhp);
 		goto retry;
Signature · 1 line
2.45.2
author maintainer
#8 lore raw

PATCH 3/3 nfsd: count nfsd_file allocations

In reply to Jeff Layton

We already count the frees (via nfsd_file_releases). Count the allocations as well. Also switch the direct call to nfsd_file_slab_free in nfsd_file_do_acquire to nfsd_file_free, so that the allocs and releases match up.

---
 fs/nfsd/filecache.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
fs/nfsd/filecache.c +7 -2
index 52063f2cf0df..8997058ddbcb 100644
@@ -56,6 +56,7 @@
 
 static DEFINE_PER_CPU(unsigned long, nfsd_file_cache_hits);
 static DEFINE_PER_CPU(unsigned long, nfsd_file_acquisitions);
+static DEFINE_PER_CPU(unsigned long, nfsd_file_allocations);
 static DEFINE_PER_CPU(unsigned long, nfsd_file_releases);
 static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age);
 static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions);
@@ -215,6 +216,7 @@ nfsd_file_alloc(struct net *net, struct inode *inode, unsigned char need,
 	if (unlikely(!nf))
 		return NULL;
 
+	this_cpu_inc(nfsd_file_allocations);
 	INIT_LIST_HEAD(&nf->nf_lru);
 	INIT_LIST_HEAD(&nf->nf_gc);
 	nf->nf_birthtime = ktime_get();
@@ -912,6 +914,7 @@ nfsd_file_cache_shutdown(void)
 	for_each_possible_cpu(i) {
 		per_cpu(nfsd_file_cache_hits, i) = 0;
 		per_cpu(nfsd_file_acquisitions, i) = 0;
+		per_cpu(nfsd_file_allocations, i) = 0;
 		per_cpu(nfsd_file_releases, i) = 0;
 		per_cpu(nfsd_file_total_age, i) = 0;
 		per_cpu(nfsd_file_evictions, i) = 0;
@@ -1027,7 +1030,7 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
 	if (unlikely(nf)) {
 		spin_unlock(&inode->i_lock);
 		rcu_read_unlock();
-		nfsd_file_slab_free(&new->nf_rcu);
+		nfsd_file_free(new);
 		goto wait_for_construction;
 	}
 	nf = new;
@@ -1205,7 +1208,7 @@ nfsd_file_acquire_opened(struct svc_rqst *rqstp, struct svc_fh *fhp,
  */
 int nfsd_file_cache_stats_show(struct seq_file *m, void *v)
 {
-	unsigned long releases = 0, evictions = 0;
+	unsigned long allocations = 0, releases = 0, evictions = 0;
 	unsigned long hits = 0, acquisitions = 0;
 	unsigned int i, count = 0, buckets = 0;
 	unsigned long lru = 0, total_age = 0;
@@ -1230,6 +1233,7 @@ int nfsd_file_cache_stats_show(struct seq_file *m, void *v)
 	for_each_possible_cpu(i) {
 		hits += per_cpu(nfsd_file_cache_hits, i);
 		acquisitions += per_cpu(nfsd_file_acquisitions, i);
+		allocations += per_cpu(nfsd_file_allocations, i);
 		releases += per_cpu(nfsd_file_releases, i);
 		total_age += per_cpu(nfsd_file_total_age, i);
 		evictions += per_cpu(nfsd_file_evictions, i);
@@ -1240,6 +1244,7 @@ int nfsd_file_cache_stats_show(struct seq_file *m, void *v)
 	seq_printf(m, "lru entries:   %lu\n", lru);
 	seq_printf(m, "cache hits:    %lu\n", hits);
 	seq_printf(m, "acquisitions:  %lu\n", acquisitions);
+	seq_printf(m, "allocations:   %lu\n", allocations);
 	seq_printf(m, "releases:      %lu\n", releases);
 	seq_printf(m, "evictions:     %lu\n", evictions);
 	if (releases)
Signature · 1 line
2.45.2
#9 lore raw
In reply to Jeff Layton
Quoted 16 lines · Jeff Layton

Youzhong Yang sent an email to the list (along with some draft patches) that indicated some nfsd_file refcount leaks. I went crawling over the filecache code (again) and found a couple of places where we didn't put references when we should. I'm not sure if it'll fix the problem they reported, but they are bugs.

Plus, let's start counting nfsd_file allocations. The last patch adds support for this.

---
Jeff Layton (3):
nfsd: fix refcount leak when failing to hash nfsd_file
nfsd: fix refcount leak when file is unhashed after being found
nfsd: count nfsd_file allocations

Quoted 10 lines
 fs/nfsd/filecache.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

---
base-commit: 24decb225ed20a5ba46a79f4609e109cb0e4a359
change-id: 20240710-nfsd-next-01e2afdebb31

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>

I've browsed these, they seem straightforward.

Since this is the week before a merge window, I would like to save these and Youzhong's patch for v6.12. The Fixes: tags will get them pulled back into the stable kernels once they are merged.

'Salright?

Signature · 1 line
Chuck Lever
author maintainer
#10 lore raw
Quoted 34 lines · Chuck Lever
Quoted 22 lines · Jeff Layton

Youzhong Yang sent an email to the list (along with some draft patches)
that indicated some nfsd_file refcount leaks. I went crawling over the
filecache code (again) and found a couple of places where we didn't put
references when we should. I'm not sure if it'll fix the problem they
reported, but they are bugs.

Plus, let's start counting nfsd_file allocations. The last patch adds
support for this.

---
Jeff Layton (3):
      nfsd: fix refcount leak when failing to hash nfsd_file
      nfsd: fix refcount leak when file is unhashed after being found
      nfsd: count nfsd_file allocations

Quoted 10 lines

 fs/nfsd/filecache.c | 14 +++++++++++---

 1 file changed, 11 insertions(+), 3 deletions(-)

---
base-commit: 24decb225ed20a5ba46a79f4609e109cb0e4a359
change-id: 20240710-nfsd-next-01e2afdebb31

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>
Quoted 8 lines

I've browsed these, they seem straightforward.

Since this is the week before a merge window, I would like to save these and Youzhong's patch for v6.12. The Fixes: tags will get them pulled back into the stable kernels once they are merged.

'Salright?

Agreed. Leaks suck, but they (usually) aren't fatal, and interactions in the filecache can be...subtle. I wouldn't mind a bit more testing before we send this on.

Thanks!

Signature · 1 line
Jeff Layton <[email protected]>