Skip to contentSkip to conversation

nfsd: COPY refcounting fix and cleanup

PATCH In review Jeff Layton 4 participants 31 messages Jan 17, 2023 – Jan 23, 2023

Subsystems (2)
lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

cover nfsd: COPY refcounting fix and cleanup

Just a couple of fixes for refcounting issues in the COPY codepaths. I sort of doubt either of these are the cause of the more recent crash reports around nfsd_file refcount underflows, but they do seem like real (and potential) bugs.

Jeff Layton (2):
nfsd: zero out pointers after putting nfsd_files on COPY setup error
nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

 fs/nfsd/nfs4proc.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)
Signature · 1 line
2.39.0
author maintainer
#1 lore raw

PATCH 1/2 nfsd: zero out pointers after putting nfsd_files on COPY setup error

At first, I thought this might be a source of nfsd_file overputs, but the current callers seem to avoid an extra put when nfsd4_verify_copy returns an error.

Still, it's "bad form" to leave the pointers filled out when we don't have a reference to them anymore, and that might lead to bugs later. Zero them out as a defensive coding measure.

---
 fs/nfsd/nfs4proc.c | 2 ++
 1 file changed, 2 insertions(+)
fs/nfsd/nfs4proc.c +2 -0
index dbaf33398c82..37a9cc8ae7ae 100644
@@ -1214,8 +1214,10 @@ nfsd4_verify_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	return status;
 out_put_dst:
 	nfsd_file_put(*dst);
+	*dst = NULL;
 out_put_src:
 	nfsd_file_put(*src);
+	*src = NULL;
 	goto out;
 }
 
Signature · 1 line
2.39.0
author maintainer
#2 lore raw

PATCH 2/2 nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

In reply to Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +10 -13
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
 	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
 	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);
 	fput(filp);
 
 	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
 				 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
 	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
 	dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
 	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
 	spin_lock(&copy->cp_clp->async_lock);
 	list_del(&copy->copies);
 	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
 	} else {
 		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
 				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
 	}
 
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
 	} else {
 		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
 				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
 	}
 out:
+	release_copy_files(copy);
 	return status;
 out_err:
 	if (async_copy)
Signature · 1 line
2.39.0
#3 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 11 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

Quoted 75 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +10 -14
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--

2.39.0

author maintainer
#4 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Quoted 4 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

Quoted 8 lines

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

Quoted 11 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

Quoted 76 lines
Quoted 76 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +10 -14
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--

2.39.0

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 4 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

Quoted 8 lines

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

Quoted 6 lines
Quoted 2 lines

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

#6 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 49 lines · Olga Kornievskaia
Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

       if (async_copy)
               cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 7 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

In cases like this, it makes sense to consider using types to ensure the code can't do the wrong thing. So you might want to have a struct nfs4_copy_A for the inter code to use, and a struct nfs4_copy_B for the intra code to use. Sharing the same struct for both use cases is probably what's confusing to human readers.

I've never been a stickler for removing every last ounce of code duplication. Here, it might help to have a little duplication just to make it easier to reason about the reference counting in the two use cases.

That's my view from the mountain top, worth every penny you paid for it.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

Quoted 7 lines
Quoted 3 lines

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

--
Chuck Lever

author maintainer
#7 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 62 lines · Chuck Lever III
Quoted 48 lines · Olga Kornievskaia
Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

       if (async_copy)
               cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 7 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

In cases like this, it makes sense to consider using types to ensure the code can't do the wrong thing. So you might want to have a struct nfs4_copy_A for the inter code to use, and a struct nfs4_copy_B for the intra code to use. Sharing the same struct for both use cases is probably what's confusing to human readers.

Quoted 8 lines

I've never been a stickler for removing every last ounce of code duplication. Here, it might help to have a little duplication just to make it easier to reason about the reference counting in the two use cases.

That's my view from the mountain top, worth every penny you paid for it.

+1

The nfsd4_copy structure has a lot of fields in it that only matter for the async copy case. ISTM that nfsd4_copy (the function) should dynamically allocate a struct nfsd4_async_copy that contains a nfsd4_copy and whatever other fields are needed.

Then, we could trim down struct nfsd4_copy to just the info needed.

For instance, the nf_src and nf_dst fields really don't need to be in nfsd4_copy. For the synchronous copy case, we can just keep those pointers on the stack, and for the async case they would be inside the larger structure.

That would allow us to trim down the footprint of the compound union too.

Signature · 1 line
Jeff Layton <[email protected]>
#8 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 76 lines · Jeff Layton
Quoted 62 lines · Chuck Lever III
Quoted 48 lines · Olga Kornievskaia
Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

      if (async_copy)
              cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 7 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

In cases like this, it makes sense to consider using types to ensure the code can't do the wrong thing. So you might want to have a struct nfs4_copy_A for the inter code to use, and a struct nfs4_copy_B for the intra code to use. Sharing the same struct for both use cases is probably what's confusing to human readers.

Quoted 8 lines

I've never been a stickler for removing every last ounce of code duplication. Here, it might help to have a little duplication just to make it easier to reason about the reference counting in the two use cases.

That's my view from the mountain top, worth every penny you paid for it.

Quoted 8 lines

+1

The nfsd4_copy structure has a lot of fields in it that only matter for the async copy case. ISTM that nfsd4_copy (the function) should dynamically allocate a struct nfsd4_async_copy that contains a nfsd4_copy and whatever other fields are needed.

Then, we could trim down struct nfsd4_copy to just the info needed.

Yeah, some of those fields are actually quite large, like filehandles.

Quoted 7 lines

For instance, the nf_src and nf_dst fields really don't need to be in nfsd4_copy. For the synchronous copy case, we can just keep those pointers on the stack, and for the async case they would be inside the larger structure.

That would allow us to trim down the footprint of the compound union too.

That seems sensible. Do you feel like redriving this clean-up series with the changes you describe above?

--
Chuck Lever

author maintainer
#9 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 92 lines · Chuck Lever III
Quoted 75 lines · Jeff Layton
Quoted 62 lines · Chuck Lever III
Quoted 48 lines · Olga Kornievskaia
Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

      if (async_copy)
              cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 7 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

In cases like this, it makes sense to consider using types to ensure the code can't do the wrong thing. So you might want to have a struct nfs4_copy_A for the inter code to use, and a struct nfs4_copy_B for the intra code to use. Sharing the same struct for both use cases is probably what's confusing to human readers.

Quoted 8 lines

I've never been a stickler for removing every last ounce of code duplication. Here, it might help to have a little duplication just to make it easier to reason about the reference counting in the two use cases.

That's my view from the mountain top, worth every penny you paid for it.

Quoted 8 lines

+1

The nfsd4_copy structure has a lot of fields in it that only matter for the async copy case. ISTM that nfsd4_copy (the function) should dynamically allocate a struct nfsd4_async_copy that contains a nfsd4_copy and whatever other fields are needed.

Then, we could trim down struct nfsd4_copy to just the info needed.

Yeah, some of those fields are actually quite large, like filehandles.

Quoted 5 lines

For instance, the nf_src and nf_dst fields really don't need to be in nfsd4_copy. For the synchronous copy case, we can just keep those pointers on the stack, and for the async case they would be inside the larger structure.

Quoted 6 lines
Quoted 2 lines

That would allow us to trim down the footprint of the compound union too.

That seems sensible. Do you feel like redriving this clean-up series with the changes you describe above?

I can, unless Olga, Dai or someone else would rather do it. Not sure how soon I can get to it though.

Signature · 1 line
Jeff Layton <[email protected]>
#10 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 97 lines · Jeff Layton
Quoted 95 lines · Chuck Lever III
Quoted 75 lines · Jeff Layton
Quoted 62 lines · Chuck Lever III
Quoted 48 lines · Olga Kornievskaia
Quoted 21 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

Quoted 7 lines

out_err:

      if (async_copy)
              cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 7 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

In cases like this, it makes sense to consider using types to ensure the code can't do the wrong thing. So you might want to have a struct nfs4_copy_A for the inter code to use, and a struct nfs4_copy_B for the intra code to use. Sharing the same struct for both use cases is probably what's confusing to human readers.

Quoted 8 lines

I've never been a stickler for removing every last ounce of code duplication. Here, it might help to have a little duplication just to make it easier to reason about the reference counting in the two use cases.

That's my view from the mountain top, worth every penny you paid for it.

Quoted 8 lines

+1

The nfsd4_copy structure has a lot of fields in it that only matter for the async copy case. ISTM that nfsd4_copy (the function) should dynamically allocate a struct nfsd4_async_copy that contains a nfsd4_copy and whatever other fields are needed.

Then, we could trim down struct nfsd4_copy to just the info needed.

Yeah, some of those fields are actually quite large, like filehandles.

Quoted 7 lines

For instance, the nf_src and nf_dst fields really don't need to be in nfsd4_copy. For the synchronous copy case, we can just keep those pointers on the stack, and for the async case they would be inside the larger structure.

That would allow us to trim down the footprint of the compound union too.

Quoted 6 lines
Quoted 3 lines

That seems sensible. Do you feel like redriving this clean-up series with the changes you describe above?

I can, unless Olga, Dai or someone else would rather do it. Not sure how soon I can get to it though.

I'm not going to volunteer as I don't believe in the suggested change.

I think there is a performance advantage to having this structure preallocated and ready for use.

Quoted 2 lines

--
Jeff Layton <[email protected]>

author maintainer
#11 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

In reply to Olga Kornievskaia
Quoted 26 lines · Olga Kornievskaia
Quoted 25 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);
Quoted 6 lines
Quoted 3 lines

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Can you please point out where do we take a reference on the original copy?

In the case of an inter-server copy, nf_dst is set in nfsd4_setup_inter_ssc. For intraserver copy, both pointers are set via the call to nfsd4_verify_copy. Both functions call nfs4_preprocess_stateid_op, which returns a reference to the nfsd_file in the second to last arg.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 3 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

I sure as hell had a hard time dissecting how all of that was supposed to work. There is clear bug here, and I think this patch makes the result clearer and more robust in the face of changes.

There are actually 4 different cases here: sync vs. async, alongside intra vs. interserver copy. These are all overloaded onto a nfsd4_copy structure, seemingly for no good reason.

The cleanup, in particular seems quite fragile to me, and there is a dearth of defensive coding measures. If you subtly call the "wrong" cleanup function at the wrong point in time, then things may go awry.

I'll leave it up to Chuck to make the final determination, but I see this patch as an improvement.

Quoted 100 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

Quoted 6 lines
Quoted 2 lines

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

Signature · 1 line
Jeff Layton <[email protected]>
#12 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 32 lines · Jeff Layton
Quoted 30 lines · Olga Kornievskaia
Quoted 28 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Quoted 8 lines
Quoted 2 lines

Can you please point out where do we take a reference on the original copy?

In the case of an inter-server copy, nf_dst is set in nfsd4_setup_inter_ssc. For intraserver copy, both pointers are set via the call to nfsd4_verify_copy. Both functions call nfs4_preprocess_stateid_op, which returns a reference to the nfsd_file in the second to last arg.

Ah. Thank you. I didn't know that nfs4_preprocess_stateid_op() takes a reference on it's 5th argument. I think I was previously looking at nfsd4_read() function which calls nfs4_preprocess_stateid_op() and gets back read->rd_nf but it never does a put on it when it returns. However, I now looked at other functions that call nfs4_preproess_stateid_op() such as nfsd4_fallocate() and I see that it does a put.

Quoted 142 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

I sure as hell had a hard time dissecting how all of that was supposed to work. There is clear bug here, and I think this patch makes the result clearer and more robust in the face of changes.

There are actually 4 different cases here: sync vs. async, alongside intra vs. interserver copy. These are all overloaded onto a nfsd4_copy structure, seemingly for no good reason.

The cleanup, in particular seems quite fragile to me, and there is a dearth of defensive coding measures. If you subtly call the "wrong" cleanup function at the wrong point in time, then things may go awry.

I'll leave it up to Chuck to make the final determination, but I see this patch as an improvement.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

Quoted 88 lines
Quoted 3 lines

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

--
Jeff Layton <[email protected]>

#13 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 43 lines · Olga Kornievskaia
Quoted 32 lines · Jeff Layton
Quoted 30 lines · Olga Kornievskaia
Quoted 28 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Quoted 8 lines
Quoted 2 lines

Can you please point out where do we take a reference on the original copy?

In the case of an inter-server copy, nf_dst is set in nfsd4_setup_inter_ssc. For intraserver copy, both pointers are set via the call to nfsd4_verify_copy. Both functions call nfs4_preprocess_stateid_op, which returns a reference to the nfsd_file in the second to last arg.

Quoted 7 lines

Ah. Thank you. I didn't know that nfs4_preprocess_stateid_op() takes a reference on it's 5th argument. I think I was previously looking at nfsd4_read() function which calls nfs4_preprocess_stateid_op() and gets back read->rd_nf but it never does a put on it when it returns. However, I now looked at other functions that call nfs4_preproess_stateid_op() such as nfsd4_fallocate() and I see that it does a put.

So is there a refcount leak in the nfsd4_read() then since it doesn't do a put? Or the internals obscure and that even though it calls the same function and passes that parameter no refcount was increased. Is it based on the "WR_STATE, RD_STATE" parameter. I see that nfsd4_write() does do a put. For copy, we call the src_fd with RD_STATE and dst_fd with WR_STATE. If I were to follow the logic of nfsd4_read/nfsd4_write, the the copy doesn't need to do a put for src but will need it for the dst. The proposed patch does it for both.

So I'm still confused if this patch is the correct solution.

Quoted 142 lines
Quoted 142 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

I sure as hell had a hard time dissecting how all of that was supposed to work. There is clear bug here, and I think this patch makes the result clearer and more robust in the face of changes.

There are actually 4 different cases here: sync vs. async, alongside intra vs. interserver copy. These are all overloaded onto a nfsd4_copy structure, seemingly for no good reason.

The cleanup, in particular seems quite fragile to me, and there is a dearth of defensive coding measures. If you subtly call the "wrong" cleanup function at the wrong point in time, then things may go awry.

I'll leave it up to Chuck to make the final determination, but I see this patch as an improvement.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

Quoted 88 lines
Quoted 3 lines

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

--
Jeff Layton <[email protected]>

author maintainer
#14 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 52 lines · Olga Kornievskaia
Quoted 43 lines · Olga Kornievskaia
Quoted 32 lines · Jeff Layton
Quoted 30 lines · Olga Kornievskaia
Quoted 28 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Quoted 8 lines
Quoted 2 lines

Can you please point out where do we take a reference on the original copy?

In the case of an inter-server copy, nf_dst is set in nfsd4_setup_inter_ssc. For intraserver copy, both pointers are set via the call to nfsd4_verify_copy. Both functions call nfs4_preprocess_stateid_op, which returns a reference to the nfsd_file in the second to last arg.

Quoted 7 lines

Ah. Thank you. I didn't know that nfs4_preprocess_stateid_op() takes a reference on it's 5th argument. I think I was previously looking at nfsd4_read() function which calls nfs4_preprocess_stateid_op() and gets back read->rd_nf but it never does a put on it when it returns. However, I now looked at other functions that call nfs4_preproess_stateid_op() such as nfsd4_fallocate() and I see that it does a put.

Quoted 5 lines

So is there a refcount leak in the nfsd4_read() then since it doesn't do a put? Or the internals obscure and that even though it calls the same function and passes that parameter no refcount was increased. Is it based on the "WR_STATE, RD_STATE" parameter.

I don't think so. The put is done just below there, in nfsd4_read_release:

        if (u->read.rd_nf)
                nfsd_file_put(u->read.rd_nf);

That said, I am hunting a refcount overput with nfsd_files that I've not been able to nail down yet (which is why I've been auditing the nfsd_file refcounting). If you see anything that looks hinky, please do point it out.

Quoted 6 lines

I see that
nfsd4_write() does do a put. For copy, we call the src_fd with RD_STATE and dst_fd with WR_STATE. If I were to follow the logic of nfsd4_read/nfsd4_write, the the copy doesn't need to do a put for src but will need it for the dst. The proposed patch does it for both.

That'd be wrong. READs have to hold a ref to the open file while the reply is being marshalled. A WRITE can release it once the data has been written to the file. Maybe that's worth a comment.

Note that I just sent a patch to the list to add a comment that (hopefully) makes it clear that nfs4_preprocess_stateid_op returns a reference in that field.

FWIW, it wouldn't be safe for it to do anything else. Returning a pointer to a refcounted object without taking a reference would be very problematic.

Quoted 2 lines

So I'm still confused if this patch is the correct solution.

Fair enough. I'm not sure I understand the pushback, as the result seems clearer to me. If you want to propose an alternate fix, I'm happy to take a look.

Quoted 142 lines
Quoted 142 lines
Quoted 142 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

I sure as hell had a hard time dissecting how all of that was supposed to work. There is clear bug here, and I think this patch makes the result clearer and more robust in the face of changes.

There are actually 4 different cases here: sync vs. async, alongside intra vs. interserver copy. These are all overloaded onto a nfsd4_copy structure, seemingly for no good reason.

The cleanup, in particular seems quite fragile to me, and there is a dearth of defensive coding measures. If you subtly call the "wrong" cleanup function at the wrong point in time, then things may go awry.

I'll leave it up to Chuck to make the final determination, but I see this patch as an improvement.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

Quoted 88 lines
Quoted 3 lines

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

--
Jeff Layton <[email protected]>

Signature · 1 line
Jeff Layton <[email protected]>
#15 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 63 lines · Jeff Layton
Quoted 52 lines · Olga Kornievskaia
Quoted 43 lines · Olga Kornievskaia
Quoted 32 lines · Jeff Layton
Quoted 30 lines · Olga Kornievskaia
Quoted 28 lines · Jeff Layton
Quoted 9 lines · Olga Kornievskaia
Quoted 8 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

Quoted 7 lines
Quoted 3 lines

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

I don't believe that's true. If kthread_create thread fails we call cleanup_async_copy() that does a put on the file descriptors.

You mean this?

out_err:

        if (async_copy)
                cleanup_async_copy(async_copy);

That puts the references that were taken in dup_copy_fields, but the original (embedded) nfsd4_copy also holds references and those are not being put in this codepath.

Quoted 8 lines
Quoted 2 lines

Can you please point out where do we take a reference on the original copy?

In the case of an inter-server copy, nf_dst is set in nfsd4_setup_inter_ssc. For intraserver copy, both pointers are set via the call to nfsd4_verify_copy. Both functions call nfs4_preprocess_stateid_op, which returns a reference to the nfsd_file in the second to last arg.

Quoted 7 lines

Ah. Thank you. I didn't know that nfs4_preprocess_stateid_op() takes a reference on it's 5th argument. I think I was previously looking at nfsd4_read() function which calls nfs4_preprocess_stateid_op() and gets back read->rd_nf but it never does a put on it when it returns. However, I now looked at other functions that call nfs4_preproess_stateid_op() such as nfsd4_fallocate() and I see that it does a put.

Quoted 5 lines

So is there a refcount leak in the nfsd4_read() then since it doesn't do a put? Or the internals obscure and that even though it calls the same function and passes that parameter no refcount was increased. Is it based on the "WR_STATE, RD_STATE" parameter.

I don't think so. The put is done just below there, in nfsd4_read_release:

Quoted 7 lines
        if (u->read.rd_nf)
                nfsd_file_put(u->read.rd_nf);

That said, I am hunting a refcount overput with nfsd_files that I've not been able to nail down yet (which is why I've been auditing the nfsd_file refcounting). If you see anything that looks hinky, please do point it out.

Thank you for the explanation about the read. I didn't know about the op_release.

Quoted 19 lines
Quoted 6 lines

I see that
nfsd4_write() does do a put. For copy, we call the src_fd with RD_STATE and dst_fd with WR_STATE. If I were to follow the logic of nfsd4_read/nfsd4_write, the the copy doesn't need to do a put for src but will need it for the dst. The proposed patch does it for both.

That'd be wrong. READs have to hold a ref to the open file while the reply is being marshalled. A WRITE can release it once the data has been written to the file. Maybe that's worth a comment.

Note that I just sent a patch to the list to add a comment that (hopefully) makes it clear that nfs4_preprocess_stateid_op returns a reference in that field.

FWIW, it wouldn't be safe for it to do anything else. Returning a pointer to a refcounted object without taking a reference would be very problematic.

Quoted 6 lines
Quoted 2 lines

So I'm still confused if this patch is the correct solution.

Fair enough. I'm not sure I understand the pushback, as the result seems clearer to me. If you want to propose an alternate fix, I'm happy to take a look.

After your explanation about the read, things make sense to me so, no pushback.

Quoted 145 lines
Quoted 140 lines
Quoted 140 lines
Quoted 140 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure).

That's not true. There is a careful distinction between intra -- which had 2 valid file pointers and does a get on both as they both point to something that's opened on this server--- but inter -- only does a get on the dst file descriptor, the src doesn't exit. And yes I realize the code checks for nfs_src being null which it should be but it makes the code less clear and at some point somebody might want to decide to really do a put on it.

Quoted 8 lines
Quoted 3 lines

This is part of the problem here. We have a nfsd4_copy structure, and depending on what has been done to it, you need to call different methods to clean it up. That seems like a real antipattern to me.

But they call different methods because different things need to be done there and it makes it clear what needs to be for what type of copy.

I sure as hell had a hard time dissecting how all of that was supposed to work. There is clear bug here, and I think this patch makes the result clearer and more robust in the face of changes.

There are actually 4 different cases here: sync vs. async, alongside intra vs. interserver copy. These are all overloaded onto a nfsd4_copy structure, seemingly for no good reason.

The cleanup, in particular seems quite fragile to me, and there is a dearth of defensive coding measures. If you subtly call the "wrong" cleanup function at the wrong point in time, then things may go awry.

I'll leave it up to Chuck to make the final determination, but I see this patch as an improvement.

Quoted 100 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

They are both put at the end of
nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

Quoted 6 lines
Quoted 2 lines

I feel by combining the cleanup for both it obscures a very important destication that src filehandle doesn't exist for inter.

If the src filehandle doesn't exist, then the pointer to it will be NULL. I don't see what we gain by keeping these two distinct, other than avoiding a NULL pointer check.

My reason would be for code clarity because different things are supposed to happen for intra and inter. Difference of opinion it seems.

Quoted 80 lines
Quoted 62 lines
Quoted 62 lines
---
 fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +9 -12
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
        nfs42_ssc_close(filp);
-       nfsd_file_put(dst);
        fput(filp);
 
        spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                                 &copy->nf_dst);
 }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-       nfsd_file_put(src);
-       nfsd_file_put(dst);
-}
-
 static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
 {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
 }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+       if (copy->nf_src)
+               nfsd_file_put(copy->nf_src);
+       if (copy->nf_dst)
+               nfsd_file_put(copy->nf_dst);
+}
+
 static void cleanup_async_copy(struct nfsd4_copy *copy)
 {
        nfs4_free_copy_state(copy);
-       nfsd_file_put(copy->nf_dst);
-       if (!nfsd4_ssc_is_inter(copy))
-               nfsd_file_put(copy->nf_src);
+       release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);
        spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
                nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, false);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 
Quoted 15 lines
Quoted 13 lines
fs/nfsd/nfs4proc.c +1 -2
 do_callback:
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        } else {
                status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                                       copy->nf_dst->nf_file, true);
-               nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
 out:
+       release_copy_files(copy);
        return status;
 out_err:
        if (async_copy)
--
Quoted 2 lines

2.39.0

--
Jeff Layton <[email protected]>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
Jeff Layton <[email protected]>

--
Jeff Layton <[email protected]>

#16 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

In reply to Jeff Layton
Quoted 32 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
  fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
  1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +0 -0
index 37a9cc8ae7ae..62b9d6c1b18b 100644

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, · line 1515

Quoted 4 lines
fs/nfsd/nfs4proc.c +0 -1
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
  	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
  	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

Quoted 48 lines
fs/nfsd/nfs4proc.c +9 -11
  	fput(filp);
 
  	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
  				 &copy->nf_dst);
  }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
  static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
  {
  	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
  	dst->ss_nsui = src->ss_nsui;
  }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
  static void cleanup_async_copy(struct nfsd4_copy *copy)
  {
  	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
  	spin_lock(&copy->cp_clp->async_lock);
  	list_del(&copy->copies);
  	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
  	} else {
  		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
  				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
  	}
 
  do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1815

Quoted 9 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
  	} else {
  		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
  				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
  	}
  out:
+	release_copy_files(copy);
  	return status;
  out_err:

This is unrelated to the reference count issue.

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

-Dai

Quoted 1 line
fs/nfsd/nfs4proc.c +0 -0
  	if (async_copy)
author maintainer
#17 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 35 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 34 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
  fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
  1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +0 -0
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
  	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, · line 1515

Quoted 2 lines
fs/nfsd/nfs4proc.c +0 -1
  	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Quoted 60 lines
Quoted 48 lines
fs/nfsd/nfs4proc.c +9 -11
  	fput(filp);
 
  	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
  				 &copy->nf_dst);
  }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
  static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
  {
  	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
  	dst->ss_nsui = src->ss_nsui;
  }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
  static void cleanup_async_copy(struct nfsd4_copy *copy)
  {
  	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
  	spin_lock(&copy->cp_clp->async_lock);
  	list_del(&copy->copies);
  	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
  	} else {
  		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
  				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
  	}
 
  do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1815

Quoted 9 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
  	} else {
  		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
  				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
  	}
  out:
+	release_copy_files(copy);
  	return status;
  out_err:

This is unrelated to the reference count issue.

Quoted 4 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +10 -3
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
        struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
-       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
 
        spin_lock(&nn->nfsd_ssc_lo
        list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
        release_copy_files(copy);
        return status;
 out_err:
-       if (async_copy)
+       if (async_copy) {
                cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))
+                       nfsd4_cleanup_inter_ssc(copy->ss_nsui, NULL,
+                                               copy->nf_dst);
+
+       }
        status = nfserrno(-ENOMEM);
        /*
         * source's vfsmount of inter-copy will be unmounted
Signature · 1 line
Jeff Layton <[email protected]>
#18 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 44 lines · Jeff Layton
Quoted 35 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 34 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
   fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
   1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +0 -0
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
   	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
Quoted 8 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, · line 1515

Quoted 2 lines
fs/nfsd/nfs4proc.c +0 -1
   	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

Quoted 5 lines

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 87 lines
Quoted 61 lines
Quoted 50 lines
fs/nfsd/nfs4proc.c +9 -11
   	fput(filp);
 
   	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
   				 &copy->nf_dst);
   }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
   static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
   {
   	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
   	dst->ss_nsui = src->ss_nsui;
   }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
   static void cleanup_async_copy(struct nfsd4_copy *copy)
   {
   	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
   	spin_lock(&copy->cp_clp->async_lock);
   	list_del(&copy->copies);
   	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
   	} else {
   		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
   				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
   	}
 
   do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1815

Quoted 9 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
   	} else {
   		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
   				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
   	}
   out:
+	release_copy_files(copy);
   	return status;
   out_err:

This is unrelated to the reference count issue.

Quoted 4 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
         struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
-       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
 
         spin_lock(&nn->nfsd_ssc_lo
         list_del(&nsui->nsui_list);

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1820

Quoted 7 lines
fs/nfsd/nfs4proc.c +2 -1
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
         release_copy_files(copy);
         return status;
  out_err:
-       if (async_copy)
+       if (async_copy) {
                 cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

-Dai

Quoted 8 lines
fs/nfsd/nfs4proc.c +4 -0
+                       nfsd4_cleanup_inter_ssc(copy->ss_nsui, NULL,
+                                               copy->nf_dst);
+
+       }
         status = nfserrno(-ENOMEM);
         /*
          * source's vfsmount of inter-copy will be unmounted
author maintainer
#19 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 139 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 44 lines · Jeff Layton
Quoted 35 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 34 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
   fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
   1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +0 -0
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
   	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
Quoted 8 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, · line 1515

Quoted 2 lines
fs/nfsd/nfs4proc.c +0 -1
   	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

Quoted 5 lines

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 84 lines
Quoted 61 lines
Quoted 50 lines
fs/nfsd/nfs4proc.c +9 -11
   	fput(filp);
 
   	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
   				 &copy->nf_dst);
   }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
   static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
   {
   	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
   	dst->ss_nsui = src->ss_nsui;
   }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
   static void cleanup_async_copy(struct nfsd4_copy *copy)
   {
   	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
   	spin_lock(&copy->cp_clp->async_lock);
   	list_del(&copy->copies);
   	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
   	} else {
   		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
   				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
   	}
 
   do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1815

Quoted 9 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
   	} else {
   		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
   				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
   	}
   out:
+	release_copy_files(copy);
   	return status;
   out_err:

This is unrelated to the reference count issue.

Quoted 4 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
         struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
-       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
 
Quoted 17 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1820

Quoted 10 lines
fs/nfsd/nfs4proc.c +2 -1
         spin_lock(&nn->nfsd_ssc_lo
         list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
         release_copy_files(copy);
         return status;
  out_err:
-       if (async_copy)
+       if (async_copy) {
                 cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Signature · 1 line
Jeff Layton <[email protected]>
#20 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 153 lines · Jeff Layton
Quoted 136 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 44 lines · Jeff Layton
Quoted 35 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 34 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
    1 file changed, 10 insertions(+), 13 deletions(-)
fs/nfsd/nfs4proc.c +0 -0
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
    	long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
Quoted 8 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, · line 1515

Quoted 2 lines
fs/nfsd/nfs4proc.c +0 -1
    	nfs42_ssc_close(filp);
-	nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

Quoted 5 lines

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 82 lines
Quoted 60 lines
Quoted 49 lines
fs/nfsd/nfs4proc.c +9 -11
    	fput(filp);
 
    	spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
    				 &copy->nf_dst);
    }
 
-static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)
-{
-	nfsd_file_put(src);
-	nfsd_file_put(dst);
-}
-
    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
    {
    	struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct nfsd4_copy *src, struct nfsd4_copy *dst)
    	dst->ss_nsui = src->ss_nsui;
    }
 
+static void release_copy_files(struct nfsd4_copy *copy)
+{
+	if (copy->nf_src)
+		nfsd_file_put(copy->nf_src);
+	if (copy->nf_dst)
+		nfsd_file_put(copy->nf_dst);
+}
+
    static void cleanup_async_copy(struct nfsd4_copy *copy)
    {
    	nfs4_free_copy_state(copy);
-	nfsd_file_put(copy->nf_dst);
-	if (!nfsd4_ssc_is_inter(copy))
-		nfsd_file_put(copy->nf_src);
+	release_copy_files(copy);
    	spin_lock(&copy->cp_clp->async_lock);
    	list_del(&copy->copies);
    	spin_unlock(&copy->cp_clp->async_lock);
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
    	} else {
    		nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
    				       copy->nf_dst->nf_file, false);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
    	}
 
    do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1815

Quoted 9 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
    	} else {
    		status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
    				       copy->nf_dst->nf_file, true);
-		nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
    	}
    out:
+	release_copy_files(copy);
    	return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 4 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp,
          struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
          long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
 
-       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
 
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1820

Quoted 10 lines
fs/nfsd/nfs4proc.c +2 -1
          spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
          release_copy_files(copy);
          return status;
   out_err:
-       if (async_copy)
+       if (async_copy) {
                  cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Quoted 3 lines

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Thanks,
-Dai

Quoted 1 line
#21 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 154 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 152 lines · Jeff Layton
Quoted 151 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 44 lines · Jeff Layton
Quoted 28 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 27 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any nfsd_files held on its behalf. For the async one, the cleanup is a bit more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the kthread_create call fails, then both the src and dst nfsd_files in the original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy case, we'll have up to four nfsd_file references (src and dst for both flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------

    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 15 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
             nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for the count added by dup_copy_fields.

Quoted 5 lines

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 86 lines
Quoted 59 lines
Quoted 53 lines
fs/nfsd/nfs4proc.c +0 -0
         fput(filp);
             spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                      &copy->nf_dst);
     }

    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)

fs/nfsd/nfs4proc.c +0 -5
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-

    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb) {
        struct nfsd4_cb_offload *cbo =

fs/nfsd/nfs4proc.c +8 -3
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct 
 nfsd4_copy *src, struct nfsd4_copy *dst)
         dst->ss_nsui = src->ss_nsui;
     }
     +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
     static void cleanup_async_copy(struct nfsd4_copy *copy)
     {
         nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
         spin_lock(&copy->cp_clp->async_lock);
         list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
         } else {
             nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }

        do_callback:

fs/nfsd/nfs4proc.c +0 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
         } else {
             status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
Quoted 4 lines

    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 4 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -1
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
           struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
           long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
    -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }

             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);

fs/nfsd/nfs4proc.c +0 -0
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
           release_copy_files(copy);
           return status;
Quoted 5 lines

   out_err:
-       if (async_copy)
+       if (async_copy) { cleanup_async_copy(async_copy); +               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Quoted 6 lines
Quoted 3 lines

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

-Dai

author maintainer
#22 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 166 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 165 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 163 lines · Jeff Layton
Quoted 162 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------

    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
             nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 87 lines
Quoted 59 lines
Quoted 53 lines
fs/nfsd/nfs4proc.c +0 -0
         fput(filp);
             spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                      &copy->nf_dst);
     }

    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file *dst)

fs/nfsd/nfs4proc.c +0 -5
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-

    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb) {
        struct nfsd4_cb_offload *cbo =

fs/nfsd/nfs4proc.c +8 -3
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct 
 nfsd4_copy *src, struct nfsd4_copy *dst)
         dst->ss_nsui = src->ss_nsui;
     }
     +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
     static void cleanup_async_copy(struct nfsd4_copy *copy)
     {
         nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
         spin_lock(&copy->cp_clp->async_lock);
         list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
         } else {
             nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }

        do_callback:

fs/nfsd/nfs4proc.c +0 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
         } else {
             status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
Quoted 4 lines

    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -1
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
           struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
           long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
    -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }

             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);

fs/nfsd/nfs4proc.c +0 -0
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
           release_copy_files(copy);
           return status;
Quoted 5 lines

   out_err:
-       if (async_copy)
+       if (async_copy) { cleanup_async_copy(async_copy); +               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway. Just make yourself the patch author and keep my S- o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Signature · 1 line
Jeff Layton <[email protected]>
#23 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 175 lines · Jeff Layton
Quoted 170 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 169 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 166 lines · Jeff Layton
Quoted 153 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
            nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 89 lines
Quoted 59 lines
Quoted 47 lines
fs/nfsd/nfs4proc.c +8 -9
        fput(filp);
            spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                     &copy->nf_dst);
    }
    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file 
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
    {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct 
 nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
    }
    +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
    static void cleanup_async_copy(struct nfsd4_copy *copy)
    {
        nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
            nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
        do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
        } else {
            status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct 
 nfsd4_ssc_umount_item *nsui, struct file *filp,
          struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
          long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
   -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct 
 nfsd4_compound_state *cstate,
          release_copy_files(copy);
          return status;
   out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, · line 1820

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                  cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Quoted 15 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Quoted 9 lines

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

#24 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
     fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
     1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
             nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
         fput(filp);
             spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                      &copy->nf_dst);
     }
     -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
     static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
     {
         struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
         dst->ss_nsui = src->ss_nsui;
     }
     +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
     static void cleanup_async_copy(struct nfsd4_copy *copy)
     {
         nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
         spin_lock(&copy->cp_clp->async_lock);
         list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
         } else {
             nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
         do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
         } else {
             status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
     out:
+    release_copy_files(copy);
         return status;
     out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
           struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
           long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
    -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
              spin_lock(&nn->nfsd_ssc_lo
           list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
           release_copy_files(copy);
           return status;
    out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                   cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 3 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

-Dai

Quoted 28 lines
Quoted 14 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Quoted 9 lines

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

#25 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 189 lines · Dai Ngo
Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
            nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
        fput(filp);
            spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                     &copy->nf_dst);
    }
    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
    {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
    }
    +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
    static void cleanup_async_copy(struct nfsd4_copy *copy)
    {
        nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
            nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
        do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
        } else {
            status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
          struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
          long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
   -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
          release_copy_files(copy);
          return status;
   out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                  cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Correct?

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Quoted 1 line

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

And this one hasn't been posted yet, right? Or did I miss it?

Quoted 2 lines

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

That should be fine.

Quoted 27 lines

-Dai

Quoted 25 lines
Quoted 14 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Quoted 9 lines

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

--
Chuck Lever

#26 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 203 lines · Chuck Lever III
Quoted 190 lines · Dai Ngo
Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
   fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
   1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
       long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
           nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
       fput(filp);
           spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                    &copy->nf_dst);
   }
   -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
   static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
   {
       struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
       dst->ss_nsui = src->ss_nsui;
   }
   +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
   static void cleanup_async_copy(struct nfsd4_copy *copy)
   {
       nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
       spin_lock(&copy->cp_clp->async_lock);
       list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
       } else {
           nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                          copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
       }
       do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
       } else {
           status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                          copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
       }
   out:
+    release_copy_files(copy);
       return status;
   out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
  -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
            spin_lock(&nn->nfsd_ssc_lo
         list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
         release_copy_files(copy);
         return status;
  out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                 cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

Quoted 8 lines

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Correct?

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Or not...

This one does not apply cleanly to nfsd-fixes, but does apply to nfsd-next. Also, the patch description says "clean up" and does not provide a Fixes:
tag. So, either:

- Jeff needs to test and redrive this patch against nfsd-fixes if we all agree that it fixes a real and urgent bug, not a potential one; or

- I will apply it as it stands to nfsd-next; or

- You were referring to something else in 1. above.

Let me know how you'd both like to proceed.

Quoted 34 lines
Quoted 1 line

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

And this one hasn't been posted yet, right? Or did I miss it?

Quoted 2 lines

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

That should be fine.

Quoted 23 lines

-Dai

Quoted 21 lines
Quoted 21 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

--
Chuck Lever

--
Chuck Lever

author maintainer
#27 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 217 lines · Chuck Lever III
Quoted 202 lines · Chuck Lever III
Quoted 190 lines · Dai Ngo
Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
   fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
   1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
       long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
           nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
       fput(filp);
           spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                    &copy->nf_dst);
   }
   -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
   static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
   {
       struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
       dst->ss_nsui = src->ss_nsui;
   }
   +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
   static void cleanup_async_copy(struct nfsd4_copy *copy)
   {
       nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
       spin_lock(&copy->cp_clp->async_lock);
       list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
       } else {
           nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                          copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
       }
       do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
       } else {
           status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                          copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
       }
   out:
+    release_copy_files(copy);
       return status;
   out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
  -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
            spin_lock(&nn->nfsd_ssc_lo
         list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
         release_copy_files(copy);
         return status;
  out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                 cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

Quoted 8 lines

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Correct?

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Or not...

This one does not apply cleanly to nfsd-fixes, but does apply to nfsd-next. Also, the patch description says "clean up" and does not provide a Fixes:
tag. So, either:

Quoted 9 lines

- Jeff needs to test and redrive this patch against nfsd-fixes if we all agree that it fixes a real and urgent bug, not a potential one; or

- I will apply it as it stands to nfsd-next; or

- You were referring to something else in 1. above.

Let me know how you'd both like to proceed.

I'm fine with nfsd-next here. These are not a bugs that people are going to hit under normal circumstances. It's something we need to fix, but it's not urgent.

Signature · 1 line
Jeff Layton <[email protected]>
#28 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

In reply to Chuck Lever III
Quoted 200 lines · Chuck Lever III
Quoted 190 lines · Dai Ngo
Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
            nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
        fput(filp);
            spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                     &copy->nf_dst);
    }
    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
    {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
    }
    +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
    static void cleanup_async_copy(struct nfsd4_copy *copy)
    {
        nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
            nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
        do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
        } else {
            status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
          struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
          long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
   -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
          release_copy_files(copy);
          return status;
   out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                  cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

Quoted 8 lines

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Correct?

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Sorry folks but I got a bit lost in the thread. I thought Dai pointed out that we can't remove the put from the nfsd4_cleanup_inter_ssc() because that's the put for the copied structure and not the original file references which is what Jeff's patch is trying to address.

Quoted 44 lines
Quoted 1 line

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

And this one hasn't been posted yet, right? Or did I miss it?

Quoted 2 lines

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

That should be fine.

Quoted 27 lines

-Dai

Quoted 25 lines
Quoted 21 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Quoted 2 lines

--
Jeff Layton <[email protected]>

--
Chuck Lever

--
Chuck Lever

author maintainer
#29 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

Quoted 207 lines · Olga Kornievskaia
Quoted 206 lines · Chuck Lever III
Quoted 190 lines · Dai Ngo
Quoted 171 lines · Chuck Lever III
Quoted 171 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
    fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
    1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
        long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
            nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
        fput(filp);
            spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                     &copy->nf_dst);
    }
    -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
    static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
    {
        struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
        dst->ss_nsui = src->ss_nsui;
    }
    +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
    static void cleanup_async_copy(struct nfsd4_copy *copy)
    {
        nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
        spin_lock(&copy->cp_clp->async_lock);
        list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
        } else {
            nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
        do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
        } else {
            status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                           copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
        }
    out:
+    release_copy_files(copy);
        return status;
    out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
          struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
          long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
   -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
             spin_lock(&nn->nfsd_ssc_lo
          list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
          release_copy_files(copy);
          return status;
   out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                  cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Yes.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Correct?

Quoted 8 lines
Quoted 2 lines

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Sorry folks but I got a bit lost in the thread. I thought Dai pointed out that we can't remove the put from the nfsd4_cleanup_inter_ssc() because that's the put for the copied structure and not the original file references which is what Jeff's patch is trying to address.

...and then I replied:

With this patch, the references held by the original copy structure are put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Quoted 45 lines
Quoted 45 lines
Quoted 2 lines

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

And this one hasn't been posted yet, right? Or did I miss it?

Quoted 2 lines

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

That should be fine.

Quoted 27 lines

-Dai

Quoted 25 lines
Quoted 21 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Quoted 2 lines

--
Jeff Layton <[email protected]>

--
Chuck Lever

--
Chuck Lever

Signature · 1 line
Jeff Layton <[email protected]>
#30 lore raw

Re: [PATCH 2/2] nfsd: clean up potential nfsd_file refcount leaks in COPY codepath

In reply to Chuck Lever III
Quoted 197 lines · Chuck Lever III
Quoted 183 lines · Dai Ngo
Quoted 170 lines · Chuck Lever III
Quoted 170 lines · Jeff Layton
Quoted 167 lines · On Sat, 2023-01-21 at 11:50 -0800, [email protected] wrote:
Quoted 166 lines · On 1/21/23 10:56 AM, [email protected] wrote:
Quoted 165 lines · Jeff Layton
Quoted 152 lines · On Thu, 2023-01-19 at 10:38 -0800, [email protected] wrote:
Quoted 53 lines · Jeff Layton
Quoted 36 lines · On Wed, 2023-01-18 at 21:05 -0800, [email protected] wrote:
Quoted 35 lines · Jeff Layton

There are two different flavors of the nfsd4_copy struct. One is embedded in the compound and is used directly in synchronous copies. The
other is dynamically allocated, refcounted and tracked in the client struture. For the embedded one, the cleanup just involves releasing any
nfsd_files held on its behalf. For the async one, the cleanup is a bit
more involved, and we need to dequeue it from lists, unhash it, etc.

There is at least one potential refcount leak in this code now. If the
kthread_create call fails, then both the src and dst nfsd_files in the
original nfsd4_copy object are leaked.

The cleanup in this codepath is also sort of weird. In the async copy
case, we'll have up to four nfsd_file references (src and dst for both
flavors of copy structure). They are both put at the end of nfsd4_do_async_copy, even though the ones held on behalf of the embedded
one outlive that structure.

Change it so that we always clean up the nfsd_file refs held by the embedded copy structure before nfsd4_copy returns. Rework cleanup_async_copy to handle both inter and intra copies. Eliminate nfsd4_cleanup_intra_ssc since it now becomes a no-op.

---
     fs/nfsd/nfs4proc.c | 23 ++++++++++-------------
     1 file changed, 10 insertions(+), 13 deletions(-)
Quoted 16 lines

fs/nfsd/nfs4proc.c nfsd4_cleanup_inter_ssc(struct · line 1515

Quoted 9 lines
fs/nfsd/nfs4proc.c +0 -1
index 37a9cc8ae7ae..62b9d6c1b18b 100644
@@ -1512,7 +1512,6 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
         long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
             nfs42_ssc_close(filp);
-    nfsd_file_put(dst);

I think we still need this, in addition to release_copy_files called from cleanup_async_copy. For async inter-copy, there are 2 reference count added to the destination file, one from nfsd4_setup_inter_ssc and the other one from dup_copy_fields. The above nfsd_file_put is for
the count added by dup_copy_fields.

Quoted 6 lines

With this patch, the references held by the original copy structure are
put by the call to release_copy_files at the end of nfsd4_copy. That means that the kthread task is only responsible for putting the references held by the (kmalloc'ed) async_copy structure. So, I think this gets the nfsd_file refcounting right.

Yes, I see. One refcount is decremented by release_copy_files at end of nfsd4_copy and another is decremented by release_copy_files in cleanup_async_copy.

Quoted 88 lines
Quoted 58 lines
Quoted 46 lines
fs/nfsd/nfs4proc.c +8 -9
         fput(filp);
             spin_lock(&nn->nfsd_ssc_lock);
@@ -1562,13 +1561,6 @@ nfsd4_setup_intra_ssc(struct svc_rqst *rqstp,
                      &copy->nf_dst);
     }
     -static void
-nfsd4_cleanup_intra_ssc(struct nfsd_file *src, struct nfsd_file
 *dst)
-{
-    nfsd_file_put(src);
-    nfsd_file_put(dst);
-}
-
     static void nfsd4_cb_offload_release(struct nfsd4_callback *cb)
     {
         struct nfsd4_cb_offload *cbo =
@@ -1683,12 +1675,18 @@ static void dup_copy_fields(struct
 nfsd4_copy *src, struct nfsd4_copy *dst)
         dst->ss_nsui = src->ss_nsui;
     }
     +static void release_copy_files(struct nfsd4_copy *copy)
+{
+    if (copy->nf_src)
+        nfsd_file_put(copy->nf_src);
+    if (copy->nf_dst)
+        nfsd_file_put(copy->nf_dst);
+}
+
     static void cleanup_async_copy(struct nfsd4_copy *copy)
     {
         nfs4_free_copy_state(copy);
-    nfsd_file_put(copy->nf_dst);
-    if (!nfsd4_ssc_is_inter(copy))
-        nfsd_file_put(copy->nf_src);
+    release_copy_files(copy);
         spin_lock(&copy->cp_clp->async_lock);
         list_del(&copy->copies);

spin_unlock(&copy->cp_clp->async_lock);

fs/nfsd/nfs4proc.c +0 -1
@@ -1748,7 +1746,6 @@ static int nfsd4_do_async_copy(void *data)
         } else {
             nfserr = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, false);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
         do_callback:

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1816

Quoted 10 lines
fs/nfsd/nfs4proc.c +1 -1
@@ -1811,9 +1808,9 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
         } else {
             status = nfsd4_do_copy(copy, copy->nf_src->nf_file,
                            copy->nf_dst->nf_file, true);
-        nfsd4_cleanup_intra_ssc(copy->nf_src, copy->nf_dst);
         }
     out:
+    release_copy_files(copy);
         return status;
     out_err:

This is unrelated to the reference count issue.

Quoted 5 lines

Here if this is an inter-copy then we need to decrement the reference count of the nfsd4_ssc_umount_item so that the vfsmount can be unmounted
later.

Oh, I think I see what you mean. Maybe something like the (untested) patch below on top of the original patch would fix that?

fs/nfsd/nfs4proc.c +4 -2
index c9057462b973..7475c593553c 100644
@@ -1511,8 +1511,10 @@ nfsd4_cleanup_inter_ssc(struct
 nfsd4_ssc_umount_item *nsui, struct file *filp,
           struct nfsd_net *nn = net_generic(dst->nf_net, nfsd_net_id);
           long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout);
    -       nfs42_ssc_close(filp);
-       fput(filp);
+       if (filp) {
+               nfs42_ssc_close(filp);
+               fput(filp);
+       }
              spin_lock(&nn->nfsd_ssc_lo
           list_del(&nsui->nsui_list);
@@ -1813,8 +1815,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct
 nfsd4_compound_state *cstate,
           release_copy_files(copy);
           return status;
    out_err:
-       if (async_copy)
Quoted 9 lines

fs/nfsd/nfs4proc.c nfsd4_copy(struct svc_rqst *rqstp, struct · line 1821

Quoted 3 lines
fs/nfsd/nfs4proc.c +2 -0
+       if (async_copy) {
                   cleanup_async_copy(async_copy);
+               if (nfsd4_ssc_is_inter(async_copy))

We don't need to call nfsd4_cleanup_inter_ssc since the thread nfsd4_do_async_copy has not started yet so the file is not opened. We just need to do refcount_dec(&copy->ss_nsui->nsui_refcnt), unless you want to change nfsd4_cleanup_inter_ssc to detect this error condition and only decrement the reference count.

Oh yeah, and this would break anyway since the nsui_list head is not being initialized. Dai, would you mind spinning up a patch for this since you're more familiar with the cleanup here?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Will do. My patch will only fix the unmount issue. Your patch does the clean up potential nfsd_file refcount leaks in COPY codepath.

Or do you want me to merge your patch and mine into one?

It probably is best to merge them, since backporters will probably want both patches anyway.

Unless these two changes are somehow interdependent, I'd like to keep them separate. They address two separate issues, yes?

Quoted 8 lines

Yes.

Quoted 2 lines

And -- narrow fixes need to go to nfsd-fixes, but clean-ups can wait for nfsd-next. I'd rather not mix the two types of change.

Ok. Can we do this:

1. Jeff's patch goes to nfsd-fixes since it has the fix for missing reference count.

To make sure I haven't lost track of anything:

The patch you refer to here is this one:

https://lore.kernel.org/linux-nfs/[email protected]/

Quoted 8 lines

Correct?

(I was waiting for Jeff and Olga to come to consensus, and I think they have, so I can apply it to nfsd-fixes now).

Quoted 1 line

2. My fix for the cleanup of allocated memory goes to nfsd-fixes.

And this one hasn't been posted yet, right? Or did I miss it?

I will post this patch soon.

Quoted 5 lines
Quoted 4 lines

3. I will do the optimization Jeff proposed about list_head and nfsd4_compound in a separate patch that goes into nfsd-next.

That should be fine.

Thanks,
-Dai

Quoted 34 lines
Quoted 29 lines

-Dai

Quoted 25 lines
Quoted 14 lines

Just make yourself the patch author and keep my S-o-b line.

Quoted 5 lines

I think we need a bit more cleanup in addition to your patch. When kmalloc(sizeof(*async_copy->cp_src), ..) or nfs4_init_copy_state fails, the async_copy is not initialized yet so calling cleanup_async_copy can be a problem.

Yeah.

It may even be best to ensure that the list_head and such are fully initialized for both allocated and embedded struct nfsd4_copy's. You might shave off a few cpu cycles by not doing that, but it makes things more fragile.

Quoted 9 lines

Even better, we really ought to split a lot of the fields in nfsd4_copy into a different structure (maybe nfsd4_async_copy). Trimming down struct nfsd4_copy would cut down the size of nfsd4_compound as well since it has a union that contains it. I was planning on doing that eventually, but if you want to take that on, then that would be fine too.

Signature · 1 line
Jeff Layton <[email protected]>

--
Chuck Lever

--
Chuck Lever