Skip to contentSkip to conversation

nfsd: fix handling of error from unshare_fs_struct()

PATCH Superseded NeilBrown 4 participants 10 messages Jul 29, 2024 – Jul 30, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

NeilBrown

author
#0 lore raw

cover nfsd: fix handling of error from unshare_fs_struct()

These two patches replace my previous patch:
[PATCH 07/14] Change unshare_fs_struct() to never fail.

I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.

Thanks,
NeilBrown

[PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly

author
#1 lore raw

PATCH 1/2 sunrpc: merge svc_rqst_alloc() into svc_prepare_thread()

From: NeilBrown <[email protected]>

The only caller of svc_rqst_alloc() is svc_prepare_thread(). So merge the one into the other and simplify.

---
 include/linux/sunrpc/svc.h |  2 --
 net/sunrpc/svc.c           | 28 ++++++++--------------------
 2 files changed, 8 insertions(+), 22 deletions(-)
include/linux/sunrpc/svc.h +0 -2
index 23617da0e565..01383ea077ad 100644
@@ -403,8 +403,6 @@ void svc_rpcb_cleanup(struct svc_serv *serv, struct net *net);
 int svc_bind(struct svc_serv *serv, struct net *net);
 struct svc_serv *svc_create(struct svc_program *, unsigned int,
 			    int (*threadfn)(void *data));
-struct svc_rqst *svc_rqst_alloc(struct svc_serv *serv,
-					struct svc_pool *pool, int node);
 bool		   svc_rqst_replace_page(struct svc_rqst *rqstp,
 					 struct page *page);
 void		   svc_rqst_release_pages(struct svc_rqst *rqstp);
net/sunrpc/svc.c +8 -20
index d9cda1e53a01..c0e91bea72e7 100644
@@ -633,8 +633,8 @@ svc_release_buffer(struct svc_rqst *rqstp)
 			put_page(rqstp->rq_pages[i]);
 }
 
-struct svc_rqst *
-svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node)
+static struct svc_rqst *
+svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)
 {
 	struct svc_rqst	*rqstp;
 
@@ -662,22 +662,6 @@ svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node)
 	if (!svc_init_buffer(rqstp, serv->sv_max_mesg, node))
 		goto out_enomem;
 
-	return rqstp;
-out_enomem:
-	svc_rqst_free(rqstp);
-	return NULL;
-}
-EXPORT_SYMBOL_GPL(svc_rqst_alloc);
-
-static struct svc_rqst *
-svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)
-{
-	struct svc_rqst	*rqstp;
-
-	rqstp = svc_rqst_alloc(serv, pool, node);
-	if (!rqstp)
-		return ERR_PTR(-ENOMEM);
-
 	spin_lock_bh(&serv->sv_lock);
 	serv->sv_nrthreads += 1;
 	spin_unlock_bh(&serv->sv_lock);
@@ -690,6 +674,10 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)
 	list_add_rcu(&rqstp->rq_all, &pool->sp_all_threads);
 
 	return rqstp;
+
+out_enomem:
+	svc_rqst_free(rqstp);
+	return NULL;
 }
 
 /**
@@ -779,8 +767,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs)
 		node = svc_pool_map_get_node(chosen_pool->sp_id);
 
 		rqstp = svc_prepare_thread(serv, chosen_pool, node);
-		if (IS_ERR(rqstp))
-			return PTR_ERR(rqstp);
+		if (!rqstp)
+			return -ENOMEM;
 		task = kthread_create_on_node(serv->sv_threadfn, rqstp,
 					      node, "%s", serv->sv_name);
 		if (IS_ERR(task)) {
Signature · 1 line
2.44.0
author
#2 lore raw

PATCH 2/2 sunrpc: allow svc threads to fail initialisation cleanly

In reply to NeilBrown

If an svc thread needs to perform some initialisation that might fail, it has no good way to handle the failure.

Before the thread can exit it must call svc_exit_thread(), but that requires the service mutex to be held. The thread cannot simply take the mutex as that could deadlock if there is a concurrent attempt to shut down all threads (which is unlikely, but not impossible).

nfsd currently call svc_exit_thread() unprotected in the unlikely event that unshare_fs_struct() fails.

We can clean this up by introducing svc_thread_init_status() by which an svc thread can report whether initialisation has succeeded. If it has, it continues normally into the action loop. If it has not, svc_thread_init_status() immediately aborts the thread. svc_start_kthread() waits for either of these to happen, and calls svc_exit_thread() (under the mutex) if the thread aborted.

---
 fs/lockd/svc.c             |  2 ++
 fs/nfs/callback.c          |  2 ++
 fs/nfsd/nfssvc.c           |  9 +++------
 include/linux/sunrpc/svc.h | 28 ++++++++++++++++++++++++++++
 net/sunrpc/svc.c           | 11 +++++++++++
 5 files changed, 46 insertions(+), 6 deletions(-)
fs/lockd/svc.c +2 -0
index ab8042a5b895..7d74858e1abe 100644
@@ -125,6 +125,8 @@ lockd(void *vrqstp)
 	struct net *net = &init_net;
 	struct lockd_net *ln = net_generic(net, lockd_net_id);
 
+	svc_thread_init_status(rqstp, 0);
+
 	/* try_to_freeze() is called from svc_recv() */
 	set_freezable();
 
fs/nfs/callback.c +2 -0
index 8adfcd4c8c1a..6cf92498a5ac 100644
@@ -76,6 +76,8 @@ nfs4_callback_svc(void *vrqstp)
 {
 	struct svc_rqst *rqstp = vrqstp;
 
+	svc_thread_init_status(rqstp, 0);
+
 	set_freezable();
 
 	while (!svc_thread_should_stop(rqstp))
fs/nfsd/nfssvc.c +3 -6
index 89d7918de7b1..2a99787d6a86 100644
@@ -915,11 +915,9 @@ nfsd(void *vrqstp)
 
 	/* At this point, the thread shares current->fs
 	 * with the init process. We need to create files with the
-	 * umask as defined by the client instead of init's umask. */
-	if (unshare_fs_struct() < 0) {
-		printk("Unable to start nfsd thread: out of memory\n");
-		goto out;
-	}
+	 * umask as defined by the client instead of init's umask.
+	 */
+	svc_thread_init_status(rqstp, unshare_fs_struct());
 
 	current->fs->umask = 0;
 
@@ -941,7 +939,6 @@ nfsd(void *vrqstp)
 
 	atomic_dec(&nfsd_th_cnt);
 
-out:
 	/* Release the thread */
 	svc_exit_thread(rqstp);
 	return 0;
include/linux/sunrpc/svc.h +28 -0
index 01383ea077ad..2c4366afc9bd 100644
@@ -21,6 +21,7 @@
 #include <linux/wait.h>
 #include <linux/mm.h>
 #include <linux/pagevec.h>
+#include <linux/kthread.h>
 
 /*
  *
@@ -231,6 +232,11 @@ struct svc_rqst {
 	struct net		*rq_bc_net;	/* pointer to backchannel's
 						 * net namespace
 						 */
+
+	int			rq_err;		/* Thread sets this to inidicate
+						 * initialisation success.
+						 */
+
 	unsigned long	bc_to_initval;
 	unsigned int	bc_to_retries;
 	void **			rq_lease_breaker; /* The v4 client breaking a lease */
@@ -304,6 +310,28 @@ static inline bool svc_thread_should_stop(struct svc_rqst *rqstp)
 	return test_bit(RQ_VICTIM, &rqstp->rq_flags);
 }
 
+/**
+ * svc_thread_init_status - report whether thread has initialised successfully
+ * @rqstp: the thread in question
+ * @err: errno code
+ *
+ * After performing any initialisation that could fail, and before starting
+ * normal work, each sunrpc svc_thread must call svc_thread_init_status()
+ * with an appropriate error, or zero.
+ *
+ * If zero is passed, the thread is ready and must continue until
+ * svc_thread_should_stop() returns true.  If a non-zero error is passed
+ * the call will not return - the thread will exit.
+ */
+static inline void svc_thread_init_status(struct svc_rqst *rqstp, int err)
+{
+	/* store_release ensures svc_start_kthreads() sees the error */
+	smp_store_release(&rqstp->rq_err, err);
+	wake_up_var(&rqstp->rq_err);
+	if (err)
+		kthread_exit(1);
+}
+
 struct svc_deferred_req {
 	u32			prot;	/* protocol (UDP or TCP) */
 	struct svc_xprt		*xprt;
net/sunrpc/svc.c +11 -0
index c0e91bea72e7..0fc9c97ae419 100644
@@ -662,6 +662,8 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)
 	if (!svc_init_buffer(rqstp, serv->sv_max_mesg, node))
 		goto out_enomem;
 
+	rqstp->rq_err = -EAGAIN; /* No error yet */
+
 	spin_lock_bh(&serv->sv_lock);
 	serv->sv_nrthreads += 1;
 	spin_unlock_bh(&serv->sv_lock);
@@ -760,6 +762,7 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs)
 	struct svc_pool *chosen_pool;
 	unsigned int state = serv->sv_nrthreads-1;
 	int node;
+	int err;
 
 	do {
 		nrservs--;
@@ -782,6 +785,14 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs)
 
 		svc_sock_update_bufs(serv);
 		wake_up_process(task);
+
+		/* load_acquire ensures we get value stored in svc_thread_init_status() */
+		wait_var_event(&rqstp->rq_err, smp_load_acquire(&rqstp->rq_err) != -EAGAIN);
+		err = rqstp->rq_err;
+		if (err) {
+			svc_exit_thread(rqstp);
+			return err;
+		}
 	} while (nrservs > 0);
 
 	return 0;
Signature · 1 line
2.44.0
maintainer Reviewed-by on the v1 series
#3 lore raw
In reply to NeilBrown
Quoted 12 lines · NeilBrown

These two patches replace my previous patch:
 [PATCH 07/14] Change unshare_fs_struct() to never fail.

I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc
layer.  So the first patch here is a simple cleanup, and the second adds
simple infrastructure for an svc thread to confirm that it has started
up and to report if it was successful in that.

Quoted 7 lines

Thanks,
NeilBrown

 [PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly

I like this solution better:

#4 lore raw
In reply to NeilBrown
Quoted 9 lines · NeilBrown

These two patches replace my previous patch:
[PATCH 07/14] Change unshare_fs_struct() to never fail.

I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.

Quoted 7 lines

Thanks,
NeilBrown

[PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly

This series does not apply to nfsd-next. It looks like 1/2 expects to see the EXPORT_SYMBOL after svc_rqst_alloc() that you already removed in "SUNRPC: make various functions static, or not exported."

Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).

Can you rebase and resend?

In 2/2, what is the reason to make svc_thread_init_status() a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?

Signature · 1 line
Chuck Lever
author
#5 lore raw
Quoted 18 lines · Chuck Lever
Quoted 9 lines · NeilBrown

These two patches replace my previous patch:
[PATCH 07/14] Change unshare_fs_struct() to never fail.

I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.

Quoted 7 lines

Thanks,
NeilBrown

[PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly

Quoted 6 lines

This series does not apply to nfsd-next. It looks like 1/2 expects to see the EXPORT_SYMBOL after svc_rqst_alloc() that you already removed in "SUNRPC: make various functions static, or not exported."

Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).

Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.

Quoted 2 lines

Can you rebase and resend?

I can't see "SUNRPC: make various functions static, or not exported." in
git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
Am I looking in the wrong place? If not, can you push out nfsd-next? (and if you could delete "for-next" from there, and possibly other old cruft, that might help too)

Thanks,
NeilBrown

Quoted 10 lines

In 2/2, what is the reason to make svc_thread_init_status() a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?

Signature · 1 line
Chuck Lever
#6 lore raw
Quoted 27 lines · NeilBrown
Quoted 22 lines · Chuck Lever
Quoted 9 lines · NeilBrown

These two patches replace my previous patch:
[PATCH 07/14] Change unshare_fs_struct() to never fail.

I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.

Quoted 7 lines

Thanks,
NeilBrown

[PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly

This series does not apply to nfsd-next. It looks like 1/2 expects to see the EXPORT_SYMBOL after svc_rqst_alloc() that you already removed in "SUNRPC: make various functions static, or not exported."

Quoted 6 lines
Quoted 2 lines

Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).

Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.

Some patch linting tools like the From: to match the SoB tag. But again, I'm not sure that matters except to the legally pedantic.

Quoted 5 lines
Quoted 1 line

Can you rebase and resend?

I can't see "SUNRPC: make various functions static, or not exported." in
git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
Am I looking in the wrong place? If not, can you push out nfsd-next?

For some reason, git keeps pushing my local nfsd-next branch to cel's "master" branch. I've pushed again to cel's nfsd-next, and it seems to be showing up now on the web UI as I intended.

Quoted 2 lines

(and if you could delete "for-next" from there, and possibly other old cruft, that might help too)

I'm looking at the list of branches, and I don't see a for-next in the cel git repo. The only branches I can see are:

nfsd-next
nfsd-6.1.y
nfsd-testing
master
nfsd-5.10.y
nfsd-5.15.y
nfsd-fixes
exportfs-next
topic-xdr-tracepoints
topic-rdma-fault-injection

I've been deleting old tags as I notice them, but "git push :branch" doesn't always seem to work against git.kernel.org.

Quoted 13 lines

Thanks,
NeilBrown

Quoted 9 lines

In 2/2, what is the reason to make svc_thread_init_status() a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?

Signature · 1 line
Chuck Lever

--
Chuck Lever

author
#7 lore raw
Quoted 23 lines · Chuck Lever III
Quoted 7 lines · NeilBrown

(and if you could delete "for-next" from there, and possibly other old cruft, that might help too)

I'm looking at the list of branches, and I don't see a for-next in the cel git repo. The only branches I can see are:

nfsd-next
nfsd-6.1.y
nfsd-testing
master
nfsd-5.10.y
nfsd-5.15.y
nfsd-fixes
exportfs-next
topic-xdr-tracepoints
topic-rdma-fault-injection

Quoted 2 lines

I've been deleting old tags as I notice them, but "git push :branch" doesn't always seem to work against git.kernel.org.

Ahh... "git fetch" does discard branched that have been deleted remotely.
I need
git remote prune $remote
or to set remote.$remote.prune to true

Thanks,
NeilBrown

#8 lore raw
In reply to NeilBrown
Quoted 6 lines · NeilBrown
Quoted 2 lines

Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).

Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.

The signoff is supposed to match the author address. If you send it out using a different email account you need to add a separate From:
line to the mail body. git-send-email will do that automatically for you.

author
#9 lore raw
In reply to Chuck Lever
Quoted 6 lines · Chuck Lever

In 2/2, what is the reason to make svc_thread_init_status() a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?

Sorry - I missed this question the first time.

My reasoning was that it was a tiny function which would be optimised even smaller in two out of three call sites where the err number is a constant zero. Also my original draft didn't even have this as a function but was open coded.

I don't think it matters much either way.

Thanks,
NeilBrown