nfsd: fix handling of error from unshare_fs_struct()
Subsystems (3)
cover nfsd: fix handling of error from unshare_fs_struct()
These two patches replace my previous patch:
[PATCH 07/14] Change unshare_fs_struct() to never fail.
I had explored ways to change kthread_create() to avoid the need for GFP_NOFAIL and concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.
Thanks,
NeilBrown
[PATCH 1/2] sunrpc: merge svc_rqst_alloc() into svc_prepare_thread() [PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly
PATCH 1/2 sunrpc: merge svc_rqst_alloc() into svc_prepare_thread()
From: NeilBrown <[email protected]>
The only caller of svc_rqst_alloc() is svc_prepare_thread(). So merge the one into the other and simplify.
--- include/linux/sunrpc/svc.h | 2 -- net/sunrpc/svc.c | 28 ++++++++-------------------- 2 files changed, 8 insertions(+), 22 deletions(-)
include/linux/sunrpc/svc.h +0 -2
@@ -403,8 +403,6 @@ void svc_rpcb_cleanup(struct svc_serv *serv, struct net *net);int svc_bind (struct svc_serv *serv,struct net *net);struct svc_serv *svc_create (struct svc_program *,unsigned int ,int (*threadfn)(void *data)); -struct svc_rqst *svc_rqst_alloc (struct svc_serv *serv, -struct svc_pool *pool,int node);bool svc_rqst_replace_page (struct svc_rqst *rqstp,struct page *page);void svc_rqst_release_pages (struct svc_rqst *rqstp);
net/sunrpc/svc.c +8 -20
@@ -633,8 +633,8 @@ svc_release_buffer(struct svc_rqst *rqstp)put_page (rqstp->rq_pages[i]); } -struct svc_rqst * -svc_rqst_alloc (struct svc_serv *serv,struct svc_pool *pool,int node) +static struct svc_rqst * +svc_prepare_thread (struct svc_serv *serv,struct svc_pool *pool,int node) {struct svc_rqst *rqstp;
@@ -662,22 +662,6 @@ svc_rqst_alloc(struct svc_serv *serv, struct svc_pool *pool, int node)if (!svc_init_buffer (rqstp, serv->sv_max_mesg, node)) goto out_enomem; -return rqstp; -out_enomem: -svc_rqst_free (rqstp); -return NULL ; -} -EXPORT_SYMBOL_GPL (svc_rqst_alloc); - -static struct svc_rqst * -svc_prepare_thread (struct svc_serv *serv,struct svc_pool *pool,int node) -{ -struct svc_rqst *rqstp; - - rqstp =svc_rqst_alloc (serv, pool, node); -if (!rqstp) -return ERR_PTR (-ENOMEM ); -spin_lock_bh (&serv->sv_lock); serv->sv_nrthreads +=1 ;spin_unlock_bh (&serv->sv_lock);
@@ -690,6 +674,10 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)list_add_rcu (&rqstp->rq_all, &pool->sp_all_threads);return rqstp; + +out_enomem: +svc_rqst_free (rqstp); +return NULL ; }/**
@@ -779,8 +767,8 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs) node =svc_pool_map_get_node (chosen_pool->sp_id); rqstp =svc_prepare_thread (serv, chosen_pool, node); -if (IS_ERR (rqstp)) -return PTR_ERR (rqstp); +if (!rqstp) +return -ENOMEM ; task =kthread_create_on_node (serv->sv_threadfn, rqstp, node,"%s" , serv->sv_name);if (IS_ERR (task)) {
Signature · 1 line
2.44.0
PATCH 2/2 sunrpc: allow svc threads to fail initialisation cleanly
In reply to NeilBrownIf an svc thread needs to perform some initialisation that might fail, it has no good way to handle the failure.
Before the thread can exit it must call svc_exit_thread(), but that requires the service mutex to be held. The thread cannot simply take the mutex as that could deadlock if there is a concurrent attempt to shut down all threads (which is unlikely, but not impossible).
nfsd currently call svc_exit_thread() unprotected in the unlikely event that unshare_fs_struct() fails.
We can clean this up by introducing svc_thread_init_status() by which an svc thread can report whether initialisation has succeeded. If it has, it continues normally into the action loop. If it has not, svc_thread_init_status() immediately aborts the thread. svc_start_kthread() waits for either of these to happen, and calls svc_exit_thread() (under the mutex) if the thread aborted.
--- fs/lockd/svc.c | 2 ++ fs/nfs/callback.c | 2 ++ fs/nfsd/nfssvc.c | 9 +++------ include/linux/sunrpc/svc.h | 28 ++++++++++++++++++++++++++++ net/sunrpc/svc.c | 11 +++++++++++ 5 files changed, 46 insertions(+), 6 deletions(-)
fs/lockd/svc.c +2 -0
@@ -125,6 +125,8 @@ lockd(void *vrqstp)struct net *net = &init_net;struct lockd_net *ln =net_generic (net, lockd_net_id); +svc_thread_init_status (rqstp,0 ); +/* try_to_freeze() is called from svc_recv() */ set_freezable ();
fs/nfs/callback.c +2 -0
@@ -76,6 +76,8 @@ nfs4_callback_svc(void *vrqstp) {struct svc_rqst *rqstp = vrqstp; +svc_thread_init_status (rqstp,0 ); +set_freezable ();while (!svc_thread_should_stop (rqstp))
fs/nfsd/nfssvc.c +3 -6
@@ -915,11 +915,9 @@ nfsd(void *vrqstp)/* At this point, the thread shares current->fs * with the init process. We need to create files with the -* umask as defined by the client instead of init's umask. */ -if (unshare_fs_struct () <0 ) { -printk ("Unable to start nfsd thread: out of memory\n" ); - goto out; - } +* umask as defined by the client instead of init's umask. +*/ +svc_thread_init_status (rqstp,unshare_fs_struct ()); current->fs->umask =0 ;
@@ -941,7 +939,6 @@ nfsd(void *vrqstp)atomic_dec (&nfsd_th_cnt); -out:/* Release the thread */ svc_exit_thread (rqstp);return 0 ;
include/linux/sunrpc/svc.h +28 -0
@@ -21,6 +21,7 @@#include <linux/wait.h> #include <linux/mm.h> #include <linux/pagevec.h> +#include <linux/kthread.h> /* *
@@ -231,6 +232,11 @@ struct svc_rqst {struct net *rq_bc_net;/* pointer to backchannel's * net namespace */ + +int rq_err;/* Thread sets this to inidicate +* initialisation success. +*/ +unsigned long bc_to_initval;unsigned int bc_to_retries;void ** rq_lease_breaker;/* The v4 client breaking a lease */
@@ -304,6 +310,28 @@ static inline bool svc_thread_should_stop(struct svc_rqst *rqstp)return test_bit (RQ_VICTIM , &rqstp->rq_flags); } +/** +* svc_thread_init_status - report whether thread has initialised successfully +* @rqstp: the thread in question +* @err: errno code +* +* After performing any initialisation that could fail, and before starting +* normal work, each sunrpc svc_thread must call svc_thread_init_status() +* with an appropriate error, or zero. +* +* If zero is passed, the thread is ready and must continue until +* svc_thread_should_stop() returns true. If a non-zero error is passed +* the call will not return - the thread will exit. +*/ +static inline void svc_thread_init_status (struct svc_rqst *rqstp,int err) +{ +/* store_release ensures svc_start_kthreads() sees the error */ +smp_store_release (&rqstp->rq_err, err); +wake_up_var (&rqstp->rq_err); +if (err) +kthread_exit (1 ); +} +struct svc_deferred_req {u32 prot;/* protocol (UDP or TCP) */ struct svc_xprt *xprt;
net/sunrpc/svc.c +11 -0
@@ -662,6 +662,8 @@ svc_prepare_thread(struct svc_serv *serv, struct svc_pool *pool, int node)if (!svc_init_buffer (rqstp, serv->sv_max_mesg, node)) goto out_enomem; + rqstp->rq_err = -EAGAIN ;/* No error yet */ +spin_lock_bh (&serv->sv_lock); serv->sv_nrthreads +=1 ;spin_unlock_bh (&serv->sv_lock);
@@ -760,6 +762,7 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs)struct svc_pool *chosen_pool;unsigned int state = serv->sv_nrthreads-1 ;int node; +int err; do { nrservs--;
@@ -782,6 +785,14 @@ svc_start_kthreads(struct svc_serv *serv, struct svc_pool *pool, int nrservs)svc_sock_update_bufs (serv);wake_up_process (task); + +/* load_acquire ensures we get value stored in svc_thread_init_status() */ +wait_var_event (&rqstp->rq_err,smp_load_acquire (&rqstp->rq_err) != -EAGAIN ); + err = rqstp->rq_err; +if (err) { +svc_exit_thread (rqstp); +return err; + } }while (nrservs >0 );return 0 ;
Signature · 1 line
2.44.0
Quoted 12 lines · NeilBrown
These two patches replace my previous patch:
[PATCH 07/14] Changeunshare_fs_struct()to never fail.I had explored ways to change
kthread_create()to avoid the need forGFP_NOFAILand concluded that we can do everything we need in the sunrpc
layer. So the first patch here is a simple cleanup, and the second adds
simple infrastructure for an svc thread to confirm that it has started
up and to report if it was successful in that.
Quoted 7 lines
Thanks,
NeilBrown[PATCH 1/2] sunrpc: merge
svc_rqst_alloc()intosvc_prepare_thread()[PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly
I like this solution better:
Quoted 9 lines · NeilBrown
These two patches replace my previous patch:
[PATCH 07/14] Changeunshare_fs_struct()to never fail.I had explored ways to change
kthread_create()to avoid the need forGFP_NOFAILand concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.
Quoted 7 lines
Thanks,
NeilBrown[PATCH 1/2] sunrpc: merge
svc_rqst_alloc()intosvc_prepare_thread()[PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly
This series does not apply to nfsd-next. It looks like 1/2 expects to see the EXPORT_SYMBOL after svc_rqst_alloc() that you already removed in "SUNRPC: make various functions static, or not exported."
Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).
Can you rebase and resend?
In 2/2, what is the reason to make svc_thread_init_status() a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?
Signature · 1 line
Chuck Lever
Quoted 18 lines · Chuck Lever
Quoted 9 lines · NeilBrown
These two patches replace my previous patch:
[PATCH 07/14] Changeunshare_fs_struct()to never fail.I had explored ways to change
kthread_create()to avoid the need forGFP_NOFAILand concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.Quoted 7 lines
Thanks,
NeilBrown[PATCH 1/2] sunrpc: merge
svc_rqst_alloc()intosvc_prepare_thread()[PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanly
Quoted 6 lines
This series does not apply to nfsd-next. It looks like 1/2 expects to see the
EXPORT_SYMBOLaftersvc_rqst_alloc()that you already removed in "SUNRPC: make various functions static, or not exported."Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).
Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.
Quoted 2 lines
Can you rebase and resend?
I can't see "SUNRPC: make various functions static, or not exported." in
git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
Am I looking in the wrong place? If not, can you push out nfsd-next? (and if you could delete "for-next" from there, and possibly other old cruft, that might help too)
Thanks,
NeilBrown
Quoted 10 lines
In 2/2, what is the reason to make
svc_thread_init_status()a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?Signature · 1 line
Chuck Lever
Quoted 27 lines · NeilBrown
Quoted 22 lines · Chuck Lever
Quoted 9 lines · NeilBrown
These two patches replace my previous patch:
[PATCH 07/14] Changeunshare_fs_struct()to never fail.I had explored ways to change
kthread_create()to avoid the need forGFP_NOFAILand concluded that we can do everything we need in the sunrpc layer. So the first patch here is a simple cleanup, and the second adds simple infrastructure for an svc thread to confirm that it has started up and to report if it was successful in that.Quoted 7 lines
Thanks,
NeilBrown[PATCH 1/2] sunrpc: merge
svc_rqst_alloc()intosvc_prepare_thread()[PATCH 2/2] sunrpc: allow svc threads to fail initialisation cleanlyThis series does not apply to nfsd-next. It looks like 1/2 expects to see the
EXPORT_SYMBOLaftersvc_rqst_alloc()that you already removed in "SUNRPC: make various functions static, or not exported."
Quoted 6 lines
Quoted 2 lines
Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).
Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.
Some patch linting tools like the From: to match the SoB tag. But again, I'm not sure that matters except to the legally pedantic.
Quoted 5 lines
Quoted 1 line
Can you rebase and resend?
I can't see "SUNRPC: make various functions static, or not exported." in
git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
Am I looking in the wrong place? If not, can you push out nfsd-next?
For some reason, git keeps pushing my local nfsd-next branch to cel's "master" branch. I've pushed again to cel's nfsd-next, and it seems to be showing up now on the web UI as I intended.
Quoted 2 lines
(and if you could delete "for-next" from there, and possibly other old cruft, that might help too)
I'm looking at the list of branches, and I don't see a for-next in the cel git repo. The only branches I can see are:
nfsd-next
nfsd-6.1.y
nfsd-testing
master
nfsd-5.10.y
nfsd-5.15.y
nfsd-fixes
exportfs-next
topic-xdr-tracepoints
topic-rdma-fault-injection
I've been deleting old tags as I notice them, but "git push :branch" doesn't always seem to work against git.kernel.org.
Quoted 13 lines
Thanks,
NeilBrownQuoted 9 lines
In 2/2, what is the reason to make
svc_thread_init_status()a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?Signature · 1 line
Chuck Lever
--
Chuck Lever
Quoted 23 lines · Chuck Lever III
Quoted 7 lines · NeilBrown
(and if you could delete "for-next" from there, and possibly other old cruft, that might help too)
I'm looking at the list of branches, and I don't see a for-next in the cel git repo. The only branches I can see are:
nfsd-next
nfsd-6.1.y
nfsd-testing
master
nfsd-5.10.y
nfsd-5.15.y
nfsd-fixes
exportfs-next
topic-xdr-tracepoints
topic-rdma-fault-injection
Quoted 2 lines
I've been deleting old tags as I notice them, but "git push :branch" doesn't always seem to work against git.kernel.org.
Ahh... "git fetch" does discard branched that have been deleted remotely.
I need
git remote prune $remote
or to set remote.$remote.prune to true
Thanks,
NeilBrown
Quoted 6 lines · NeilBrown
Quoted 2 lines
Also, 1/2 is From: your brown.name account, but the SoB is your SuSE email. (Maybe that doesn't matter).
Probably don't matter. That happened because I wrote that patch on my notebook instead of my desktop and they have different defaults. I'll try to remember that for next time.
The signoff is supposed to match the author address. If you send it out using a different email account you need to add a separate From:
line to the mail body. git-send-email will do that automatically for you.
Quoted 6 lines · Chuck Lever
In 2/2, what is the reason to make
svc_thread_init_status()a static inline rather than an exported function? I don't think this is going to be a performance hot path, but maybe it becomes one in a future patch?
Sorry - I missed this question the first time.
My reasoning was that it was a tiny function which would be optimised even smaller in two out of three call sites where the err number is a constant zero. Also my original draft didn't even have this as a function but was open coded.
I don't think it matters much either way.
Thanks,
NeilBrown