Skip to contentSkip to conversation

NFSD: Don't start nfsd threads from a pool_threads write

PATCH In review Chuck Lever 3 participants 5 messages Oct 9 – Oct 10

Subsystems (2)
lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

PATCH 1/1 NFSD: Don't start nfsd threads from a pool_threads write

Writing to pool_threads calls nfsd_set_nrthreads() directly, which starts nfsd threads as soon as a svc_serv exists. A portlist write creates the svc_serv without running nfsd_startup_net(), so a pool_threads write that follows it starts threads in a namespace whose NFSv4 state has not been set up. If the server in that namespace has been started and shut down before, the hash table pointers in struct nfsd_net still point at memory that nfs4_state_destroy_net() freed, and the first EXCHANGE_ID those threads handle reads it:

  BUG: KASAN: slab-use-after-free in add_to_unconfirmed
  nfsd4_exchange_id
  nfsd4_proc_compound
  nfsd_dispatch

A namespace that has never been started dereferences NULL instead.

write_pool_threads() already starts nothing when no svc_serv exists, and its comment documents that only the threads file starts NFS. Ignore the thread counts in a pool_threads write while NFSD_NET_UP is clear as well, so a pool_threads write never starts threads before nfsd_startup_net() has run. The reply still reports the current thread count for each pool.

Fixes 0a7ec37727dc ("nfsd: make unconf_id_hashtbl allocated per net")
Reported-by [email protected]
---
 fs/nfsd/nfsctl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/nfsd/nfsctl.c +2 -1
index 2379b838c028..5a606086b811 100644
@@ -497,7 +497,8 @@ static ssize_t write_pool_threads(struct file *file, char *buf, size_t size)
 	if (nthreads == NULL)
 		goto out_free;
 
-	if (size > 0) {
+	/* Threads started now would run without NFSv4 state. */
+	if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
 		for (i = 0; i < npools; i++) {
 			rv = get_int(&mesg, &nthreads[i]);
 			if (rv == -ENOENT)
Signature · 1 line
2.54.0
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 37 lines · Chuck Lever

Writing to pool_threads calls nfsd_set_nrthreads() directly, which starts nfsd threads as soon as a svc_serv exists. A portlist write creates the svc_serv without running nfsd_startup_net(), so a pool_threads write that follows it starts threads in a namespace whose NFSv4 state has not been set up. If the server in that namespace has been started and shut down before, the hash table pointers in struct nfsd_net still point at memory that nfs4_state_destroy_net() freed, and the first EXCHANGE_ID those threads handle reads it:

  BUG: KASAN: slab-use-after-free in add_to_unconfirmed
  nfsd4_exchange_id
  nfsd4_proc_compound
  nfsd_dispatch

A namespace that has never been started dereferences NULL instead.

write_pool_threads() already starts nothing when no svc_serv exists, and its comment documents that only the threads file starts NFS. Ignore the thread counts in a pool_threads write while NFSD_NET_UP is clear as well, so a pool_threads write never starts threads before nfsd_startup_net() has run. The reply still reports the current thread count for each pool.

Fixes 0a7ec37727dc ("nfsd: make unconf_id_hashtbl allocated per net")
Reported-by [email protected]
---
 fs/nfsd/nfsctl.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/nfsd/nfsctl.c +0 -0
index 2379b838c028..5a606086b811 100644

fs/nfsd/nfsctl.c static ssize_t write_pool_threads(struct file *file, char *buf, size_t size) · line 504

Quoted 9 lines
fs/nfsd/nfsctl.c +2 -1
@@ -497,7 +497,8 @@ static ssize_t write_pool_threads(struct file *file, char *buf, size_t size)
 	if (nthreads == NULL)
 		goto out_free;
 
-	if (size > 0) {
+	/* Threads started now would run without NFSv4 state. */
+	if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
 		for (i = 0; i < npools; i++) {
 			rv = get_int(&mesg, &nthreads[i]);
 			if (rv == -ENOENT)
#2 lore raw
In reply to Chuck Lever
kernel test robot noticed the following build errors:

Hi Chuck,

kernel test robot noticed the following build errors:

[auto build test ERROR on brauner-vfs/vfs.all]
[also build test ERROR on linus/master v7.3-rc6 next-20261008] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Chuck-Lever/NFSD-Don-t-start-nfsd-threads-from-a-pool_threads-write/20261009-104017
base:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.all
patch link:    https://lore.kernel.org/r/20261009144017.1284586-1-cel%40kernel.org
patch subject: [PATCH v1] NFSD: Don't start nfsd threads from a pool_threads write
config: parisc64-defconfig (https://download.01.org/0day-ci/archive/20261010/[email protected]/config)
compiler: hppa64-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261010/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from include/linux/log2.h:12,

                    from include/asm-generic/div64.h:55,
                    from ./arch/parisc/include/generated/asm/div64.h:1,
                    from include/linux/math.h:6,
                    from include/linux/math64.h:6,
                    from include/linux/time64.h:5,
                    from include/linux/restart_block.h:9,
                    from include/linux/thread_info.h:14,
                    from include/asm-generic/preempt.h:5,
                    from ./arch/parisc/include/generated/asm/preempt.h:1,
                    from include/linux/preempt.h:107,
                    from include/linux/spinlock.h:56,
                    from include/linux/mmzone.h:8,
                    from include/linux/gfp.h:7,
                    from include/linux/slab.h:17,
                    from fs/nfsd/nfsctl.c:8:
   fs/nfsd/nfsctl.c: In function 'write_pool_threads':
>> fs/nfsd/nfsctl.c:504:48: error: 'nn' undeclared (first use in this function)
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
   include/linux/bitops.h:44:44: note: in definition of macro 'bitop'
      44 |           __builtin_constant_p((uintptr_t)(addr) != (uintptr_t)NULL) && \
         |                                            ^~~~
   fs/nfsd/nfsctl.c:504:25: note: in expansion of macro 'test_bit'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                         ^~~~~~~~
   fs/nfsd/nfsctl.c:504:48: note: each undeclared identifier is reported only once for each function it appears in
   include/linux/bitops.h:44:44: note: in definition of macro 'bitop'
      44 |           __builtin_constant_p((uintptr_t)(addr) != (uintptr_t)NULL) && \
         |                                            ^~~~
   fs/nfsd/nfsctl.c:504:25: note: in expansion of macro 'test_bit'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                         ^~~~~~~~

vim +/nn +504 fs/nfsd/nfsctl.c

449
450 /*
451 * write_pool_threads - Set or report the configured number of threads per pool
452 *
453 * Input:
454 * buf: ignored
455 * size: zero
456 *
457 * OR
458 *
459 * Input:
460 * buf: C string containing whitespace-
461 * separated unsigned integer values
462 * representing the number of NFSD
463 * threads to start in each pool
464 * size: non-zero length of C string in @buf
465 * Output:
466 * On success: passed-in buffer filled with '\n'-terminated C
467 * string containing integer values representing the
468 * configured number of NFSD threads in each pool;
469 * return code is the size in bytes of the string
470 * On error: return code is zero or a negative errno value
471 */
472 static ssize_t write_pool_threads(struct file *file, char *buf, size_t size)
473 {
474 /* if size > 0, look for an array of number of threads per node
475 * and apply them then write out number of threads per node as reply
476 */
477 char *mesg = buf;
478 int i;
479 int rv;
480 int len;
481 int npools;
482 int *nthreads;
483 struct net *net = netns(file);
484
485 mutex_lock(&nfsd_mutex);
486 npools = nfsd_nrpools(net);
487 if (npools == 0) {
488 /*
489 * NFS is shut down. The admin can start it by
490 * writing to the threads file but NOT the pool_threads
491 * file, sorry. Report zero threads.
492 */
493 mutex_unlock(&nfsd_mutex);
494 strcpy(buf, "0\n");
495 return strlen(buf);
496 }
497
498 nthreads = kzalloc_objs(int, npools);
499 rv = -ENOMEM;
500 if (nthreads == NULL)
501 goto out_free;
502
503 /* Threads started now would run without NFSv4 state. */

 > 504		if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {

505 for (i = 0; i < npools; i++) {
506 rv = get_int(&mesg, &nthreads[i]);
507 if (rv == -ENOENT)
508 break; /* fewer numbers than pools */
509 if (rv)
510 goto out_free; /* syntax error */
511 rv = -EINVAL;
512 if (nthreads[i] < 0)
513 goto out_free;
514 trace_nfsd_ctl_pool_threads(net, i, nthreads[i]);
515 }
516
517 /*
518 * There must always be a thread in pool 0; the admin
519 * can't shut down NFS completely using pool_threads.
520 */
521 if (nthreads[0] == 0)
522 nthreads[0] = 1;
523
524 rv = nfsd_set_nrthreads(i, nthreads, net);
525 if (rv)
526 goto out_free;
527 }
528
529 rv = nfsd_get_nrthreads(npools, nthreads, net);
530 if (rv)
531 goto out_free;
532
533 mesg = buf;
534 size = SIMPLE_TRANSACTION_LIMIT;
535 for (i = 0; i < npools && size > 0; i++) {
536 snprintf(mesg, size, "%d%c", nthreads[i], (i == npools-1 ? '\n' : ' '));
537 len = strlen(mesg);
538 size -= len;
539 mesg += len;
540 }
541 rv = mesg - buf;
542 out_free:
543 kfree(nthreads);
544 mutex_unlock(&nfsd_mutex);
545 return rv;
546 }
547

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

author maintainer
#3 lore raw
Quoted 177 lines · kernel test robot

Hi Chuck,

kernel test robot noticed the following build errors:

[auto build test ERROR on brauner-vfs/vfs.all]
[also build test ERROR on linus/master v7.3-rc6 next-20261008] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:
https://github.com/intel-lab-lkp/linux/commits/Chuck-Lever/NFSD-Don-t-start-nfsd-threads-from-a-pool_threads-write/20261009-104017
base: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
vfs.all
patch link:
https://lore.kernel.org/r/20261009144017.1284586-1-cel%40kernel.org
patch subject: [PATCH v1] NFSD: Don't start nfsd threads from a pool_threads write
config: parisc64-defconfig
(https://download.01.org/0day-ci/archive/20261010/[email protected]/config)
compiler: hppa64-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build):
(https://download.01.org/0day-ci/archive/20261010/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes:
https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

   In file included from include/linux/log2.h:12,
                    from include/asm-generic/div64.h:55,
                    from ./arch/parisc/include/generated/asm/div64.h:1,
                    from include/linux/math.h:6,
                    from include/linux/math64.h:6,
                    from include/linux/time64.h:5,
                    from include/linux/restart_block.h:9,
                    from include/linux/thread_info.h:14,
                    from include/asm-generic/preempt.h:5,
                    from ./arch/parisc/include/generated/asm/preempt.h:1,
                    from include/linux/preempt.h:107,
                    from include/linux/spinlock.h:56,
                    from include/linux/mmzone.h:8,
                    from include/linux/gfp.h:7,
                    from include/linux/slab.h:17,
                    from fs/nfsd/nfsctl.c:8:
   fs/nfsd/nfsctl.c: In function 'write_pool_threads':
Quoted 1 line
Quoted 1 line

fs/nfsd/nfsctl.c:504:48: error: 'nn' undeclared (first use in this function)

     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
   include/linux/bitops.h:44:44: note: in definition of macro 'bitop'
      44 |           __builtin_constant_p((uintptr_t)(addr) != 
(uintptr_t)NULL) && \
         |                                            ^~~~
   fs/nfsd/nfsctl.c:504:25: note: in expansion of macro 'test_bit'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                         ^~~~~~~~
   fs/nfsd/nfsctl.c:504:48: note: each undeclared identifier is 
reported only once for each function it appears in
   include/linux/bitops.h:44:44: note: in definition of macro 'bitop'
      44 |           __builtin_constant_p((uintptr_t)(addr) != 
(uintptr_t)NULL) && \
         |                                            ^~~~
   fs/nfsd/nfsctl.c:504:25: note: in expansion of macro 'test_bit'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                         ^~~~~~~~

vim +/nn +504 fs/nfsd/nfsctl.c

   449	
   450	/*
   451	 * write_pool_threads - Set or report the configured number of 
threads per pool
   452	 *
   453	 * Input:
   454	 *			buf:		ignored
   455	 *			size:		zero
   456	 *
   457	 * OR
   458	 *
   459	 * Input:
   460	 *			buf:		C string containing whitespace-
   461	 *					separated unsigned integer values
   462	 *					representing the number of NFSD
   463	 *					threads to start in each pool
   464	 *			size:		non-zero length of C string in @buf
   465	 * Output:
   466	 *	On success:	passed-in buffer filled with '\n'-terminated C
   467	 *			string containing integer values representing the
   468	 *			configured number of NFSD threads in each pool;
   469	 *			return code is the size in bytes of the string
   470	 *	On error:	return code is zero or a negative errno value
   471	 */
   472	static ssize_t write_pool_threads(struct file *file, char *buf, 

size_t size)
473 {
474 /* if size > 0, look for an array of number of threads per node
475 * and apply them then write out number of threads per node as
reply
476 */

   477		char *mesg = buf;
   478		int i;
   479		int rv;
   480		int len;
   481		int npools;
   482		int *nthreads;
   483		struct net *net = netns(file);
   484	
   485		mutex_lock(&nfsd_mutex);
   486		npools = nfsd_nrpools(net);
   487		if (npools == 0) {
   488			/*
   489			 * NFS is shut down.  The admin can start it by
   490			 * writing to the threads file but NOT the pool_threads
   491			 * file, sorry.  Report zero threads.
   492			 */
   493			mutex_unlock(&nfsd_mutex);
   494			strcpy(buf, "0\n");
   495			return strlen(buf);
   496		}
   497	
   498		nthreads = kzalloc_objs(int, npools);
   499		rv = -ENOMEM;
   500		if (nthreads == NULL)
   501			goto out_free;
   502	
   503		/* Threads started now would run without NFSv4 state. */
Quoted 1 line

504 if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {

   505			for (i = 0; i < npools; i++) {
   506				rv = get_int(&mesg, &nthreads[i]);
   507				if (rv == -ENOENT)
   508					break;		/* fewer numbers than pools */
   509				if (rv)
   510					goto out_free;	/* syntax error */
   511				rv = -EINVAL;
   512				if (nthreads[i] < 0)
   513					goto out_free;
   514				trace_nfsd_ctl_pool_threads(net, i, nthreads[i]);
   515			}
   516	
   517			/*
   518			 * There must always be a thread in pool 0; the admin
   519			 * can't shut down NFS completely using pool_threads.
   520			 */
   521			if (nthreads[0] == 0)
   522				nthreads[0] = 1;
   523	
   524			rv = nfsd_set_nrthreads(i, nthreads, net);
   525			if (rv)
   526				goto out_free;
   527		}
   528	
   529		rv = nfsd_get_nrthreads(npools, nthreads, net);
   530		if (rv)
   531			goto out_free;
   532	
   533		mesg = buf;
   534		size = SIMPLE_TRANSACTION_LIMIT;
   535		for (i = 0; i < npools && size > 0; i++) {
   536			snprintf(mesg, size, "%d%c", nthreads[i], (i == npools-1 ? 
'\n' : ' '));
   537			len = strlen(mesg);
   538			size -= len;
   539			mesg += len;
   540		}
   541		rv = mesg - buf;
   542	out_free:
   543		kfree(nthreads);
   544		mutex_unlock(&nfsd_mutex);
   545		return rv;
   546	}
   547	
Quoted 3 lines

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

The build bot applied this patch to trees that do not include the work that's pending in the cel/nfsd-testing branch. It applied cleanly, but those trees are missing the pre-requisite nfsd-per-net-mutex series from Jeff Layton.

Interestingly, sashiko reported the same failure yesterday, but I was not able to reproduce it because I didn't realize where it was trying to apply the patch.

Since this patch is marked Cc: stable, I'll make it easy to backport by sinking it on my nfsd-testing branch below the pre-requisites and then adjusting those patches.

Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
#4 lore raw
In reply to Chuck Lever
kernel test robot noticed the following build errors:

Hi Chuck,

kernel test robot noticed the following build errors:

[auto build test ERROR on brauner-vfs/vfs.all]
[also build test ERROR on linus/master v7.3-rc6 next-20261008] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Chuck-Lever/NFSD-Don-t-start-nfsd-threads-from-a-pool_threads-write/20261009-104017
base:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.all
patch link:    https://lore.kernel.org/r/20261009144017.1284586-1-cel%40kernel.org
patch subject: [PATCH v1] NFSD: Don't start nfsd threads from a pool_threads write
config: loongarch-defconfig (https://download.01.org/0day-ci/archive/20261010/[email protected]/config)
compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 242243a5814307de825db3428662d67b064e2cbc)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261010/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> fs/nfsd/nfsctl.c:504:41: error: use of undeclared identifier 'nn'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
>> fs/nfsd/nfsctl.c:504:41: error: use of undeclared identifier 'nn'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
>> fs/nfsd/nfsctl.c:504:41: error: use of undeclared identifier 'nn'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
>> fs/nfsd/nfsctl.c:504:41: error: use of undeclared identifier 'nn'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~
>> fs/nfsd/nfsctl.c:504:41: error: use of undeclared identifier 'nn'
     504 |         if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {
         |                                                ^~

5 errors generated.

vim +/nn +504 fs/nfsd/nfsctl.c

449
450 /*
451 * write_pool_threads - Set or report the configured number of threads per pool
452 *
453 * Input:
454 * buf: ignored
455 * size: zero
456 *
457 * OR
458 *
459 * Input:
460 * buf: C string containing whitespace-
461 * separated unsigned integer values
462 * representing the number of NFSD
463 * threads to start in each pool
464 * size: non-zero length of C string in @buf
465 * Output:
466 * On success: passed-in buffer filled with '\n'-terminated C
467 * string containing integer values representing the
468 * configured number of NFSD threads in each pool;
469 * return code is the size in bytes of the string
470 * On error: return code is zero or a negative errno value
471 */
472 static ssize_t write_pool_threads(struct file *file, char *buf, size_t size)
473 {
474 /* if size > 0, look for an array of number of threads per node
475 * and apply them then write out number of threads per node as reply
476 */
477 char *mesg = buf;
478 int i;
479 int rv;
480 int len;
481 int npools;
482 int *nthreads;
483 struct net *net = netns(file);
484
485 mutex_lock(&nfsd_mutex);
486 npools = nfsd_nrpools(net);
487 if (npools == 0) {
488 /*
489 * NFS is shut down. The admin can start it by
490 * writing to the threads file but NOT the pool_threads
491 * file, sorry. Report zero threads.
492 */
493 mutex_unlock(&nfsd_mutex);
494 strcpy(buf, "0\n");
495 return strlen(buf);
496 }
497
498 nthreads = kzalloc_objs(int, npools);
499 rv = -ENOMEM;
500 if (nthreads == NULL)
501 goto out_free;
502
503 /* Threads started now would run without NFSv4 state. */

 > 504		if (size > 0 && test_bit(NFSD_NET_UP, &nn->flags)) {

505 for (i = 0; i < npools; i++) {
506 rv = get_int(&mesg, &nthreads[i]);
507 if (rv == -ENOENT)
508 break; /* fewer numbers than pools */
509 if (rv)
510 goto out_free; /* syntax error */
511 rv = -EINVAL;
512 if (nthreads[i] < 0)
513 goto out_free;
514 trace_nfsd_ctl_pool_threads(net, i, nthreads[i]);
515 }
516
517 /*
518 * There must always be a thread in pool 0; the admin
519 * can't shut down NFS completely using pool_threads.
520 */
521 if (nthreads[0] == 0)
522 nthreads[0] = 1;
523
524 rv = nfsd_set_nrthreads(i, nthreads, net);
525 if (rv)
526 goto out_free;
527 }
528
529 rv = nfsd_get_nrthreads(npools, nthreads, net);
530 if (rv)
531 goto out_free;
532
533 mesg = buf;
534 size = SIMPLE_TRANSACTION_LIMIT;
535 for (i = 0; i < npools && size > 0; i++) {
536 snprintf(mesg, size, "%d%c", nthreads[i], (i == npools-1 ? '\n' : ' '));
537 len = strlen(mesg);
538 size -= len;
539 mesg += len;
540 }
541 rv = mesg - buf;
542 out_free:
543 kfree(nthreads);
544 mutex_unlock(&nfsd_mutex);
545 return rv;
546 }
547

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki