Skip to contentSkip to conversation

nfsdctl: new nfs-utils tool for managing the kernel NFS server

PATCH Superseded Jeff Layton 2 participants 6 messages Jun 13, 2024 – Jun 17, 2024

Subsystems (2)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Jeff Layton

author maintainer
#0 lore raw

v5 cover nfsdctl: new nfs-utils tool for managing the kernel NFS server

Hi Steve,

The new netlink management interfaces [1] have been merged for v6.10 [2]. Please consider merging this series into nfs-utils. I think the code is fine but it may need some autoconf/automake love. See below...

This series adds a new tool to nfs-utils called nfsdctl, which is intended as an eventual replacement for rpc.nfsd (and maybe other tools). It's a subcommand based interface like nmcli or virsh, so we can easily expand the interface later to deal with new sorts of configuration.

This version of the tool should be at feature parity with rpc.nfsd, at least as far as autostarting the server. This posting also includes a manpage and an update to the nfs-server systemctl service, to start using the new interface when possible.

I've also included a patch that adds the manpage source. It's much nicer to edit that and regenerate it if we have to update it later. We can drop that patch if you just want to keep the result though.

This does ship a copy of nfsd_netlink.h along with the tool that gets used if the uapi headers aren't new enough. This is a temporary measure. Once new enough kernels are shipping in the field (in a year or so?), we can drop that and some related autoconf junk.

[1]: https://lore.kernel.org/linux-nfs/[email protected]/T/#m5fd847189894f58e93706c40340e18858f242a27
[2]: https://lore.kernel.org/linux-nfs/171606732267.14195.18399250065227381901.pr-tracker-bot@kernel.org/T/#t

---
Changes in v5:
- add support for pool-mode setting
- fix up the handling of nfsd_netlink.h in autoconf
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- add ability to specify an array of pool thread counts in nfs.conf
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- split nfsdctl.h so we can include the UAPI header as-is
- squash the patches together that added Lorenzo's version and convert it to the new interface
- adapt to latest version of netlink interface changes
+ have THREADS_SET/GET report an array of thread counts (one per pool)
+ pass scope in as a string to THREADS_SET instead of using unshare() trick

Changes in v2:
- Adapt to latest kernel netlink interface changes (in particular, send the leastime and gracetime when they are set in the config).
- More help text for different subcommands
- New nfsdctl(8) manpage
- Patch to make systemd preferentially use nfsdctl instead of rpc.nfsd
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Jeff Layton (3):
nfsdctl: asciidoc source for the manpage
systemd: use nfsdctl to start and stop the nfs server
nfsdctl: add support for pool-mode

Lorenzo Bianconi (1):
nfsdctl: add the nfsdctl utility to nfs-utils

 configure.ac                 |   19 +
 systemd/nfs-server.service   |    4 +-
 utils/Makefile.am            |    4 +
 utils/nfsdctl/Makefile.am    |   13 +
 utils/nfsdctl/nfsd_netlink.h |   96 +++
 utils/nfsdctl/nfsdctl.8      |  293 ++++++++
 utils/nfsdctl/nfsdctl.adoc   |  152 ++++
 utils/nfsdctl/nfsdctl.c      | 1584 ++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdctl/nfsdctl.h      |   93 +++
 9 files changed, 2256 insertions(+), 2 deletions(-)

---
base-commit: 94b48ccc0b0304809027fcead03343f4c716c4f4
change-id: 20240412-nfsdctl-fa8bd8430cfd

Best regards,

Signature · 1 line
Jeff Layton <[email protected]>
author maintainer
#1 lore raw

PATCH v5 1/4 nfsdctl: add the nfsdctl utility to nfs-utils

From: Lorenzo Bianconi <[email protected]>

This tool is based on Lorenzo's original nfsdctl tool [1]. His original tool used getopt_long to indicate the command, but that's somewhat limiting. This converts it to a subcommand-based interface, where each subcommand can take its own options, in the spirit of commands like nmcli or virsh.

There are currently 5 different subcommands:

    listener             get/set listener info
    version              get/set supported NFS versions
    threads              get/set nfsd thread settings
    status               get current RPC processing info
    autostart            start server with settings from /etc/nfs.conf

Each can take different options, and we can expand this interface later with more commands as necessary.

[1]: https://github.com/LorenzoBianconi/nfsdctl

---
 configure.ac                 |   19 +
 utils/Makefile.am            |    4 +
 utils/nfsdctl/Makefile.am    |   13 +
 utils/nfsdctl/nfsd_netlink.h |   86 +++
 utils/nfsdctl/nfsdctl.8      |  274 ++++++++
 utils/nfsdctl/nfsdctl.c      | 1460 ++++++++++++++++++++++++++++++++++++++++++
 utils/nfsdctl/nfsdctl.h      |   93 +++
 7 files changed, 1949 insertions(+)
configure.ac +19 -0
index 58d1728c5bc6..8cb42ebc5846 100644
@@ -252,6 +252,24 @@ AC_ARG_ENABLE(nfsdcltrack,
 	enable_nfsdcltrack=$enableval,
 	enable_nfsdcltrack="yes")
 
+AC_ARG_ENABLE(nfsdctl,
+	[AS_HELP_STRING([--disable-nfsdctl],[disable nfsdctl program for controlling nfsd@<:@default=no@:>@])],
+	enable_nfsdctl=$enableval,
+	enable_nfsdctl="yes")
+	AM_CONDITIONAL(CONFIG_NFSDCTL, [test "$enable_nfsdctl" = "yes" ])
+	if test "$enable_nfsdctl" = yes; then
+		PKG_CHECK_MODULES(LIBNL3, libnl-3.0 >= 3.1)
+		PKG_CHECK_MODULES(LIBNLGENL3, libnl-genl-3.0 >= 3.1)
+		PKG_CHECK_MODULES(LIBREADLINE, readline)
+		AC_CHECK_HEADERS(linux/nfsd_netlink.h)
+
+		# ensure we have the pool-mode commands
+		AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <linux/nfsd_netlink.h>]],
+				                   [[int foo = NFSD_CMD_LISTENER_GET;]])],
+				   [AC_DEFINE([USE_SYSTEM_NFSD_NETLINK_H], 1,
+					      ["Use system's linux/nfsd_netlink.h"])])
+	fi
+
 AC_ARG_ENABLE(nfsv4server,
 	[AS_HELP_STRING([--enable-nfsv4server],[enable support for NFSv4 only server  @<:@default=no@:>@])],
 	enable_nfsv4server=$enableval,
@@ -739,6 +757,7 @@ AC_CONFIG_FILES([
 	utils/mountd/Makefile
 	utils/exportd/Makefile
 	utils/nfsd/Makefile
+	utils/nfsdctl/Makefile
 	utils/nfsref/Makefile
 	utils/nfsstat/Makefile
 	utils/nfsidmap/Makefile
utils/Makefile.am +4 -0
index ab584190f17e..e5cb81e796bd 100644
@@ -35,6 +35,10 @@ if CONFIG_JUNCTION
 OPTDIRS += nfsref
 endif
 
+if CONFIG_NFSDCTL
+OPTDIRS += nfsdctl
+endif
+
 SUBDIRS = \
 	exportfs \
 	mountd \
utils/nfsdctl/Makefile.am new file +13 -0
new file mode 100644
index 000000000000..89c7ecd6f30b
@@ -0,0 +1,13 @@
+## Process this file with automake to produce Makefile.in
+
+man8_MANS       = nfsdctl.8
+EXTRA_DIST      = $(man8_MANS)
+
+sbin_PROGRAMS	= nfsdctl
+
+noinst_HEADERS = nfsdctl.h
+nfsdctl_SOURCES = nfsdctl.c
+nfsdctl_CFLAGS = $(LIBNL3_CFLAGS) $(LIBNLGENL3_CFLAGS) $(LIBREADLINE_CFLAGS)
+nfsdctl_LDADD = ../../support/nfs/libnfs.la $(LIBNL3_LIBS) $(LIBNLGENL3_LIBS) $(LIBREADLINE_LIBS)
+
+MAINTAINERCLEANFILES = Makefile.in
utils/nfsdctl/nfsd_netlink.h new file +86 -0

86 lines not on this page · raw message

utils/nfsdctl/nfsdctl.8 new file +274 -0

275 lines not on this page · raw message

utils/nfsdctl/nfsdctl.c new file +1460 -0

1,460 lines not on this page · raw message

utils/nfsdctl/nfsdctl.h new file +93 -0

93 lines not on this page · raw message

Signature · 1 line
2.45.2
author maintainer
#2 lore raw

PATCH v5 2/4 nfsdctl: asciidoc source for the manpage

In reply to Jeff Layton

Convert to manpage with:

asciidoctor -b manpage nfsdctl.adoc

---
 utils/nfsdctl/nfsdctl.adoc | 140 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 140 insertions(+)
utils/nfsdctl/nfsdctl.adoc new file +140 -0
new file mode 100644
index 000000000000..b2f00c11829e
@@ -0,0 +1,140 @@
+= nfsdctl(8)
+Jeff Layton
+:doctype: manpage
+
+== NAME
+
+nfsdctl - control program for the Linux kernel NFS server
+
+== SYNOPSIS
+
+*nfsdctl* [ _OPTION_ ] COMMAND ...
+
+== DESCRIPTION
+
+nfsdctl is a control and query program for the in-kernel NFS server. There are several
+subcommands (documented below) that allow the admin to configure or query different
+aspects of the NFS server.
+
+To get information about different subcommand usage, pass the subcommand the
+--help parameter. For example:
+
+    nfsdctl listener --help
+
+== OPTIONS
+
+*-d, --debug*::
+  Enable debug logging
+
+*-h, --help*::
+  Print program help text
+
+*-V, --version*::
+  Print program version
+
+== SUBCOMMANDS
+
+Each subcommand can also accept its own set of options and arguments. The
+--help option is standard for all subcommands:
+
+*autostart*::
+  Start the server using the settings in the [nfsd] section of /etc/nfs.conf.
+  This subcommand takes no arguments.
+
+*listener*::
+
+  Get/set the listening sockets for the server. Run this without arguments to
+  get a list of the sockets on which the server is currently listening. To add
+  or remove sockets, pass it whitespace-separated strings in the format:
+
+    { +|- }{ protocol }:{ address }:{ port }
+
+  The fields are:
+
+    + to add a listener, - to remove one
+    protocol: protocol name (e.g. tcp, udp, rdma)
+    address: hostname or address
+    port: port number or service name
+
+  All fields are required, except for the address. If address is an empty string,
+  then the listeners will be opened for INADDR_ANY and IN6ADDR_ANY_INIT for ipv6
+  (if enabled). The address can be either a hostname or an IP address. IPv4
+  addresses must be in dotted-quad form. IPv6 addresses should be in standard
+  colon separated form, and must be enclosed in square brackets.
+
+*status*::
+
+  Get information about RPCs currently executing in the server. This subcommand
+  takes no arguments.
+
+*threads*::
+
+  Get/set the number of running nfsd threads in each pool. Pass a list of
+  integers to change the currently active number of threads. Passing it a
+  value of 0 will shut down the NFS server. Run this without arguments to
+  get the current number of running threads in each pool.
+
+*version*::
+
+  Get/set the enabled NFS versions for the server. Run without arguments to
+  get a list of supported versions and whether they are currently enabled or
+  disabled. To enable or disable a version, pass it a string in the format:
+
+        { +|- }{ MAJOR }{.{ MINOR }}
+
+  The fields are:
+
+    + to enable a version, - to disable
+    MAJOR: the major version integer value
+    MINOR: the minor version integet value
+
+  The minorversion field is optional. If not given, it will disable or enable
+  all minorversions for that major version.
+
+== EXAMPLES
+
+Start the server with the settings in nfs.conf:
+
+  nfsdctl autostart
+
+Get a list of current listening sockets:
+
+  nfsdctl listener
+
+Show the supported and enabled NFS versions:
+
+  nfsdctl version
+
+Add TCP listener on all addresses (both v4 and v6), port 2049:
+
+  nfsdctl listener +tcp::2049
+
+Add RDMA listener on 1.2.3.4 port 20049:
+
+  nfsdctl listener +rdma:1.2.3.4:20049
+
+Add same listener on IPv6 address f00::ba4 port 20050:
+
+  nfsdctl listener +rdma:[f00::ba4]:20050
+
+Enable NFS version 3, disable v4.0:
+
+  nfsdctl version +3 -4.0
+
+Change the number of running threads in first pool to 256:
+
+  nfsdctl threads 256
+
+== NOTES
+
+nfsdctl is intended to supersede rpc.nfsd(8), which controls the nfs server
+using the files under /proc/fs/nfsd. nfsdctl instead uses a netlink(7)
+interface to achieve the same goals.
+
+Most of the commands that query the NFS server can be run as an unprivileged
+user, but configuring the server typically requires an account with elevated
+privileges.
+
+== SEE ALSO
+
+nfs.conf(5), rpc.nfsd(8), rpc.mountd(8), exports(5), exportfs(8), nfs.conf(5), rpc.rquotad(8),  nfsstat(8), netconfig(5)
Signature · 1 line
2.45.2
author maintainer
#3 lore raw

PATCH v5 3/4 systemd: use nfsdctl to start and stop the nfs server

In reply to Jeff Layton

Attempt to use nfsdctl to start and stop the nfs-server. If that fails for any reason, use rpc.nfsd to do it instead.

---
 systemd/nfs-server.service | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
systemd/nfs-server.service +2 -2
index ac17d5286d3b..8793856aa707 100644
@@ -23,8 +23,8 @@ After=rpc-gssd.service gssproxy.service rpc-svcgssd.service
 Type=oneshot
 RemainAfterExit=yes
 ExecStartPre=-/usr/sbin/exportfs -r
-ExecStart=/usr/sbin/rpc.nfsd
-ExecStop=/usr/sbin/rpc.nfsd 0
+ExecStart=/bin/sh -c '/usr/sbin/nfsdctl autostart || /usr/sbin/rpc.nfsd'
+ExecStop=/bin/sh -c '/usr/sbin/nfsdctl threads 0 || /usr/sbin/rpc.nfsd 0'
 ExecStopPost=/usr/sbin/exportfs -au
 ExecStopPost=/usr/sbin/exportfs -f
 
Signature · 1 line
2.45.2
author maintainer
#4 lore raw

PATCH v5 4/4 nfsdctl: add support for pool-mode

In reply to Jeff Layton

Allow nfsdctl to use the new POOL_MODE_GET/SET operations to configure the pool-mode on the host. This setting mirrors the behavior of the sunrpc module parameter. The patch also adds support for a new pool-mode option in /etc/nfs.conf. With this change, you can now configure a multipool NFS server by just properly setting up /etc/nfs.conf.

---
 configure.ac                 |   2 +-
 utils/nfsdctl/nfsd_netlink.h |  10 +++
 utils/nfsdctl/nfsdctl.8      |  23 ++++++-
 utils/nfsdctl/nfsdctl.adoc   |  12 ++++
 utils/nfsdctl/nfsdctl.c      | 154 ++++++++++++++++++++++++++++++++++++++-----
 5 files changed, 183 insertions(+), 18 deletions(-)
configure.ac +1 -1
index 8cb42ebc5846..1aebc3a767a7 100644
@@ -265,7 +265,7 @@ AC_ARG_ENABLE(nfsdctl,
 
 		# ensure we have the pool-mode commands
 		AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <linux/nfsd_netlink.h>]],
-				                   [[int foo = NFSD_CMD_LISTENER_GET;]])],
+				                   [[int foo = NFSD_CMD_POOL_MODE_GET;]])],
 				   [AC_DEFINE([USE_SYSTEM_NFSD_NETLINK_H], 1,
 					      ["Use system's linux/nfsd_netlink.h"])])
 	fi
utils/nfsdctl/nfsd_netlink.h +10 -0
index 24c86dbc7ed5..887cbd12b695 100644
@@ -70,6 +70,14 @@ enum {
 	NFSD_A_SERVER_SOCK_MAX = (__NFSD_A_SERVER_SOCK_MAX - 1)
 };
 
+enum {
+	NFSD_A_POOL_MODE_MODE = 1,
+	NFSD_A_POOL_MODE_NPOOLS,
+
+	__NFSD_A_POOL_MODE_MAX,
+	NFSD_A_POOL_MODE_MAX = (__NFSD_A_POOL_MODE_MAX - 1)
+};
+
 enum {
 	NFSD_CMD_RPC_STATUS_GET = 1,
 	NFSD_CMD_THREADS_SET,
@@ -78,6 +86,8 @@ enum {
 	NFSD_CMD_VERSION_GET,
 	NFSD_CMD_LISTENER_SET,
 	NFSD_CMD_LISTENER_GET,
+	NFSD_CMD_POOL_MODE_SET,
+	NFSD_CMD_POOL_MODE_GET,
 
 	__NFSD_CMD_MAX,
 	NFSD_CMD_MAX = (__NFSD_CMD_MAX - 1)
utils/nfsdctl/nfsdctl.8 +21 -2
index 7f5d981042a5..d55862cc1fc1 100644
@@ -2,12 +2,12 @@
 .\"     Title: nfsdctl
 .\"    Author: Jeff Layton
 .\" Generator: Asciidoctor 2.0.20
-.\"      Date: 2024-04-23
+.\"      Date: 2024-06-13
 .\"    Manual: \ \&
 .\"    Source: \ \&
 .\"  Language: English
 .\"
-.TH "NFSDCTL" "8" "2024-04-23" "\ \&" "\ \&"
+.TH "NFSDCTL" "8" "2024-06-13" "\ \&" "\ \&"
 .ie \n(.g .ds Aq \(aq
 .el       .ds Aq '
 .ss \n[.ss] 0
@@ -176,6 +176,25 @@ all minorversions for that major version.
 .fi
 .if n .RE
 .RE
+.sp
+\fBpool\-mode\fP
+.RS 4
+Get/set the host\(cqs pool mode. This will cause the server to start threads
+that are pinned to either the CPU or the NUMA node. This can only be set
+when there are no nfsd threads running.
+.sp
+.if n .RS 4
+.nf
+.fam C
+The available options are:
+  global:  single large pool
+  percpu:  pool per CPU
+  pernode: pool per NUMA node
+  auto:    choose a mode based on host configuration
+.fam
+.fi
+.if n .RE
+.RE
 .SH "EXAMPLES"
 .sp
 Start the server with the settings in nfs.conf:
utils/nfsdctl/nfsdctl.adoc +12 -0
index b2f00c11829e..03c5c281d92e 100644
@@ -91,6 +91,18 @@ Each subcommand can also accept its own set of options and arguments. The
   The minorversion field is optional. If not given, it will disable or enable
   all minorversions for that major version.
 
+*pool-mode*::
+
+  Get/set the host's pool mode. This will cause the server to start threads
+  that are pinned to either the CPU or the NUMA node. This can only be set
+  when there are no nfsd threads running.
+
+  The available options are:
+    global:  single large pool
+    percpu:  pool per CPU
+    pernode: pool per NUMA node
+    auto:    choose a mode based on host configuration
+
 == EXAMPLES
 
 Start the server with the settings in nfs.conf:
utils/nfsdctl/nfsdctl.c +139 -15

214 lines not on this page · raw message

Signature · 1 line
2.45.2
#5 lore raw
In reply to Jeff Layton

Jeff,

Quoted 5 lines · Jeff Layton

Hi Steve,

The new netlink management interfaces [1] have been merged for v6.10 [2]. Please consider merging this series into nfs-utils. I think the code is fine but it may need some autoconf/automake love. See below...

With these... I'm going to make a release, once I get a couple other patces sorted out, and then apply these as the first patches after the release since they are kernel dependent.

steved.

Quoted 75 lines

This series adds a new tool to nfs-utils called nfsdctl, which is intended as an eventual replacement for rpc.nfsd (and maybe other tools). It's a subcommand based interface like nmcli or virsh, so we can easily expand the interface later to deal with new sorts of configuration.

This version of the tool should be at feature parity with rpc.nfsd, at least as far as autostarting the server. This posting also includes a manpage and an update to the nfs-server systemctl service, to start using the new interface when possible.

I've also included a patch that adds the manpage source. It's much nicer to edit that and regenerate it if we have to update it later. We can drop that patch if you just want to keep the result though.

This does ship a copy of nfsd_netlink.h along with the tool that gets used if the uapi headers aren't new enough. This is a temporary measure. Once new enough kernels are shipping in the field (in a year or so?), we can drop that and some related autoconf junk.

[1]: https://lore.kernel.org/linux-nfs/[email protected]/T/#m5fd847189894f58e93706c40340e18858f242a27
[2]: https://lore.kernel.org/linux-nfs/171606732267.14195.18399250065227381901.pr-tracker-bot@kernel.org/T/#t

---
Changes in v5:
- add support for pool-mode setting
- fix up the handling of nfsd_netlink.h in autoconf
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- add ability to specify an array of pool thread counts in nfs.conf
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- split nfsdctl.h so we can include the UAPI header as-is
- squash the patches together that added Lorenzo's version and convert it to the new interface
- adapt to latest version of netlink interface changes
+ have THREADS_SET/GET report an array of thread counts (one per pool)
+ pass scope in as a string to THREADS_SET instead of using unshare() trick

Changes in v2:
- Adapt to latest kernel netlink interface changes (in particular, send the leastime and gracetime when they are set in the config).
- More help text for different subcommands
- New nfsdctl(8) manpage
- Patch to make systemd preferentially use nfsdctl instead of rpc.nfsd
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Jeff Layton (3):
nfsdctl: asciidoc source for the manpage
systemd: use nfsdctl to start and stop the nfs server
nfsdctl: add support for pool-mode

Lorenzo Bianconi (1):
nfsdctl: add the nfsdctl utility to nfs-utils

  configure.ac                 |   19 +
  systemd/nfs-server.service   |    4 +-
  utils/Makefile.am            |    4 +
  utils/nfsdctl/Makefile.am    |   13 +
  utils/nfsdctl/nfsd_netlink.h |   96 +++
  utils/nfsdctl/nfsdctl.8      |  293 ++++++++
  utils/nfsdctl/nfsdctl.adoc   |  152 ++++
  utils/nfsdctl/nfsdctl.c      | 1584 ++++++++++++++++++++++++++++++++++++++++++
  utils/nfsdctl/nfsdctl.h      |   93 +++
  9 files changed, 2256 insertions(+), 2 deletions(-)

---
base-commit: 94b48ccc0b0304809027fcead03343f4c716c4f4
change-id: 20240412-nfsdctl-fa8bd8430cfd

Best regards,