Skip to contentSkip to conversation

nfsd: expose NFSv4 client state through Netlink

PATCH In review Prabhakar Pujeri 2 participants 4 messages Sep 3

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v3 cover nfsd: expose NFSv4 client state through Netlink

Administrators can inspect NFSv4 clients through the nfsd filesystem, but collecting a server-wide view requires walking one directory and opening multiple files for every client. That interface remains useful for detailed inspection, but it is awkward for monitoring and automation.

Add a compact, privileged query to the existing nfsd Generic Netlink family:

1. Dump basic NFSv4 client identity, address, lease, confirmation, reclaim, and callback state.
2. Add O(1) snapshots of per-client session and stateid record counts.

The existing nfsd filesystem files and their output remain unchanged. A companion nfs-utils v2 series adds nfsdctl clients; its cover Message-ID is:

<[email protected]>

This reroll intentionally drops the grace-status patch. As discussed with Chuck, that interface needs an explicitly scoped design that can represent both current server-wide grace and possible per-export grace in the future. I will address it in a separate proposal.

Changes since v2 (feedback from Jeff Layton):

- rename the client lookup helper to nfsd_nl_find_client() ("get" connotes refcounting)
- validate the client-table argument explicitly in nfsd_nl_find_client() instead of aliasing NFSD4_NL_CLIENT_DONE to "unconfirmed"
- rename nfsd4_nl_client_put_address() to nfsd4_nl_client_marshal_address() ("put" connotes refcounting)
- document the confirmed-then-unconfirmed table ordering in the dump loop comments
- rebase onto nfsd-testing at ac579868af0c

Changes since v1:

- replace the raw sockaddr attribute with separate IPv4/IPv6 address, port, and optional IPv6 scope-ID attributes
- add a nonzero client-table generation and genl_dump_check_consistent(), so churn marks the dump NLM_F_DUMP_INTR
- replace session-list and stateid-IDR walks under cl_lock with maintained u64 counters, making every per-client snapshot O(1)
- encode counts as variable-width Netlink uint attributes
- drop the grace-status patch for a separately scoped design
- rebase onto nfsd-testing at 65b583e874b3

Validation performed:

- strict checkpatch on both patches
- focused NFSD W=1 build of fs/nfsd on nfsd-testing at ac579868af0c

v1: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]

Prabhakar Pujeri (2):
nfsd: add a Netlink dump of NFSv4 clients
nfsd: report per-client NFSv4 state usage through Netlink

 .../admin-guide/nfs/nfsd-admin-interfaces.rst |  22 ++
 Documentation/netlink/specs/nfsd.yaml         | 118 ++++++
 fs/nfsd/netlink.c                             |   5 +
 fs/nfsd/netlink.h                             |   1 +
 fs/nfsd/netns.h                               |   1 +
 fs/nfsd/nfs4ctl.h                             |  10 +
 fs/nfsd/nfs4layouts.c                         |   2 +-
 fs/nfsd/nfs4state.c                           | 353 +++++++++++++++++-
 fs/nfsd/nfsctl.c                              |  13 +
 fs/nfsd/state.h                               |   7 +
 include/uapi/linux/nfsd_netlink.h             |  43 +++
 11 files changed, 565 insertions(+), 10 deletions(-)

base-commit: ac579868af0c900d8fd9c8cfe9e10db46ccc5a75

Signature · 1 line
2.55.0
#1 lore raw

PATCH v3 1/2 nfsd: add a Netlink dump of NFSv4 clients

Administrators currently have to walk one nfsd filesystem directory per client to correlate basic NFSv4 identity, lease, and callback information. That interface is useful for detailed inspection, but it is awkward for monitoring tools and provides no atomic way to enumerate the client set.

Add a privileged client-get dump to the nfsd Generic Netlink family. Emit one bounded message per confirmed or unconfirmed client with its server-generated client ID, peer address, minor version, client and callback states, signed lease time remaining, and RECLAIM_COMPLETE status.

Represent the peer as separate IPv4 or IPv6 address, port, and optional scope-ID attributes instead of exposing a raw sockaddr structure.

Pin each client while taking a snapshot, protect mutable lease and confirmation fields with the per-net client lock, and serialize against server shutdown with nfsd_mutex. Track client-table changes with a nonzero generation counter and use genl_dump_check_consistent() so a dump that can skip or repeat a client is marked NLM_F_DUMP_INTR. Leave the existing nfsd filesystem interface unchanged.

---
Changes since v2 (feedback from Jeff Layton):

- rename the client lookup helper to nfsd_nl_find_client()
- validate the client-table argument explicitly instead of aliasing NFSD4_NL_CLIENT_DONE to "unconfirmed"
- rename nfsd4_nl_client_put_address() to nfsd4_nl_client_marshal_address()
- document the confirmed-then-unconfirmed table ordering in the dump loop comments

Changes since v1:

- split raw sockaddr data into address, port, and scope-ID attributes
- detect client-table churn with a generation counter and NLM_F_DUMP_INTR

v1: https://lore.kernel.org/r/6b42a390ce3ea9a5930e2704137fd435c6f7d27e.1787638668.git.prabhakar.pujeri@dell.com
v2: https://lore.kernel.org/r/2d682b47dca234b5c270b0b1ea03022c28bbeaa3.1787988917.git.prabhakar.pujeri@dell.com

 .../admin-guide/nfs/nfsd-admin-interfaces.rst |  16 ++
 Documentation/netlink/specs/nfsd.yaml         |  93 +++++++
 fs/nfsd/netlink.c                             |   5 +
 fs/nfsd/netlink.h                             |   1 +
 fs/nfsd/netns.h                               |   1 +
 fs/nfsd/nfs4ctl.h                             |  10 +
 fs/nfsd/nfs4state.c                           | 260 ++++++++++++++++++
 fs/nfsd/nfsctl.c                              |  13 +
 include/uapi/linux/nfsd_netlink.h             |  38 +++
 9 files changed, 437 insertions(+)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +16 -0
index c05926f7905..de2a5402587 100644
@@ -29,6 +29,22 @@ Between startup and shutdown, the number of threads may be adjusted up
 or down by additional writes to nfsd/threads or by writes to
 nfsd/pool_threads.
 
+NFSv4 client visibility
+=======================
+
+The privileged ``client-get`` dump in the ``nfsd`` Generic Netlink family
+emits one message for each NFSv4 client.  Each message identifies the client
+by its server-generated client ID and transport address, then reports its
+minor version, client and callback states, signed lease time remaining, and
+whether an NFSv4.1 or later client sent RECLAIM_COMPLETE.
+
+Clients can change between messages.  If that can make the dump skip or repeat
+a record, the kernel sets ``NLM_F_DUMP_INTR`` and userspace should retry.
+
+The existing ``/proc/fs/nfsd/clients/`` files remain available for inspection.
+The ``states`` file contains individual stateids, and writing ``expire`` to
+``ctl`` forcibly removes the client and all state it owns.
+
 For more detail about files under nfsd/ and what they control, see
 fs/nfsd/nfsctl.c; most of them have detailed comments.
 
Documentation/netlink/specs/nfsd.yaml +93 -0

111 lines not on this page · raw message

fs/nfsd/netlink.c +5 -0
index eba8b353f41..48bc499136b 100644
@@ -230,6 +230,11 @@ static const struct genl_split_ops nfsd_nl_ops[] = {
 		.dumpit	= nfsd_nl_server_stats_get_dumpit,
 		.flags	= GENL_CMD_CAP_DUMP,
 	},
+	{
+		.cmd	= NFSD_CMD_CLIENT_GET,
+		.dumpit	= nfsd_nl_client_get_dumpit,
+		.flags	= GENL_ADMIN_PERM | GENL_CMD_CAP_DUMP,
+	},
 };
 
 static const struct genl_multicast_group nfsd_nl_mcgrps[] = {
fs/nfsd/netlink.h +1 -0
index 027e2953db2..de7593e6408 100644
@@ -44,6 +44,7 @@ int nfsd_nl_unlock_filesystem_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_unlock_export_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
 				    struct netlink_callback *cb);
+int nfsd_nl_client_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
 
 enum {
 	NFSD_NLGRP_NONE,
fs/nfsd/netns.h +1 -0
index 0ce7da20aba..d30af86cf78 100644
@@ -105,6 +105,7 @@ struct nfsd_net {
 	struct list_head *unconf_id_hashtbl;
 	struct rb_root unconf_name_tree;
 	struct list_head *sessionid_hashtbl;
+	u32 nfs4_client_generation;	/* protected by client_lock, never zero */
 	/*
 	 * client_lru holds client queue ordered by nfs4_client.cl_time
 	 * for lease renewal.
fs/nfsd/nfs4ctl.h +10 -0
index bcec4c4ef1d..44cbb0fac58 100644
@@ -20,8 +20,10 @@
 struct net;
 struct inode;
 struct dentry;
+struct sk_buff;
 struct svc_rqst;
 struct nfsd_net;
+struct netlink_callback;
 
 #ifdef CONFIG_NFSD_V4
 extern unsigned long max_delegations;
@@ -37,6 +39,8 @@ bool nfsd4_spo_must_allow(struct svc_rqst *rqstp);
 int nfsd4_create_laundry_wq(void);
 void nfsd4_destroy_laundry_wq(void);
 bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp, struct inode *inode);
+int nfsd4_nl_client_get_dumpit(struct sk_buff *skb,
+			       struct netlink_callback *cb);
 
 extern int nfsd4_is_junction(struct dentry *dentry);
 extern int register_cld_notifier(void);
@@ -68,6 +72,12 @@ static inline bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp,
 	return false;
 }
 
+static inline int nfsd4_nl_client_get_dumpit(struct sk_buff *skb,
+					     struct netlink_callback *cb)
+{
+	return 0;
+}
+
 static inline int nfsd4_is_junction(struct dentry *dentry)
 {
 	return 0;
fs/nfsd/nfs4state.c +260 -0

302 lines not on this page · raw message

fs/nfsd/nfsctl.c +13 -0
index c6f6bc3b128..4d11592cdf8 100644
@@ -1649,6 +1649,19 @@ int nfsd_nl_rpc_status_get_dumpit(struct sk_buff *skb,
 	return ret;
 }
 
+/**
+ * nfsd_nl_client_get_dumpit - dump NFSv4 client information
+ * @skb: reply buffer
+ * @cb: netlink metadata and command arguments
+ *
+ * Returns the size of the reply or a negative errno.
+ */
+int nfsd_nl_client_get_dumpit(struct sk_buff *skb,
+			      struct netlink_callback *cb)
+{
+	return nfsd4_nl_client_get_dumpit(skb, cb);
+}
+
 /**
  * nfsd_nl_fh_key_set - helper to copy fh_key from userspace
  * @attr: nlattr NFSD_A_SERVER_FH_KEY
include/uapi/linux/nfsd_netlink.h +38 -0
index 87da1d0bb21..715acb4104c 100644
@@ -50,6 +50,26 @@ enum nfsd_xprtsec_mode {
 	NFSD_XPRTSEC_MODE_MTLS = 4,
 };
 
+/*
+ * State of an NFSv4 client record.
+ */
+enum nfsd_client_state {
+	NFSD_CLIENT_STATE_UNCONFIRMED,
+	NFSD_CLIENT_STATE_ACTIVE,
+	NFSD_CLIENT_STATE_COURTESY,
+	NFSD_CLIENT_STATE_EXPIRABLE,
+};
+
+/*
+ * State of an NFSv4 client's callback channel.
+ */
+enum nfsd_callback_state {
+	NFSD_CALLBACK_STATE_UP,
+	NFSD_CALLBACK_STATE_UNKNOWN,
+	NFSD_CALLBACK_STATE_DOWN,
+	NFSD_CALLBACK_STATE_FAULT,
+};
+
 enum {
 	NFSD_A_CACHE_NOTIFY_CACHE_TYPE = 1,
 
@@ -260,6 +280,23 @@ enum {
 	NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX - 1)
 };
 
+enum {
+	NFSD_A_CLIENT_CLIENTID = 1,
+	NFSD_A_CLIENT_PAD,
+	NFSD_A_CLIENT_ADDRESS4,
+	NFSD_A_CLIENT_ADDRESS6,
+	NFSD_A_CLIENT_ADDRESS_PORT,
+	NFSD_A_CLIENT_ADDRESS_SCOPE_ID,
+	NFSD_A_CLIENT_MINOR_VERSION,
+	NFSD_A_CLIENT_STATE,
+	NFSD_A_CLIENT_LEASE_REMAINING,
+	NFSD_A_CLIENT_RECLAIM_COMPLETE,
+	NFSD_A_CLIENT_CALLBACK_STATE,
+
+	__NFSD_A_CLIENT_MAX,
+	NFSD_A_CLIENT_MAX = (__NFSD_A_CLIENT_MAX - 1)
+};
+
 enum {
 	NFSD_CMD_RPC_STATUS_GET = 1,
 	NFSD_CMD_THREADS_SET,
@@ -280,6 +317,7 @@ enum {
 	NFSD_CMD_UNLOCK_FILESYSTEM,
 	NFSD_CMD_UNLOCK_EXPORT,
 	NFSD_CMD_SERVER_STATS_GET,
+	NFSD_CMD_CLIENT_GET,
 
 	__NFSD_CMD_MAX,
 	NFSD_CMD_MAX = (__NFSD_CMD_MAX - 1)
Signature · 1 line
2.55.0
maintainer
#2 lore raw

Re: [PATCH v3 1/2] nfsd: add a Netlink dump of NFSv4 clients

Quoted 521 lines · Prabhakar Pujeri

Administrators currently have to walk one nfsd filesystem directory per client to correlate basic NFSv4 identity, lease, and callback information. That interface is useful for detailed inspection, but it is awkward for monitoring tools and provides no atomic way to enumerate the client set.

Add a privileged client-get dump to the nfsd Generic Netlink family. Emit one bounded message per confirmed or unconfirmed client with its server-generated client ID, peer address, minor version, client and callback states, signed lease time remaining, and RECLAIM_COMPLETE status.

Represent the peer as separate IPv4 or IPv6 address, port, and optional scope-ID attributes instead of exposing a raw sockaddr structure.

Pin each client while taking a snapshot, protect mutable lease and confirmation fields with the per-net client lock, and serialize against server shutdown with nfsd_mutex. Track client-table changes with a nonzero generation counter and use genl_dump_check_consistent() so a dump that can skip or repeat a client is marked NLM_F_DUMP_INTR. Leave the existing nfsd filesystem interface unchanged.

---
Changes since v2 (feedback from Jeff Layton):

- rename the client lookup helper to nfsd_nl_find_client()
- validate the client-table argument explicitly instead of aliasing NFSD4_NL_CLIENT_DONE to "unconfirmed"
- rename nfsd4_nl_client_put_address() to nfsd4_nl_client_marshal_address()
- document the confirmed-then-unconfirmed table ordering in the dump loop comments

Changes since v1:

- split raw sockaddr data into address, port, and scope-ID attributes
- detect client-table churn with a generation counter and NLM_F_DUMP_INTR

v1: https://lore.kernel.org/r/6b42a390ce3ea9a5930e2704137fd435c6f7d27e.1787638668.git.prabhakar.pujeri@dell.com
v2: https://lore.kernel.org/r/2d682b47dca234b5c270b0b1ea03022c28bbeaa3.1787988917.git.prabhakar.pujeri@dell.com

 .../admin-guide/nfs/nfsd-admin-interfaces.rst |  16 ++
 Documentation/netlink/specs/nfsd.yaml         |  93 +++++++
 fs/nfsd/netlink.c                             |   5 +
 fs/nfsd/netlink.h                             |   1 +
 fs/nfsd/netns.h                               |   1 +
 fs/nfsd/nfs4ctl.h                             |  10 +
 fs/nfsd/nfs4state.c                           | 260 ++++++++++++++++++
 fs/nfsd/nfsctl.c                              |  13 +
 include/uapi/linux/nfsd_netlink.h             |  38 +++
 9 files changed, 437 insertions(+)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +16 -0
index c05926f7905..de2a5402587 100644
@@ -29,6 +29,22 @@ Between startup and shutdown, the number of threads may be adjusted up
 or down by additional writes to nfsd/threads or by writes to
 nfsd/pool_threads.
 
+NFSv4 client visibility
+=======================
+
+The privileged ``client-get`` dump in the ``nfsd`` Generic Netlink family
+emits one message for each NFSv4 client.  Each message identifies the client
+by its server-generated client ID and transport address, then reports its
+minor version, client and callback states, signed lease time remaining, and
+whether an NFSv4.1 or later client sent RECLAIM_COMPLETE.
+
+Clients can change between messages.  If that can make the dump skip or repeat
+a record, the kernel sets ``NLM_F_DUMP_INTR`` and userspace should retry.
+
+The existing ``/proc/fs/nfsd/clients/`` files remain available for inspection.
+The ``states`` file contains individual stateids, and writing ``expire`` to
+``ctl`` forcibly removes the client and all state it owns.
+
 For more detail about files under nfsd/ and what they control, see
 fs/nfsd/nfsctl.c; most of them have detailed comments.
 
Documentation/netlink/specs/nfsd.yaml +93 -0
index 642268819c6..9207a96fe59 100644
@@ -42,6 +42,24 @@ definitions:
       - none
       - tls
       - mtls
+  -
+    type: enum
+    name: client-state
+    doc: State of an NFSv4 client record.
+    entries:
+      - unconfirmed
+      - active
+      - courtesy
+      - expirable
+  -
+    type: enum
+    name: callback-state
+    doc: State of an NFSv4 client's callback channel.
+    entries:
+      - up
+      - unknown
+      - down
+      - fault
 
 attribute-sets:
   -
@@ -415,6 +433,63 @@ attribute-sets:
         type: nest
         nested-attributes: server-proc-entry
         multi-attr: true
+  -
+    name: client
+    attributes:
+      -
+        name: clientid
+        type: u64
+        doc: >-
+          Server-generated NFSv4 client ID, with the boot value in the upper
+          32 bits and the per-boot ID in the lower 32 bits.
+      -
+        name: pad
+        type: pad
+      -
+        name: address4
+        type: u32
+        byte-order: big-endian
+        display-hint: ipv4
+        doc: IPv4 peer address recorded when the client was created.
+      -
+        name: address6
+        type: binary
+        byte-order: big-endian
+        display-hint: ipv6
+        checks:
+          exact-len: 16
+        doc: IPv6 peer address recorded when the client was created.
+      -
+        name: address-port
+        type: u16
+        byte-order: big-endian
+        doc: Transport peer port recorded when the client was created.
+      -
+        name: address-scope-id
+        type: u32
+        doc: IPv6 scope ID recorded when the client was created, when nonzero.
+      -
+        name: minor-version
+        type: u32
+        doc: Negotiated NFSv4 minor version.
+      -
+        name: state
+        type: u32
+        enum: client-state
+        doc: Confirmation and courtesy-state status of the client record.
+      -
+        name: lease-remaining
+        type: s64
+        doc: Signed seconds until the client's lease expires; negative means overdue.
+      -
+        name: reclaim-complete
+        type: flag
+        doc: The NFSv4.1 or later client sent RECLAIM_COMPLETE.
+      -
+        name: callback-state
+        type: u32
+        enum: callback-state
+        doc: Health of the client's callback channel.
 
 operations:
   list:
@@ -627,6 +702,24 @@ operations:
             - proc4-ops
             - proc4ops-ops
             - proc4cb-ops
+    -
+      name: client-get
+      doc: dump NFSv4 clients
+      attribute-set: client
+      flags: [admin-perm]
+      dump:
+        reply:
+          attributes:
+            - clientid
+            - address4
+            - address6
+            - address-port
+            - address-scope-id
+            - minor-version
+            - state
+            - lease-remaining
+            - reclaim-complete
+            - callback-state
 
 mcast-groups:
   list:
fs/nfsd/netlink.c +5 -0
index eba8b353f41..48bc499136b 100644
@@ -230,6 +230,11 @@ static const struct genl_split_ops nfsd_nl_ops[] = {
 		.dumpit	= nfsd_nl_server_stats_get_dumpit,
 		.flags	= GENL_CMD_CAP_DUMP,
 	},
+	{
+		.cmd	= NFSD_CMD_CLIENT_GET,
+		.dumpit	= nfsd_nl_client_get_dumpit,
+		.flags	= GENL_ADMIN_PERM | GENL_CMD_CAP_DUMP,
+	},
 };
 
 static const struct genl_multicast_group nfsd_nl_mcgrps[] = {
fs/nfsd/netlink.h +1 -0
index 027e2953db2..de7593e6408 100644
@@ -44,6 +44,7 @@ int nfsd_nl_unlock_filesystem_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_unlock_export_doit(struct sk_buff *skb, struct genl_info *info);
 int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb,
 				    struct netlink_callback *cb);
+int nfsd_nl_client_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
 
 enum {
 	NFSD_NLGRP_NONE,
fs/nfsd/netns.h +1 -0
index 0ce7da20aba..d30af86cf78 100644
@@ -105,6 +105,7 @@ struct nfsd_net {
 	struct list_head *unconf_id_hashtbl;
 	struct rb_root unconf_name_tree;
 	struct list_head *sessionid_hashtbl;
+	u32 nfs4_client_generation;	/* protected by client_lock, never zero */
 	/*
 	 * client_lru holds client queue ordered by nfs4_client.cl_time
 	 * for lease renewal.
fs/nfsd/nfs4ctl.h +10 -0
index bcec4c4ef1d..44cbb0fac58 100644
@@ -20,8 +20,10 @@
 struct net;
 struct inode;
 struct dentry;
+struct sk_buff;
 struct svc_rqst;
 struct nfsd_net;
+struct netlink_callback;
 
 #ifdef CONFIG_NFSD_V4
 extern unsigned long max_delegations;
@@ -37,6 +39,8 @@ bool nfsd4_spo_must_allow(struct svc_rqst *rqstp);
 int nfsd4_create_laundry_wq(void);
 void nfsd4_destroy_laundry_wq(void);
 bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp, struct inode *inode);
+int nfsd4_nl_client_get_dumpit(struct sk_buff *skb,
+			       struct netlink_callback *cb);
 
 extern int nfsd4_is_junction(struct dentry *dentry);
 extern int register_cld_notifier(void);
@@ -68,6 +72,12 @@ static inline bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp,
 	return false;
 }
 
+static inline int nfsd4_nl_client_get_dumpit(struct sk_buff *skb,
+					     struct netlink_callback *cb)
+{
+	return 0;
+}
+
 static inline int nfsd4_is_junction(struct dentry *dentry)
 {
 	return 0;
fs/nfsd/nfs4state.c +216 -0
index ae0af9490fb..f947ac20eed 100644
@@ -59,6 +59,7 @@
 #include "pnfs.h"
 #include "filecache.h"
 #include "nfs4xdr_gen.h"
+#include "netlink.h"
 #include "trace.h"
 
 #define NFSDDBG_FACILITY                NFSDDBG_PROC
@@ -2842,6 +2843,14 @@ free_client(struct nfs4_client *clp)
 	nfsd4_put_client(clp);
 }
 
+static void nfsd4_bump_client_generation(struct nfsd_net *nn)
+{
+	lockdep_assert_held(&nn->client_lock);
+
+	if (++nn->nfs4_client_generation == 0)
+		nn->nfs4_client_generation++;
+}
+
 /* must be called under the client_lock */
 static void
 unhash_client_locked(struct nfs4_client *clp)
@@ -2856,6 +2865,7 @@ unhash_client_locked(struct nfs4_client *clp)
 	/* Make it invisible */
 	if (!list_empty(&clp->cl_idhash)) {
 		list_del_init(&clp->cl_idhash);
+		nfsd4_bump_client_generation(nn);
 		if (test_bit(NFSD4_CLIENT_CONFIRMED, &clp->cl_flags))
 			rb_erase(&clp->cl_namenode, &nn->conf_name_tree);
 		else
@@ -3232,6 +3242,253 @@ static const char *cb_state2str(int state)
 	return "UNDEFINED";
 }
 
+enum nfsd4_nl_client_table {
+	NFSD4_NL_CLIENT_CONFIRMED,
+	NFSD4_NL_CLIENT_UNCONFIRMED,
+	NFSD4_NL_CLIENT_DONE,
+};
+
+struct nfsd4_nl_client {
+	struct sockaddr_storage	address;
+	u64			clientid;
+	s64			lease_remaining;
+	u32			minor_version;
+	u32			state;
+	u32			callback_state;
+	bool			reclaim_complete;
+};
+
+static u32 nfsd4_nl_client_state(bool confirmed, unsigned int state)
+{
+	if (!confirmed)
+		return NFSD_CLIENT_STATE_UNCONFIRMED;
+
+	switch (state) {
+	case NFSD4_COURTESY:
+		return NFSD_CLIENT_STATE_COURTESY;
+	case NFSD4_EXPIRABLE:
+		return NFSD_CLIENT_STATE_EXPIRABLE;
+	default:
+		return NFSD_CLIENT_STATE_ACTIVE;
+	}
+}
+
+static u32 nfsd4_nl_callback_state(int state)
+{
+	switch (state) {
+	case NFSD4_CB_UP:
+		return NFSD_CALLBACK_STATE_UP;
+	case NFSD4_CB_DOWN:
+		return NFSD_CALLBACK_STATE_DOWN;
+	case NFSD4_CB_FAULT:
+		return NFSD_CALLBACK_STATE_FAULT;
+	default:
+		return NFSD_CALLBACK_STATE_UNKNOWN;
+	}
+}
+
+static struct nfs4_client *
+nfsd_nl_find_client(struct nfsd_net *nn, enum nfsd4_nl_client_table table,
+		    unsigned long bucket, unsigned long skip,
+		    struct netlink_callback *cb)
+{
+	struct nfs4_client *clp = NULL;
+	struct nfs4_client *pos;
+	struct list_head *head;
+	unsigned long index = 0;
+
+	lockdep_assert_held(&nfsd_mutex);
+
+	switch (table) {
+	case NFSD4_NL_CLIENT_CONFIRMED:
+		head = &nn->conf_id_hashtbl[bucket];
+		break;
+	case NFSD4_NL_CLIENT_UNCONFIRMED:
+		head = &nn->unconf_id_hashtbl[bucket];
+		break;
+	default:
+		return NULL;
+	}
+
+	spin_lock(&nn->client_lock);
+	cb->seq = nn->nfs4_client_generation;
+	list_for_each_entry(pos, head, cl_idhash) {
+		if (index++ != skip)
+			continue;
+		kref_get(&pos->cl_nfsdfs.cl_ref);
+		clp = pos;
+		break;
+	}
+	spin_unlock(&nn->client_lock);
+	return clp;
+}
+
+static void nfsd4_nl_client_snapshot(struct nfsd_net *nn,
+				     struct nfs4_client *clp,
+				     struct nfsd4_nl_client *client)
+{
+	unsigned int state;
+	time64_t last_renew;
+	bool confirmed;
+
+	spin_lock(&nn->client_lock);
+	last_renew = clp->cl_time;
+	confirmed = test_bit(NFSD4_CLIENT_CONFIRMED, &clp->cl_flags);
+	state = READ_ONCE(clp->cl_state);
+	spin_unlock(&nn->client_lock);
+
+	memcpy(&client->address, &clp->cl_addr, sizeof(client->address));
+	client->clientid = (u64)clp->cl_clientid.cl_boot << 32 |
+			   clp->cl_clientid.cl_id;
+	client->lease_remaining = last_renew ?
+		last_renew + READ_ONCE(nn->nfsd4_lease) -
+		ktime_get_boottime_seconds() : 0;
+	client->minor_version = clp->cl_minorversion;
+	client->state = nfsd4_nl_client_state(confirmed, state);
+	client->callback_state =
+		nfsd4_nl_callback_state(READ_ONCE(clp->cl_cb_state));
+	client->reclaim_complete =
+		test_bit(NFSD4_CLIENT_RECLAIM_COMPLETE, &clp->cl_flags);
+}
+
+static int
+nfsd4_nl_client_marshal_address(struct sk_buff *skb,
+				const struct sockaddr_storage *address)
+{
+	switch (address->ss_family) {
+	case AF_INET: {
+		const struct sockaddr_in *sin =
+			(const struct sockaddr_in *)address;
+
+		if (nla_put_in_addr(skb, NFSD_A_CLIENT_ADDRESS4,
+				    sin->sin_addr.s_addr) ||
+		    nla_put_be16(skb, NFSD_A_CLIENT_ADDRESS_PORT,
+				 sin->sin_port))
+			return -EMSGSIZE;
+		break;
+	}
+	case AF_INET6: {
+		const struct sockaddr_in6 *sin6 =
+			(const struct sockaddr_in6 *)address;
+
+		if (nla_put_in6_addr(skb, NFSD_A_CLIENT_ADDRESS6,
+				     &sin6->sin6_addr) ||
+		    nla_put_be16(skb, NFSD_A_CLIENT_ADDRESS_PORT,
+				 sin6->sin6_port) ||
+		    (sin6->sin6_scope_id &&
+		     nla_put_u32(skb, NFSD_A_CLIENT_ADDRESS_SCOPE_ID,
+				 sin6->sin6_scope_id)))
+			return -EMSGSIZE;
+		break;
+	}
+	}
+	return 0;
+}
+
+static int nfsd4_nl_client_compose_msg(struct sk_buff *skb,
+				       struct netlink_callback *cb,
+				       const struct nfsd4_nl_client *client)
+{
+	void *hdr;
+
+	hdr = genlmsg_put(skb, NETLINK_CB(cb->skb).portid,
+			  cb->nlh->nlmsg_seq, &nfsd_nl_family, NLM_F_MULTI,
+			  NFSD_CMD_CLIENT_GET);
+	if (!hdr)
+		return -EMSGSIZE;
+	genl_dump_check_consistent(cb, hdr);
+
+	if (nla_put_u64_64bit(skb, NFSD_A_CLIENT_CLIENTID, client->clientid,
+			      NFSD_A_CLIENT_PAD) ||
+	    nfsd4_nl_client_marshal_address(skb, &client->address) ||
+	    nla_put_u32(skb, NFSD_A_CLIENT_MINOR_VERSION,
+			client->minor_version) ||
+	    nla_put_u32(skb, NFSD_A_CLIENT_STATE, client->state) ||
+	    nla_put_s64(skb, NFSD_A_CLIENT_LEASE_REMAINING,
+			client->lease_remaining, NFSD_A_CLIENT_PAD) ||
+	    (client->reclaim_complete &&
+	     nla_put_flag(skb, NFSD_A_CLIENT_RECLAIM_COMPLETE)) ||
+	    nla_put_u32(skb, NFSD_A_CLIENT_CALLBACK_STATE,
+			client->callback_state))
+		goto err_cancel;
+
+	genlmsg_end(skb, hdr);
+	return 0;
+
+err_cancel:
+	genlmsg_cancel(skb, hdr);
+	return -EMSGSIZE;
+}
+
+/**
+ * nfsd4_nl_client_get_dumpit - dump NFSv4 client information
+ * @skb: reply buffer
+ * @cb: netlink metadata and command arguments
+ *
+ * One netlink message is emitted for each client. cb->args tracks the client
+ * table, hash bucket, and offset within that bucket. Client table changes can
+ * cause an object to be skipped or repeated between calls; in that case the
+ * affected message or NLMSG_DONE is marked with NLM_F_DUMP_INTR.
+ *
+ * Returns the size of the reply or a negative errno.
+ */
+int nfsd4_nl_client_get_dumpit(struct sk_buff *skb,
+			       struct netlink_callback *cb)
+{
+	struct nfsd4_nl_client client;
+	struct nfs4_client *clp;
+	struct nfsd_net *nn;
+	struct net *net;
+	int ret = 0;
+
+	net = sock_net(skb->sk);
+	nn = net_generic(net, nfsd_net_id);
+	mutex_lock(&nfsd_mutex);
+	if (!test_bit(NFSD_NET_UP, &nn->flags)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}

fs/nfsd/nfs4state.c static const char *cb_state2str(int state) · line 3460

Quoted 10 lines
fs/nfsd/nfs4state.c +10 -0
+
+	/*
+	 * Table values are ordered: confirmed clients first, then
+	 * unconfirmed, then done. cb->args[0] tracks which table the
+	 * dump is visiting, args[1] the hash bucket in it, and args[2]
+	 * the client's offset within that bucket.
+	 */
+	while (cb->args[0] < NFSD4_NL_CLIENT_DONE) {
+		if (cb->args[1] >= CLIENT_HASH_SIZE) {
+			cb->args[0]++;

Maybe I wasn't clear in my earlier email. cb->args[0] is an enum. Why are you incrementing through it like this? It's weird and it makes it unclear what this code is doing.

It's also brittle. If someone expands that enum later, it could be problematic.

Given that there are only two possible tables, there's no reason to make this so clever. Please rework this to be more human-readable.

What I'd do is move all of the stuff inside the while() loop into a helper function, and then just call that twice: once with NFSD4_NL_CLIENT_CONFIRMED and once with NFSD4_NL_CLIENT_UNCONFIRMED.

Quoted 145 lines
fs/nfsd/nfs4state.c +34 -0
+			cb->args[1] = 0;
+			cb->args[2] = 0;
+			continue;
+		}
+
+		clp = nfsd_nl_find_client(nn, cb->args[0], cb->args[1],
+					  cb->args[2], cb);
+		if (!clp) {
+			cb->args[1]++;
+			cb->args[2] = 0;
+			continue;
+		}
+
+		memset(&client, 0, sizeof(client));
+		nfsd4_nl_client_snapshot(nn, clp, &client);
+		ret = nfsd4_nl_client_compose_msg(skb, cb, &client);
+		nfsd4_put_client(clp);
+		if (ret) {
+			if (skb->len)
+				ret = skb->len;
+			goto out_unlock;
+		}
+		cb->args[2]++;
+	}
+	ret = skb->len;
+
+out_unlock:
+	mutex_unlock(&nfsd_mutex);
+	return ret;
+}
+
 static int client_info_show(struct seq_file *m, void *v)
 {
 	struct inode *inode = file_inode(m->file);
@@ -4026,6 +4283,7 @@ add_to_unconfirmed(struct nfs4_client *clp)
 	add_clp_to_name_tree(clp, &nn->unconf_name_tree);
 	idhashval = clientid_hashval(clp->cl_clientid.cl_id);
 	list_add(&clp->cl_idhash, &nn->unconf_id_hashtbl[idhashval]);
+	nfsd4_bump_client_generation(nn);
 	renew_client_locked(clp);
 }
 
@@ -4038,6 +4296,7 @@ move_to_confirmed(struct nfs4_client *clp)
 	lockdep_assert_held(&nn->client_lock);
 
 	list_move(&clp->cl_idhash, &nn->conf_id_hashtbl[idhashval]);
+	nfsd4_bump_client_generation(nn);
 	rb_erase(&clp->cl_namenode, &nn->unconf_name_tree);
 	add_clp_to_name_tree(clp, &nn->conf_name_tree);
 	set_bit(NFSD4_CLIENT_CONFIRMED, &clp->cl_flags);
@@ -10160,6 +10419,7 @@ static int nfs4_state_create_net(struct net *net)
 	INIT_LIST_HEAD(&nn->del_recall_lru);
 	spin_lock_init(&nn->deleg_lock);
 	spin_lock_init(&nn->client_lock);
+	nn->nfs4_client_generation = 1;
 	spin_lock_init(&nn->s2s_cp_lock);
 	idr_init(&nn->s2s_cp_stateids);
 	atomic_set(&nn->pending_async_copies, 0);
fs/nfsd/nfsctl.c +13 -0
index c6f6bc3b128..4d11592cdf8 100644
@@ -1649,6 +1649,19 @@ int nfsd_nl_rpc_status_get_dumpit(struct sk_buff *skb,
 	return ret;
 }
 
+/**
+ * nfsd_nl_client_get_dumpit - dump NFSv4 client information
+ * @skb: reply buffer
+ * @cb: netlink metadata and command arguments
+ *
+ * Returns the size of the reply or a negative errno.
+ */
+int nfsd_nl_client_get_dumpit(struct sk_buff *skb,
+			      struct netlink_callback *cb)
+{
+	return nfsd4_nl_client_get_dumpit(skb, cb);
+}
+
 /**
  * nfsd_nl_fh_key_set - helper to copy fh_key from userspace
  * @attr: nlattr NFSD_A_SERVER_FH_KEY
include/uapi/linux/nfsd_netlink.h +38 -0
index 87da1d0bb21..715acb4104c 100644
@@ -50,6 +50,26 @@ enum nfsd_xprtsec_mode {
 	NFSD_XPRTSEC_MODE_MTLS = 4,
 };
 
+/*
+ * State of an NFSv4 client record.
+ */
+enum nfsd_client_state {
+	NFSD_CLIENT_STATE_UNCONFIRMED,
+	NFSD_CLIENT_STATE_ACTIVE,
+	NFSD_CLIENT_STATE_COURTESY,
+	NFSD_CLIENT_STATE_EXPIRABLE,
+};
+
+/*
+ * State of an NFSv4 client's callback channel.
+ */
+enum nfsd_callback_state {
+	NFSD_CALLBACK_STATE_UP,
+	NFSD_CALLBACK_STATE_UNKNOWN,
+	NFSD_CALLBACK_STATE_DOWN,
+	NFSD_CALLBACK_STATE_FAULT,
+};
+
 enum {
 	NFSD_A_CACHE_NOTIFY_CACHE_TYPE = 1,
 
@@ -260,6 +280,23 @@ enum {
 	NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX - 1)
 };
 
+enum {
+	NFSD_A_CLIENT_CLIENTID = 1,
+	NFSD_A_CLIENT_PAD,
+	NFSD_A_CLIENT_ADDRESS4,
+	NFSD_A_CLIENT_ADDRESS6,
+	NFSD_A_CLIENT_ADDRESS_PORT,
+	NFSD_A_CLIENT_ADDRESS_SCOPE_ID,
+	NFSD_A_CLIENT_MINOR_VERSION,
+	NFSD_A_CLIENT_STATE,
+	NFSD_A_CLIENT_LEASE_REMAINING,
+	NFSD_A_CLIENT_RECLAIM_COMPLETE,
+	NFSD_A_CLIENT_CALLBACK_STATE,
+
+	__NFSD_A_CLIENT_MAX,
+	NFSD_A_CLIENT_MAX = (__NFSD_A_CLIENT_MAX - 1)
+};
+
 enum {
 	NFSD_CMD_RPC_STATUS_GET = 1,
 	NFSD_CMD_THREADS_SET,
@@ -280,6 +317,7 @@ enum {
 	NFSD_CMD_UNLOCK_FILESYSTEM,
 	NFSD_CMD_UNLOCK_EXPORT,
 	NFSD_CMD_SERVER_STATS_GET,
+	NFSD_CMD_CLIENT_GET,
 
 	__NFSD_CMD_MAX,
 	NFSD_CMD_MAX = (__NFSD_CMD_MAX - 1)
Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw

PATCH v3 2/2 nfsd: report per-client NFSv4 state usage through Netlink

In reply to Prabhakar Pujeri

A client can retain substantial server-side state even when its lease and callback channel look healthy. Monitoring tools need compact per-client totals to identify which client owns that state before doing a detailed inspection through the nfsd filesystem.

Extend client-get replies with separate counts for sessions and open, lock, delegation, and layout stateid records. A lock stateid represents a lock-owner/file pair and can cover multiple byte-range locks, so document that these are record counts rather than protocol operation or byte-range counts.

Maintain u64 counters under cl_lock at the existing session and stateid publish and removal points. The dump takes an O(1) snapshot instead of walking an unbounded per-client IDR while holding the spinlock. Stateid counters are decremented only at final IDR removal, preserving the existing record-count semantics for retained stateids.

Encode the counters as variable-width Netlink uint attributes, which use four bytes for values that fit in u32 and eight bytes otherwise.

---
Changes since v2:

- rebase onto nfsd-testing at ac579868af0c; no functional change

Changes since v1:

- maintain u64 counters at stateid creation/removal points instead of walking per-client stateid IDRs under cl_lock
- count sessions and open/lock/delegation/layout stateid records separately
- encode counts as variable-width Netlink uint attributes

v1: https://lore.kernel.org/r/245a1ee42d054e280629c9640c8d8225b45639d0.1787638668.git.prabhakar.pujeri@dell.com
v2: https://lore.kernel.org/r/f028a8839133f52e3ff04d5e4645308e63ce8dc6.1787988917.git.prabhakar.pujeri@dell.com

 .../admin-guide/nfs/nfsd-admin-interfaces.rst |  6 ++
 Documentation/netlink/specs/nfsd.yaml         | 27 +++++-
 fs/nfsd/nfs4layouts.c                         |  2 +-
 fs/nfsd/nfs4state.c                           | 95 +++++++++++++++++--
 fs/nfsd/state.h                               |  7 ++
 include/uapi/linux/nfsd_netlink.h             |  5 +
 6 files changed, 130 insertions(+), 12 deletions(-)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +6 -0
index de2a5402587..19914192315 100644
@@ -38,6 +38,12 @@ by its server-generated client ID and transport address, then reports its
 minor version, client and callback states, signed lease time remaining, and
 whether an NFSv4.1 or later client sent RECLAIM_COMPLETE.
 
+The dump also reports separate counts for sessions and open, lock, delegation,
+and layout stateids.  A lock stateid represents state for one lock owner and
+file, not necessarily one byte-range lock.  NFSD maintains these counters as
+state changes, so reporting a client does not walk its session or stateid
+tables.
+
 Clients can change between messages.  If that can make the dump skip or repeat
 a record, the kernel sets ``NLM_F_DUMP_INTR`` and userspace should retry.
 
Documentation/netlink/specs/nfsd.yaml +26 -1
index 9207a96fe59..8fa1d692551 100644
@@ -490,6 +490,26 @@ attribute-sets:
         type: u32
         enum: callback-state
         doc: Health of the client's callback channel.
+      -
+        name: sessions
+        type: uint
+        doc: Number of NFSv4.1 or later sessions owned by the client.
+      -
+        name: open-stateids
+        type: uint
+        doc: Number of open stateid records owned by the client.
+      -
+        name: lock-stateids
+        type: uint
+        doc: Number of lock-owner/file stateid records, not byte-range locks.
+      -
+        name: delegation-stateids
+        type: uint
+        doc: Number of delegation stateid records owned by the client.
+      -
+        name: layout-stateids
+        type: uint
+        doc: Number of pNFS layout stateid records owned by the client.
 
 operations:
   list:
@@ -704,7 +724,7 @@ operations:
             - proc4cb-ops
     -
       name: client-get
-      doc: dump NFSv4 clients
+      doc: dump NFSv4 clients and their current state usage
       attribute-set: client
       flags: [admin-perm]
       dump:
@@ -720,6 +740,11 @@ operations:
             - lease-remaining
             - reclaim-complete
             - callback-state
+            - sessions
+            - open-stateids
+            - lock-stateids
+            - delegation-stateids
+            - layout-stateids
 
 mcast-groups:
   list:
fs/nfsd/nfs4layouts.c +1 -1
index 4187202f9ac..adb9f33ef7a 100644
@@ -272,7 +272,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,
 	}
 
 	spin_lock(&clp->cl_lock);
-	stp->sc_type = SC_TYPE_LAYOUT;
+	nfs4_set_stid_type_locked(stp, SC_TYPE_LAYOUT);
 	list_add(&ls->ls_perclnt, &clp->cl_lo_states);
 	spin_unlock(&clp->cl_lock);
 
fs/nfsd/nfs4state.c +85 -10

178 lines not on this page · raw message

fs/nfsd/state.h +7 -0
index cd9294f024b..3fe130ce5de 100644
@@ -610,6 +610,12 @@ struct nfs4_client {
 
 	/* for nfs41 */
 	struct list_head	cl_sessions;
+	/* State usage counters, protected by cl_lock. */
+	u64			cl_session_count;
+	u64			cl_open_stateid_count;
+	u64			cl_lock_stateid_count;
+	u64			cl_delegation_stateid_count;
+	u64			cl_layout_stateid_count;
 	struct nfsd4_clid_slot	cl_cs_slot;	/* create_session slot */
 	u32			cl_exchange_flags;
 	/* number of rpc's in progress over an associated session: */
@@ -923,6 +929,7 @@ __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate,
 			    struct nfs4_stid **s, struct nfsd_net *nn);
 struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *slab,
 				  void (*sc_free)(struct nfs4_stid *));
+void nfs4_set_stid_type_locked(struct nfs4_stid *stid, unsigned short type);
 struct nfsd4_async_copy *nfs4_alloc_copy_stid(struct nfs4_client *clp);
 struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state(struct nfsd_net *nn,
 			struct nfs4_stid *p_stid);
include/uapi/linux/nfsd_netlink.h +5 -0
index 715acb4104c..d97783335ac 100644
@@ -292,6 +292,11 @@ enum {
 	NFSD_A_CLIENT_LEASE_REMAINING,
 	NFSD_A_CLIENT_RECLAIM_COMPLETE,
 	NFSD_A_CLIENT_CALLBACK_STATE,
+	NFSD_A_CLIENT_SESSIONS,
+	NFSD_A_CLIENT_OPEN_STATEIDS,
+	NFSD_A_CLIENT_LOCK_STATEIDS,
+	NFSD_A_CLIENT_DELEGATION_STATEIDS,
+	NFSD_A_CLIENT_LAYOUT_STATEIDS,
 
 	__NFSD_A_CLIENT_MAX,
 	NFSD_A_CLIENT_MAX = (__NFSD_A_CLIENT_MAX - 1)
Signature · 1 line
2.55.0