nfsd: expose NFSv4 client state through Netlink
Subsystems (5)
cover nfsd: expose NFSv4 client state through Netlink
Administrators can inspect NFSv4 clients through the nfsd filesystem, but collecting a server-wide view requires walking a directory and opening multiple files for every client. That is useful for detailed inspection, but awkward for monitoring and automation.
Add a compact query interface to the existing nfsd Generic Netlink family:
1. Extend threads-get with an explicit NFSv4 recovery-grace value.
2. Add a privileged client-get dump with basic identity, lease, confirmation, reclaim, and callback information.
3. Extend each client reply with session and stateid record counts.
The split keeps the grace query independent, establishes the minimum useful client ABI before adding the more expensive state traversal, and permits older user space to ignore the count extension. The existing nfsd filesystem files and their output remain unchanged.
client-get emits one bounded message per client. The dump pins each client, uses the existing client and state locks for mutable fields, and serializes against server shutdown. Client churn can still make a multi-message dump a best-effort snapshot; the interface documentation states this explicitly.
The Netlink client ID places cl_boot in the upper 32 bits and cl_id in the lower 32 bits. This is an architecture-independent representation matching NFSD's logical and tracepoint representation, rather than the legacy procfs rendering of the in-memory bytes.
A companion nfs-utils series adds nfsdctl clients and displays the grace value from nfsdctl threads.
Validation performed:
- strict checkpatch on each patch: 0 errors, 0 warnings, 0 checks
- YNL regeneration and headers_install
- scoped NFS admin and Netlink documentation builds with SPHINXOPTS=-W
- W=1 NFSD builds with CONFIG_NFSD_V4=y and CONFIG_NFSD_V4=n
- two-vCPU QEMU test with a live NFSv4.2 loopback mount
- grace yes/no transition checked through both raw Netlink and nfsdctl
- one live client with one session, open stateid, lock stateid, and delegation stateid; zero layout stateids
- legacy /proc/fs/nfsd/clients output checked for unchanged fields
The series is based on the nfsd-testing branch. The exact base is recorded in the base-commit trailer below.
Prabhakar Pujeri (3):
nfsd: report NFSv4 grace state through Netlink
nfsd: add a Netlink dump of NFSv4 clients
nfsd: report per-client NFSv4 state usage through Netlink
.../admin-guide/nfs/nfsd-admin-interfaces.rst | 22 ++ Documentation/netlink/specs/nfsd.yaml | 106 ++++++++ fs/nfsd/netlink.c | 5 + fs/nfsd/netlink.h | 1 + fs/nfsd/nfs4ctl.h | 10 + fs/nfsd/nfs4state.c | 238 +++++++++++++++++- fs/nfsd/nfsctl.c | 22 +- include/uapi/linux/nfsd_netlink.h | 41 +++ 8 files changed, 443 insertions(+), 2 deletions(-)
base-commit: 46ff234c7129c7c10ed493413d4f60d0cf4dd1f1
Signature · 1 line
2.54.0
PATCH 1/3 nfsd: report NFSv4 grace state through Netlink
Operational tooling needs to distinguish a server that is accepting normal NFSv4 state operations from one that is still accepting only recovery requests. The existing Netlink threads query reports server-wide configuration, but recovery state is available only through the nfsd filesystem.
Add an in-grace value to the threads-get reply. Return it in every reply so user space can distinguish a server that is not in grace from an older kernel that does not provide the attribute.
--- Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst | 3 +++ Documentation/netlink/specs/nfsd.yaml | 6 ++++++ fs/nfsd/nfsctl.c | 9 ++++++++- include/uapi/linux/nfsd_netlink.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +3 -0
@@ -25,6 +25,9 @@ udp and one tcp listener at port 2049 (see nfsd_init_socks). On startup, nfsd and lockd grace periods start. nfsd is shut down by a write of 0 to nfsd/threads. All locks and state are thrown away at that point. +The ``in-grace`` value in the ``threads-get`` Generic Netlink reply reports +whether the NFSv4 server's recovery grace period is still active. + Between startup and shutdown, the number of threads may be adjusted up or down by additional writes to nfsd/threads or by writes to nfsd/pool_threads.
Documentation/netlink/specs/nfsd.yaml +6 -0
@@ -130,6 +130,11 @@ attribute-sets: type: binary checks: exact-len: 16 + - + name: in-grace + type: u8 + doc: One while the NFSv4 server is in its recovery grace period, + otherwise zero. - name: version attributes:
@@ -464,6 +469,7 @@ operations: - leasetime - scope - min-threads + - in-grace - name: version-set doc: set nfs enabled versions
fs/nfsd/nfsctl.c +8 -1
@@ -1775,6 +1775,7 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info) {struct net *net =genl_info_net (info);struct nfsd_net *nn =net_generic (net, nfsd_net_id); +bool in_grace = false;void *hdr;int err;
@@ -1790,6 +1791,11 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info)mutex_lock (&nfsd_mutex); +#ifdef CONFIG_NFSD_V4 + in_grace =nfsd_vers (nn,4 ,NFSD_TEST ) && +test_bit (NFSD_NET_UP , &nn->flags) && + !test_bit (NFSD_NET_GRACE_ENDED , &nn->flags); +#endif err =nla_put_u32 (skb,NFSD_A_SERVER_GRACETIME , nn->nfsd4_grace) || nla_put_u32(skb, NFSD_A_SERVER_LEASETIME,
@@ -1797,7 +1803,8 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info)nla_put_u32 (skb,NFSD_A_SERVER_MIN_THREADS , nn->min_threads) ||nla_put_string (skb,NFSD_A_SERVER_SCOPE , - nn->nfsd_name); + nn->nfsd_name) || +nla_put_u8 (skb,NFSD_A_SERVER_IN_GRACE , in_grace);if (err) goto err_unlock;
include/uapi/linux/nfsd_netlink.h +1 -0
@@ -84,6 +84,7 @@ enum {NFSD_A_SERVER_SCOPE ,NFSD_A_SERVER_MIN_THREADS ,NFSD_A_SERVER_FH_KEY , +NFSD_A_SERVER_IN_GRACE , __NFSD_A_SERVER_MAX,NFSD_A_SERVER_MAX = (__NFSD_A_SERVER_MAX -1 )
Signature · 1 line
2.54.0
PATCH 2/3 nfsd: add a Netlink dump of NFSv4 clients
In reply to Prabhakar PujeriAdministrators currently have to walk one nfsd filesystem directory per client to correlate basic NFSv4 identity, lease, and callback information. That interface is useful for detailed inspection, but it is awkward for monitoring tools and provides no atomic way to enumerate the client set.
Add a privileged client-get dump to the nfsd Generic Netlink family. Emit one bounded message per confirmed or unconfirmed client with its server-generated client ID, peer address, minor version, client and callback states, signed lease time remaining, and RECLAIM_COMPLETE status.
Pin each client while taking a snapshot, protect mutable lease and confirmation fields with the per-net client lock, and serialize against server shutdown with nfsd_mutex. Netlink dump cursors track the table, bucket, and position; client churn can still cause a best-effort snapshot, which is documented. Leave the existing nfsd filesystem interface unchanged.
--- .../admin-guide/nfs/nfsd-admin-interfaces.rst | 15 ++ Documentation/netlink/specs/nfsd.yaml | 75 +++++++ fs/nfsd/netlink.c | 5 + fs/nfsd/netlink.h | 1 + fs/nfsd/nfs4ctl.h | 10 + fs/nfsd/nfs4state.c | 200 ++++++++++++++++++ fs/nfsd/nfsctl.c | 13 ++ include/uapi/linux/nfsd_netlink.h | 35 +++ 8 files changed, 354 insertions(+)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +15 -0
@@ -32,6 +32,21 @@ Between startup and shutdown, the number of threads may be adjusted up or down by additional writes to nfsd/threads or by writes to nfsd/pool_threads. +NFSv4 client visibility +======================= + +The privileged ``client-get`` dump in the ``nfsd`` Generic Netlink family +emits one message for each NFSv4 client. Each message identifies the client +by its server-generated client ID and transport address, then reports its +minor version, client and callback states, signed lease time remaining, and +whether an NFSv4.1 or later client sent RECLAIM_COMPLETE. + +The dump is a best-effort snapshot; clients can change between messages. + +The existing ``/proc/fs/nfsd/clients/`` files remain available for inspection. +The ``states`` file contains individual stateids, and writing ``expire`` to +``ctl`` forcibly removes the client and all state it owns. + For more detail about files under nfsd/ and what they control, see fs/nfsd/nfsctl.c; most of them have detailed comments.
Documentation/netlink/specs/nfsd.yaml +75 -0
93 lines not on this page · raw message
fs/nfsd/netlink.c +5 -0
@@ -230,6 +230,11 @@ static const struct genl_split_ops nfsd_nl_ops[] = { .dumpit = nfsd_nl_server_stats_get_dumpit, .flags = GENL_CMD_CAP_DUMP, }, + { + .cmd = NFSD_CMD_CLIENT_GET, + .dumpit = nfsd_nl_client_get_dumpit, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DUMP, + }, };static const struct genl_multicast_group nfsd_nl_mcgrps[] = {
fs/nfsd/netlink.h +1 -0
@@ -44,6 +44,7 @@ int nfsd_nl_unlock_filesystem_doit(struct sk_buff *skb, struct genl_info *info);int nfsd_nl_unlock_export_doit (struct sk_buff *skb,struct genl_info *info);int nfsd_nl_server_stats_get_dumpit (struct sk_buff *skb,struct netlink_callback *cb); +int nfsd_nl_client_get_dumpit (struct sk_buff *skb,struct netlink_callback *cb);enum {NFSD_NLGRP_NONE ,
fs/nfsd/nfs4ctl.h +10 -0
@@ -20,8 +20,10 @@struct net ;struct inode ;struct dentry ; +struct sk_buff ;struct svc_rqst ;struct nfsd_net ; +struct netlink_callback ;#ifdef CONFIG_NFSD_V4 extern unsigned long max_delegations;
@@ -37,6 +39,8 @@ bool nfsd4_spo_must_allow(struct svc_rqst *rqstp);int nfsd4_create_laundry_wq (void );void nfsd4_destroy_laundry_wq (void );bool nfsd_wait_for_delegreturn (struct svc_rqst *rqstp,struct inode *inode); +int nfsd4_nl_client_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb);extern int nfsd4_is_junction (struct dentry *dentry);extern int register_cld_notifier (void );
@@ -68,6 +72,12 @@ static inline bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp,return false; } +static inline int nfsd4_nl_client_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb) +{ +return 0 ; +} +static inline int nfsd4_is_junction (struct dentry *dentry) {return 0 ;
fs/nfsd/nfs4state.c +200 -0
212 lines not on this page · raw message
fs/nfsd/nfsctl.c +13 -0
@@ -1634,6 +1634,19 @@ int nfsd_nl_rpc_status_get_dumpit(struct sk_buff *skb,return ret; } +/** +* nfsd_nl_client_get_dumpit - dump NFSv4 client information +* @skb: reply buffer +* @cb: netlink metadata and command arguments +* +* Returns the size of the reply or a negative errno. +*/ +int nfsd_nl_client_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb) +{ +return nfsd4_nl_client_get_dumpit (skb, cb); +} +/** * nfsd_nl_fh_key_set - helper to copy fh_key from userspace * @attr: nlattr NFSD_A_SERVER_FH_KEY
include/uapi/linux/nfsd_netlink.h +35 -0
@@ -50,6 +50,26 @@ enum nfsd_xprtsec_mode { NFSD_XPRTSEC_MODE_MTLS = 4, }; +/* +* State of an NFSv4 client record. +*/ +enum nfsd_client_state { +NFSD_CLIENT_STATE_UNCONFIRMED , +NFSD_CLIENT_STATE_ACTIVE , +NFSD_CLIENT_STATE_COURTESY , +NFSD_CLIENT_STATE_EXPIRABLE , +}; + +/* +* State of an NFSv4 client's callback channel. +*/ +enum nfsd_callback_state { +NFSD_CALLBACK_STATE_UP , +NFSD_CALLBACK_STATE_UNKNOWN , +NFSD_CALLBACK_STATE_DOWN , +NFSD_CALLBACK_STATE_FAULT , +}; +enum {NFSD_A_CACHE_NOTIFY_CACHE_TYPE =1 ,
@@ -261,6 +281,20 @@ enum {NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX -1 ) }; +enum { +NFSD_A_CLIENT_CLIENTID =1 , +NFSD_A_CLIENT_PAD , +NFSD_A_CLIENT_ADDRESS , +NFSD_A_CLIENT_MINOR_VERSION , +NFSD_A_CLIENT_STATE , +NFSD_A_CLIENT_LEASE_REMAINING , +NFSD_A_CLIENT_RECLAIM_COMPLETE , +NFSD_A_CLIENT_CALLBACK_STATE , + + __NFSD_A_CLIENT_MAX, +NFSD_A_CLIENT_MAX = (__NFSD_A_CLIENT_MAX -1 ) +}; +enum {NFSD_CMD_RPC_STATUS_GET =1 ,NFSD_CMD_THREADS_SET ,
@@ -281,6 +315,7 @@ enum {NFSD_CMD_UNLOCK_FILESYSTEM ,NFSD_CMD_UNLOCK_EXPORT ,NFSD_CMD_SERVER_STATS_GET , +NFSD_CMD_CLIENT_GET , __NFSD_CMD_MAX,NFSD_CMD_MAX = (__NFSD_CMD_MAX -1 )
Signature · 1 line
2.54.0
PATCH 3/3 nfsd: report per-client NFSv4 state usage through Netlink
In reply to Prabhakar PujeriA client can retain substantial server-side state even when its lease and callback channel look healthy. Monitoring tools need compact per-client totals to identify which client owns that state before doing a detailed inspection through the nfsd filesystem.
Extend client-get replies with separate counts for sessions and open, lock, delegation, and layout stateid records. A lock stateid represents a lock-owner/file pair and can cover multiple byte-range locks, so document that these are record counts rather than protocol operation or byte-range counts.
Walk both the session list and stateid IDR under cl_lock. free_client() already detaches its session list under that lock, so a client pinned by the dump cannot race this traversal with session teardown.
--- .../admin-guide/nfs/nfsd-admin-interfaces.rst | 4 ++ Documentation/netlink/specs/nfsd.yaml | 27 ++++++++++++- fs/nfsd/nfs4state.c | 40 ++++++++++++++++++- include/uapi/linux/nfsd_netlink.h | 5 +++ 4 files changed, 73 insertions(+), 3 deletions(-)
Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +4 -0
@@ -41,6 +41,10 @@ by its server-generated client ID and transport address, then reports its minor version, client and callback states, signed lease time remaining, and whether an NFSv4.1 or later client sent RECLAIM_COMPLETE. +The dump also reports separate counts for sessions and open, lock, delegation, +and layout stateids. A lock stateid represents state for one lock owner and +file, not necessarily one byte-range lock. + The dump is a best-effort snapshot; clients can change between messages. The existing ``/proc/fs/nfsd/clients/`` files remain available for inspection.
Documentation/netlink/specs/nfsd.yaml +26 -1
@@ -480,6 +480,26 @@ attribute-sets: type: u32 enum: callback-state doc: Health of the client's callback channel. + - + name: sessions + type: u32 + doc: Number of NFSv4.1 or later sessions owned by the client. + - + name: open-stateids + type: u32 + doc: Number of open stateid records owned by the client. + - + name: lock-stateids + type: u32 + doc: Number of lock-owner/file stateid records, not byte-range locks. + - + name: delegation-stateids + type: u32 + doc: Number of delegation stateid records owned by the client. + - + name: layout-stateids + type: u32 + doc: Number of pNFS layout stateid records owned by the client. operations: list:
@@ -695,7 +715,7 @@ operations: - proc4cb-ops - name: client-get - doc: dump NFSv4 clients + doc: dump NFSv4 clients and their current state usage attribute-set: client flags: [admin-perm] dump:
@@ -708,6 +728,11 @@ operations: - lease-remaining - reclaim-complete - callback-state + - sessions + - open-stateids + - lock-stateids + - delegation-stateids + - layout-stateids mcast-groups: list:
fs/nfsd/nfs4state.c +38 -2
@@ -2822,7 +2822,7 @@ free_client(struct nfs4_client *clp) {LIST_HEAD (reaplist); -/* client_info_show() walks cl_sessions under cl_lock */ +/* Client reporting walks cl_sessions under cl_lock. */ spin_lock (&clp->cl_lock);list_splice_init (&clp->cl_sessions, &reaplist);spin_unlock (&clp->cl_lock);
@@ -3246,6 +3246,11 @@ struct nfsd4_nl_client {u32 minor_version;u32 state;u32 callback_state; +u32 sessions; +u32 open_stateids; +u32 lock_stateids; +u32 delegation_stateids; +u32 layout_stateids;bool reclaim_complete; };
@@ -3310,6 +3315,8 @@ static void nfsd4_nl_client_snapshot(struct nfsd_net *nn,struct nfs4_client *clp, struct nfsd4_nl_client *client) { + struct nfs4_stid *stid; +unsigned long id, tmp;unsigned int state;time64_t last_renew;bool confirmed;
@@ -3332,6 +3339,26 @@ static void nfsd4_nl_client_snapshot(struct nfsd_net *nn,nfsd4_nl_callback_state (READ_ONCE (clp->cl_cb_state)); client->reclaim_complete =test_bit (NFSD4_CLIENT_RECLAIM_COMPLETE , &clp->cl_flags); + +spin_lock (&clp->cl_lock); + client->sessions =list_count_nodes (&clp->cl_sessions); +idr_for_each_entry_ul (&clp->cl_stateids, stid, tmp, id) { +switch (stid->sc_type) { +case SC_TYPE_OPEN : + client->open_stateids++; +break ; +case SC_TYPE_LOCK : + client->lock_stateids++; +break ; +case SC_TYPE_DELEG : + client->delegation_stateids++; +break ; +case SC_TYPE_LAYOUT : + client->layout_stateids++; +break ; + } + } +spin_unlock (&clp->cl_lock); } static int nfsd4_nl_client_compose_msg(struct sk_buff *skb,
@@ -3359,7 +3386,16 @@ static int nfsd4_nl_client_compose_msg(struct sk_buff *skb, (client->reclaim_complete &&nla_put_flag (skb,NFSD_A_CLIENT_RECLAIM_COMPLETE )) ||nla_put_u32 (skb,NFSD_A_CLIENT_CALLBACK_STATE , - client->callback_state)) + client->callback_state) || +nla_put_u32 (skb,NFSD_A_CLIENT_SESSIONS , client->sessions) || +nla_put_u32 (skb,NFSD_A_CLIENT_OPEN_STATEIDS , + client->open_stateids) || +nla_put_u32 (skb,NFSD_A_CLIENT_LOCK_STATEIDS , + client->lock_stateids) || +nla_put_u32 (skb,NFSD_A_CLIENT_DELEGATION_STATEIDS , + client->delegation_stateids) || +nla_put_u32 (skb,NFSD_A_CLIENT_LAYOUT_STATEIDS , + client->layout_stateids)) goto err_cancel;genlmsg_end (skb, hdr);
include/uapi/linux/nfsd_netlink.h +5 -0
@@ -290,6 +290,11 @@ enum {NFSD_A_CLIENT_LEASE_REMAINING ,NFSD_A_CLIENT_RECLAIM_COMPLETE ,NFSD_A_CLIENT_CALLBACK_STATE , +NFSD_A_CLIENT_SESSIONS , +NFSD_A_CLIENT_OPEN_STATEIDS , +NFSD_A_CLIENT_LOCK_STATEIDS , +NFSD_A_CLIENT_DELEGATION_STATEIDS , +NFSD_A_CLIENT_LAYOUT_STATEIDS , __NFSD_A_CLIENT_MAX,NFSD_A_CLIENT_MAX = (__NFSD_A_CLIENT_MAX -1 )
Signature · 1 line
2.54.0
Jeff Layton
maintainerRe: [PATCH v1 1/3] nfsd: report NFSv4 grace state through Netlink
In reply to Prabhakar PujeriQuoted 69 lines · Prabhakar Pujeri
Operational tooling needs to distinguish a server that is accepting normal NFSv4 state operations from one that is still accepting only recovery requests. The existing Netlink threads query reports server-wide configuration, but recovery state is available only through the nfsd filesystem.
Add an in-grace value to the threads-get reply. Return it in every reply so user space can distinguish a server that is not in grace from an older kernel that does not provide the attribute.
Signed-off-by Prabhakar Pujeri <[email protected]>--- Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst | 3 +++ Documentation/netlink/specs/nfsd.yaml | 6 ++++++ fs/nfsd/nfsctl.c | 9 ++++++++- include/uapi/linux/nfsd_netlink.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-)Documentation/admin-guide/nfs/nfsd-admin-interfaces.rst +3
-0@@ -25,6 +25,9 @@ udp and one tcp listener at port 2049 (see nfsd_init_socks). On startup, nfsd and lockd grace periods start. nfsd is shut down by a write of 0 to nfsd/threads. All locks and state are thrown away at that point. +The ``in-grace`` value in the ``threads-get`` Generic Netlink reply reports +whether the NFSv4 server's recovery grace period is still active. + Between startup and shutdown, the number of threads may be adjusted up or down by additional writes to nfsd/threads or by writes to nfsd/pool_threads.Documentation/netlink/specs/nfsd.yaml +6
-0@@ -130,6 +130,11 @@ attribute-sets: type: binary checks: exact-len: 16 + - + name: in-grace + type: u8 + doc: One while the NFSv4 server is in its recovery grace period, + otherwise zero. - name: version attributes:@@ -464,6 +469,7 @@ operations: - leasetime - scope - min-threads + - in-grace - name: version-set doc: set nfs enabled versionsfs/nfsd/nfsctl.c +1
-0@@ -1775,6 +1775,7 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info) {struct net *net =genl_info_net (info);struct nfsd_net *nn =net_generic (net, nfsd_net_id); +bool in_grace = false;void *hdr;int err;
fs/nfsd/nfsctl.c int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info) · line 1798
Quoted 8 lines
fs/nfsd/nfsctl.c +5
-0@@ -1790,6 +1791,11 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info)mutex_lock (&nfsd_mutex); +#ifdef CONFIG_NFSD_V4 + in_grace =nfsd_vers (nn,4 ,NFSD_TEST ) && +test_bit (NFSD_NET_UP , &nn->flags) && + !test_bit (NFSD_NET_GRACE_ENDED , &nn->flags); +#endif
What's your interest in reporting this info?
Just because the v4 recovery is done does not mean that you will be able to access files. The overall system grace period is a union of both the NLM and v4 grace periods.
This patch is reporting just the v4 one. If the goal is for the admin to reliably tell when the grace period has been lifted, then this won't tell us that.
Maybe this should just return the result of locks_in_grace() instead?
Or, do you plan to add similar functionality to lockd? Then you can just do the union in userland to figure out if it has been lifted.
Quoted 25 lines
fs/nfsd/nfsctl.c +2
-1err =nla_put_u32 (skb,NFSD_A_SERVER_GRACETIME , nn->nfsd4_grace) || nla_put_u32(skb, NFSD_A_SERVER_LEASETIME,@@ -1797,7 +1803,8 @@ int nfsd_nl_threads_get_doit(struct sk_buff *skb, struct genl_info *info)nla_put_u32 (skb,NFSD_A_SERVER_MIN_THREADS , nn->min_threads) ||nla_put_string (skb,NFSD_A_SERVER_SCOPE , - nn->nfsd_name); + nn->nfsd_name) || +nla_put_u8 (skb,NFSD_A_SERVER_IN_GRACE , in_grace);if (err) goto err_unlock;include/uapi/linux/nfsd_netlink.h +1
-0@@ -84,6 +84,7 @@ enum {NFSD_A_SERVER_SCOPE ,NFSD_A_SERVER_MIN_THREADS ,NFSD_A_SERVER_FH_KEY , +NFSD_A_SERVER_IN_GRACE , __NFSD_A_SERVER_MAX,NFSD_A_SERVER_MAX = (__NFSD_A_SERVER_MAX -1 )
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH v1 1/3] nfsd: report NFSv4 grace state through Netlink
Quoted 1 line · Jeff Layton
What's your interest in reporting this info?
My interest is in making nfsd's operational state observable to monitoring and validation tooling without parsing nfsdfs.
Concretely, I'm building a validation harness for NFS server restart and failover behaviour: bounce the server, drive reclaim from a set of clients, and check that state comes back the way it should. That kind of test has to know when the grace period has actually lifted before it can move on to the next phase. Polling /proc/fs/nfsd/v4_end_grace works, but it is a text interface, it only answers for nfsd's v4 lock manager, and new nfsd control and observability interfaces are going to the netlink family instead.
So I would rather frame this as parity than as new information: grace state is already exported, this puts it on the interface that is replacing the one exporting it, and -- per your point below -- fixes the semantics on the way over.
Quoted 7 lines
Just because the v4 recovery is done does not mean that you will be able to access files. The overall system grace period is a union of both the NLM and v4 grace periods.
This patch is reporting just the v4 one. If the goal is for the admin to reliably tell when the grace period has been lifted, then this won't tell us that.
Agreed, and "has the grace period lifted" is the question I actually care about. I mirrored the semantics of /proc/fs/nfsd/v4_end_grace without stopping to ask whether those were the useful semantics to expose in a new interface. They aren't.
Quoted 4 lines
Maybe this should just return the result of
locks_in_grace()instead?Or, do you plan to add similar functionality to lockd? Then you can just do the union in userland to figure out if it has been lifted.
I don't think lockd needs to grow anything -- fs/nfs_common/grace.c already gives us both halves, and nfsd calls into both helpers today:
- locks_in_grace() is the union you describe: true while any lock manager in the net ns is still in grace.
- opens_in_grace() is true only while a manager with ->block_opens is in grace, which per the comment on struct lock_manager is the NFSv4-and-up case; NLM doesn't set it.
So for v2 I'll report locks_in_grace(), which answers "can this server hand out normal locks and opens yet" rather than "is nfsd's v4 lock manager done". If it's ever worth distinguishing the two, a second attribute carrying opens_in_grace() gets there without touching lockd, and userland can derive the NLM-only case as (locks && !opens). I'd rather not add that speculatively, but say the word if you'd like both in v2.
Two things fall out of the switch, both improvements:
- The #ifdef CONFIG_NFSD_V4 and the open-coded nfsd_vers()/NFSD_NET_UP/NFSD_NET_GRACE_ENDED test all go away. NFSD selects LOCKD which selects GRACE_PERIOD, so the helper is always available, and a lock manager that isn't running simply isn't on the grace list.
- It stops being a v4-specific answer that just happens to be served out of the nfsd netlink family.
I'll rename the attribute and fix the yaml and rst wording so the union semantics are explicit rather than implied.
One design question while I'm respinning. threads-get otherwise reports configuration, and this is dynamic state, so grace could equally live in its own operation. My inclination is to keep just this boolean on threads-get for now, since that is the direct nfsdfs parity, and to grow any richer grace/recovery detail -- elapsed time, expected vs completed reclaims, whether grace was extended or force-ended -- as a separate operation later rather than widening threads-get. But if you'd rather grace not ride on threads-get at all and start that separate op now, I'm glad to go that way instead.
Thanks for the review.
--
Prabhakar
Chuck Lever
maintainerRe: [PATCH v1 1/3] nfsd: report NFSv4 grace state through Netlink
Quoted 48 lines · Prabhakar Pujeri
Quoted 1 line · Jeff Layton
What's your interest in reporting this info?
My interest is in making nfsd's operational state observable to monitoring and validation tooling without parsing nfsdfs.
Concretely, I'm building a validation harness for NFS server restart and failover behaviour: bounce the server, drive reclaim from a set of clients, and check that state comes back the way it should. That kind of test has to know when the grace period has actually lifted before it can move on to the next phase. Polling
/proc/fs/nfsd/v4_end_graceworks, but it is a text interface, it only answers for nfsd's v4 lock manager, and new nfsd control and observability interfaces are going to the netlink family instead.So I would rather frame this as parity than as new information: grace state is already exported, this puts it on the interface that is replacing the one exporting it, and -- per your point below -- fixes the semantics on the way over.
Quoted 7 lines
Just because the v4 recovery is done does not mean that you will be able to access files. The overall system grace period is a union of both the NLM and v4 grace periods.
This patch is reporting just the v4 one. If the goal is for the admin to reliably tell when the grace period has been lifted, then this won't tell us that.
Agreed, and "has the grace period lifted" is the question I actually care about. I mirrored the semantics of
/proc/fs/nfsd/v4_end_gracewithout stopping to ask whether those were the useful semantics to expose in a new interface. They aren't.Quoted 4 lines
Maybe this should just return the result of
locks_in_grace()instead?Or, do you plan to add similar functionality to lockd? Then you can just do the union in userland to figure out if it has been lifted.
I don't think lockd needs to grow anything --
fs/nfs_common/grace.calready gives us both halves, and nfsd calls into both helpers today:-
locks_in_grace()is the union you describe: true while any lock manager in the net ns is still in grace.-
opens_in_grace()is true only while a manager with ->block_opens is in grace, which per the comment on structlock_manageris the NFSv4-and-up case; NLM doesn't set it.
Quoted 7 lines
So for v2 I'll report
locks_in_grace(), which answers "can this server hand out normal locks and opens yet" rather than "is nfsd's v4 lock manager done". If it's ever worth distinguishing the two, a second attribute carryingopens_in_grace()gets there without touching lockd, and userland can derive the NLM-only case as (locks && !opens). I'd rather not add that speculatively, but say the word if you'd like both in v2.
Although NFSD does not yet support putting individual exports in grace, that is an open possibility for handling things like the reboot of a backend NFS server for a re-exported NFS client. I'd like any new API to avoid foreclosing the ability to move exports in and out of grace individually rather than as a single unit.
Signature · 1 line
Chuck Lever
Re: [PATCH v1 1/3] nfsd: report NFSv4 grace state through Netlink
Quoted 2 lines · Chuck Lever
I'd like any new API to avoid foreclosing the ability to move exports in and out of grace individually.
Thanks, that makes sense. A network-namespace-wide boolean would remain useful as an aggregate, but the current single attribute does not make its scope explicit enough for future per-export grace.
I will avoid freezing that ABI in this reroll. I'll drop the grace-reporting patch from the client-visibility v2 and post the client dump and state-count patches separately.
I'll follow up with a dedicated grace-status proposal whose records carry an explicit scope, so it can represent the server-wide state today and individual exports later. I also prefer to get agreement on the export identity before adding that part speculatively.
Does that direction address your concern?
Thanks,
Prabhakar
Chuck Lever
maintainerRe: [PATCH v1 1/3] nfsd: report NFSv4 grace state through Netlink
Quoted 12 lines · Prabhakar Pujeri
Quoted 2 lines · Chuck Lever
I'd like any new API to avoid foreclosing the ability to move exports in and out of grace individually.
Thanks, that makes sense. A network-namespace-wide boolean would remain useful as an aggregate, but the current single attribute does not make its scope explicit enough for future per-export grace.
I will avoid freezing that ABI in this reroll. I'll drop the grace-reporting patch from the client-visibility v2 and post the client dump and state-count patches separately.
Quoted 6 lines
I'll follow up with a dedicated grace-status proposal whose records carry an explicit scope, so it can represent the server-wide state today and individual exports later. I also prefer to get agreement on the export identity before adding that part speculatively.
Does that direction address your concern?
WFM!
Signature · 1 line
Chuck Lever