nfsd/sunrpc: convert nfsstat server-side interfaces to use netlink
Subsystems (4)
Jeff Layton
maintainerv7 cover nfsd/sunrpc: convert nfsstat server-side interfaces to use netlink
I last sent this series about a month ago, and dropped the ball on sending out a newer version afterward. This version fixes some minor issues that Chuck and Sashiko pointed out in review.
The nfsstat tool currently scrapes /proc/net/rpc/nfsd for server statistics. This procfs interface has several limitations: the counters are global (not network-namespace-aware), the format is fragile to parse, and it cannot be extended without breaking existing parsers.
This series adds per-network-namespace procedure call counts to the sunrpc layer and exposes them through a new netlink handler in the nfsd generic netlink family, allowing nfsstat to retrieve server statistics via netlink with a procfs fallback for older kernels. The nfs-utils patch for that part is already merged.
Additionally, this version adds tracking of callback operation counts.
Ideally, I'd like to see this go into v7.3.
---
Changes in v7:
- Skip never-called operations in the server-stats netlink dump instead
of emitting a zero-count entry for every opcode
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Emit the entire server-stats object in a single netlink dump message
- Count CB_SEQUENCE on v4.1+ callbacks, for parity with the forechannel SEQUENCE count
- Fix a use-after-free in nfsd4_run_cb() by snapshotting the opcode, net, and minorversion
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Add NFSv4 callback call counts to nfsstat upcall
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Ensure error is returned when nla_put_*() calls fail
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Only increment per-net stats for the "primary" program
- Link to v2: https://lore.kernel.org/r/[email protected]
---
Jeff Layton (6):
sunrpc: add per-netns per-procedure call counts to svc_stat
sunrpc: use per-net counts in svc_seq_show()
nfsd: implement server-stats-get netlink handler
sunrpc: remove unused svc_version vs_count field
nfsd: count NFSv4 callback operations per netns
nfsd: export NFSv4 callback op stats via netlink
Documentation/netlink/specs/nfsd.yaml | 111 ++++++++++++++ fs/lockd/svc4proc.c | 4 - fs/lockd/svcproc.c | 7 - fs/nfs/callback_xdr.c | 6 - fs/nfsd/localio.c | 3 - fs/nfsd/netlink.c | 5 + fs/nfsd/netlink.h | 2 + fs/nfsd/netns.h | 12 +- fs/nfsd/nfs2acl.c | 3 - fs/nfsd/nfs3acl.c | 3 - fs/nfsd/nfs3proc.c | 3 - fs/nfsd/nfs4callback.c | 22 ++- fs/nfsd/nfs4proc.c | 3 - fs/nfsd/nfs4state.c | 2 - fs/nfsd/nfsctl.c | 271 +++++++++++++++++++++++++++++++++- fs/nfsd/nfsproc.c | 3 - fs/nfsd/stats.c | 2 +- fs/nfsd/stats.h | 5 +- include/linux/sunrpc/stats.h | 6 + include/linux/sunrpc/svc.h | 1 - include/uapi/linux/nfsd_netlink.h | 36 +++++ net/sunrpc/stats.c | 2 +- net/sunrpc/svc.c | 63 +++++++- 23 files changed, 528 insertions(+), 47 deletions(-)
---
base-commit: 4f30ae575e0541214c6b5afe5ee29068b6fab9e1
change-id: 20260316-exportd-netlink-1c9fb52536e3
Best regards,
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerPATCH v7 1/6 sunrpc: add per-netns per-procedure call counts to svc_stat
The existing per-procedure call counts live in global svc_version->vs_count[] arrays which are not network-namespace-aware. Add per-netns equivalents in struct svc_stat so the upcoming netlink stats interface can return namespace-scoped statistics.
Add a vs_count pointer array to struct svc_stat, along with svc_stat_alloc_counts() and svc_stat_free_counts() helpers to manage per-version percpu call count arrays.
Increment the per-net counter alongside the global one in svc_generic_init_request(). Call the alloc/free helpers from nfsd_net_init() and nfsd_net_exit().
--- fs/nfsd/nfsctl.c | 8 +++++- include/linux/sunrpc/stats.h | 6 +++++ net/sunrpc/svc.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 75 insertions(+), 1 deletion(-)
fs/nfsd/nfsctl.c +7 -1
@@ -2516,9 +2516,12 @@ static __net_init int nfsd_net_init(struct net *net)memset (&nn->nfsd_svcstats,0 ,sizeof (nn->nfsd_svcstats)); nn->nfsd_svcstats.program = &nfsd_programs[0 ]; + retval =svc_stat_alloc_counts (&nn->nfsd_svcstats); +if (retval) + goto out_proc_error;if (!nfsd_proc_stat_init (net)) { retval = -ENOMEM ; - goto out_proc_error; + goto out_svcstats_error; }for (i =0 ; i <sizeof (nn->nfsd_versions); i++)
@@ -2536,6 +2539,8 @@ static __net_init int nfsd_net_init(struct net *net)#endif return 0 ; +out_svcstats_error: +svc_stat_free_counts (&nn->nfsd_svcstats); out_proc_error:percpu_counter_destroy_many (nn->counter,NFSD_STATS_COUNTERS_NUM ); out_repcache_error:
@@ -2576,6 +2581,7 @@ static __net_exit void nfsd_net_exit(struct net *net)kfree_sensitive (nn->fh_key);nfsd_net_cb_shutdown (nn);nfsd_proc_stat_shutdown (net); +svc_stat_free_counts (&nn->nfsd_svcstats);percpu_counter_destroy_many (nn->counter,NFSD_STATS_COUNTERS_NUM );nfsd_idmap_shutdown (net);nfsd_export_shutdown (net);
include/linux/sunrpc/stats.h +6 -0
@@ -37,9 +37,15 @@ struct svc_stat { rpcbadfmt, rpcbadauth, rpcbadclnt; + +/* Per-version per-procedure call counts (per-cpu, per-netns) */ +unsigned long __percpu **vs_count; };struct net ; +int svc_stat_alloc_counts (struct svc_stat *statp); +void svc_stat_free_counts (struct svc_stat *statp); +#ifdef CONFIG_PROC_FS int rpc_proc_init (struct net *);void rpc_proc_exit (struct net *);
net/sunrpc/svc.c +62 -0
74 lines not on this page · raw message
Signature · 1 line
2.55.0
Jeff Layton
maintainerPATCH v7 2/6 sunrpc: use per-net counts in svc_seq_show()
In reply to Jeff LaytonUpdate svc_seq_show() to read from the per-netns statp->vs_count[] arrays instead of the global svc_version->vs_count[].
The only caller is nfsd, which always allocates vs_count via svc_stat_alloc_counts() in nfsd_net_init(), so the per-netns arrays are always available.
This makes /proc/net/rpc/nfsd report per-network-namespace procedure call counts.
--- net/sunrpc/stats.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/stats.c +1 -1
@@ -108,7 +108,7 @@ void svc_seq_show(struct seq_file *seq, const struct svc_stat *statp)for (j =0 ; j < vers->vs_nproc; j++) { count =0 ;for_each_possible_cpu (k) - count +=per_cpu (vers->vs_count[j], k); + count +=per_cpu (statp->vs_count[i][j], k);seq_printf (seq," %lu" , count); }seq_putc (seq,'\n' );
Signature · 1 line
2.55.0
Jeff Layton
maintainerPATCH v7 3/6 nfsd: implement server-stats-get netlink handler
In reply to Jeff LaytonImplement nfsd_nl_server_stats_get_dumpit() which exposes the NFS server statistics currently available via /proc/net/rpc/nfsd through the nfsd generic netlink family.
The handler uses a dump operation to stream statistics across one or more netlink messages. The reply is divided into sections that are emitted in order:
- scalar stats (reply cache, filehandle, IO, network, RPC), emitted once in the first message, then
- per-version procedure counts (proc2/3/4-ops) and the NFSv4 per-operation counts (proc4ops-ops), using the per-netns vs_count arrays.
cb->args[0] tracks the current section and cb->args[1] the entry index within it, so a section that does not fit in the current message is closed and resumed in the next one. This matters because the first dump message is allocated at NLMSG_GOODSIZE (a single page on most architectures) regardless of the client's receive buffer; packing every counter into one message would overflow it and fail the dump with -EMSGSIZE. Userspace merges the attributes from every message.
This allows nfsstat to retrieve server statistics via netlink with a procfs fallback for older kernels.
--- Documentation/netlink/specs/nfsd.yaml | 105 ++++++++++++++++ fs/nfsd/netlink.c | 5 + fs/nfsd/netlink.h | 2 + fs/nfsd/nfsctl.c | 222 ++++++++++++++++++++++++++++++++++ include/uapi/linux/nfsd_netlink.h | 35 ++++++ 5 files changed, 369 insertions(+)
Documentation/netlink/specs/nfsd.yaml +105 -0
@@ -330,6 +330,86 @@ attribute-sets: of which client holds the state. Intended for use after all clients have been unexported from a given path, enabling the underlying filesystem to be unmounted. + - + name: server-proc-entry + attributes: + - + name: op + type: u32 + - + name: count + type: u64 + - + name: pad + type: pad + - + name: server-stats + attributes: + - + name: rc-hits + type: u64 + - + name: rc-misses + type: u64 + - + name: rc-nocache + type: u64 + - + name: pad + type: pad + - + name: fh-stale + type: u64 + - + name: io-read + type: u64 + - + name: io-write + type: u64 + - + name: netcnt + type: u32 + - + name: netudpcnt + type: u32 + - + name: nettcpcnt + type: u32 + - + name: nettcpconn + type: u32 + - + name: rpccnt + type: u32 + - + name: rpcbadfmt + type: u32 + - + name: rpcbadauth + type: u32 + - + name: rpcbadclnt + type: u32 + - + name: proc2-ops + type: nest + nested-attributes: server-proc-entry + multi-attr: true + - + name: proc3-ops + type: nest + nested-attributes: server-proc-entry + multi-attr: true + - + name: proc4-ops + type: nest + nested-attributes: server-proc-entry + multi-attr: true + - + name: proc4ops-ops + type: nest + nested-attributes: server-proc-entry + multi-attr: true operations: list:
@@ -516,6 +596,31 @@ operations: request: attributes: - path + - + name: server-stats-get + doc: dump NFS server statistics + attribute-set: server-stats + dump: + reply: + attributes: + - rc-hits + - rc-misses + - rc-nocache + - fh-stale + - io-read + - io-write + - netcnt + - netudpcnt + - nettcpcnt + - nettcpconn + - rpccnt + - rpcbadfmt + - rpcbadauth + - rpcbadclnt + - proc2-ops + - proc3-ops + - proc4-ops + - proc4ops-ops mcast-groups: list:
fs/nfsd/netlink.c +5 -0
@@ -225,6 +225,11 @@ static const struct genl_split_ops nfsd_nl_ops[] = { .maxattr = NFSD_A_UNLOCK_EXPORT_PATH, .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, }, + { + .cmd = NFSD_CMD_SERVER_STATS_GET, + .dumpit = nfsd_nl_server_stats_get_dumpit, + .flags = GENL_CMD_CAP_DUMP, + }, };static const struct genl_multicast_group nfsd_nl_mcgrps[] = {
fs/nfsd/netlink.h +2 -0
@@ -42,6 +42,8 @@ int nfsd_nl_cache_flush_doit(struct sk_buff *skb, struct genl_info *info);int nfsd_nl_unlock_ip_doit (struct sk_buff *skb,struct genl_info *info);int nfsd_nl_unlock_filesystem_doit (struct sk_buff *skb,struct genl_info *info);int nfsd_nl_unlock_export_doit (struct sk_buff *skb,struct genl_info *info); +int nfsd_nl_server_stats_get_dumpit (struct sk_buff *skb, +struct netlink_callback *cb);enum {NFSD_NLGRP_NONE ,
fs/nfsd/nfsctl.c +222 -0
228 lines not on this page · raw message
include/uapi/linux/nfsd_netlink.h +35 -0
@@ -225,6 +225,40 @@ enum {NFSD_A_UNLOCK_EXPORT_MAX = (__NFSD_A_UNLOCK_EXPORT_MAX -1 ) }; +enum { +NFSD_A_SERVER_PROC_ENTRY_OP =1 , +NFSD_A_SERVER_PROC_ENTRY_COUNT , +NFSD_A_SERVER_PROC_ENTRY_PAD , + + __NFSD_A_SERVER_PROC_ENTRY_MAX, +NFSD_A_SERVER_PROC_ENTRY_MAX = (__NFSD_A_SERVER_PROC_ENTRY_MAX -1 ) +}; + +enum { +NFSD_A_SERVER_STATS_RC_HITS =1 , +NFSD_A_SERVER_STATS_RC_MISSES , +NFSD_A_SERVER_STATS_RC_NOCACHE , +NFSD_A_SERVER_STATS_PAD , +NFSD_A_SERVER_STATS_FH_STALE , +NFSD_A_SERVER_STATS_IO_READ , +NFSD_A_SERVER_STATS_IO_WRITE , +NFSD_A_SERVER_STATS_NETCNT , +NFSD_A_SERVER_STATS_NETUDPCNT , +NFSD_A_SERVER_STATS_NETTCPCNT , +NFSD_A_SERVER_STATS_NETTCPCONN , +NFSD_A_SERVER_STATS_RPCCNT , +NFSD_A_SERVER_STATS_RPCBADFMT , +NFSD_A_SERVER_STATS_RPCBADAUTH , +NFSD_A_SERVER_STATS_RPCBADCLNT , +NFSD_A_SERVER_STATS_PROC2_OPS , +NFSD_A_SERVER_STATS_PROC3_OPS , +NFSD_A_SERVER_STATS_PROC4_OPS , +NFSD_A_SERVER_STATS_PROC4OPS_OPS , + + __NFSD_A_SERVER_STATS_MAX, +NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX -1 ) +}; +enum {NFSD_CMD_RPC_STATUS_GET =1 ,NFSD_CMD_THREADS_SET ,
@@ -244,6 +278,7 @@ enum {NFSD_CMD_UNLOCK_IP ,NFSD_CMD_UNLOCK_FILESYSTEM ,NFSD_CMD_UNLOCK_EXPORT , +NFSD_CMD_SERVER_STATS_GET , __NFSD_CMD_MAX,NFSD_CMD_MAX = (__NFSD_CMD_MAX -1 )
Signature · 1 line
2.55.0
Jeff Layton
maintainerPATCH v7 4/6 sunrpc: remove unused svc_version vs_count field
In reply to Jeff LaytonNow that svc_seq_show() and the nfsd netlink stats handler both use the per-netns svc_stat vs_count arrays, the global per-version vs_count percpu counters are no longer read by anything. Remove the vs_count field from struct svc_version and all the associated DEFINE_PER_CPU_ALIGNED arrays and initializers across nfsd, lockd, and the NFS client callback service.
--- fs/lockd/svc4proc.c | 4 ---- fs/lockd/svcproc.c | 7 ------- fs/nfs/callback_xdr.c | 6 ------ fs/nfsd/localio.c | 3 --- fs/nfsd/nfs2acl.c | 3 --- fs/nfsd/nfs3acl.c | 3 --- fs/nfsd/nfs3proc.c | 3 --- fs/nfsd/nfs4proc.c | 3 --- fs/nfsd/nfsproc.c | 3 --- include/linux/sunrpc/svc.h | 1 - net/sunrpc/svc.c | 3 --- 11 files changed, 39 deletions(-)
fs/lockd/svc4proc.c +0 -4
@@ -1423,14 +1423,10 @@ union nlm4svc_xdrstore {struct nlm4_shareres_wrapper shareres; }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nlm4svc_call_counters[ARRAY_SIZE(nlm4svc_procedures)]); -const struct svc_version nlmsvc_version4 = { .vs_vers =4 , .vs_nproc =ARRAY_SIZE (nlm4svc_procedures), .vs_proc = nlm4svc_procedures, - .vs_count = nlm4svc_call_counters, .vs_dispatch = nlmsvc_dispatch, .vs_xdrsize =sizeof (union nlm4svc_xdrstore ), };
fs/lockd/svcproc.c +0 -7
@@ -1436,25 +1436,18 @@ union nlmsvc_xdrstore {* NLMv1 defines only procedures 1 - 15. Linux lockd also implements * procedures 0 (NULL) and 16 (SM_NOTIFY). */ -static DEFINE_PER_CPU_ALIGNED(unsigned long, nlm1svc_call_counters[17]); -const struct svc_version nlmsvc_version1 = { .vs_vers =1 , .vs_nproc =17 , .vs_proc = nlmsvc_procedures, - .vs_count = nlm1svc_call_counters, .vs_dispatch = nlmsvc_dispatch, .vs_xdrsize =sizeof (union nlmsvc_xdrstore ), }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nlm3svc_call_counters[ARRAY_SIZE(nlmsvc_procedures)]); -const struct svc_version nlmsvc_version3 = { .vs_vers =3 , .vs_nproc =ARRAY_SIZE (nlmsvc_procedures), .vs_proc = nlmsvc_procedures, - .vs_count = nlm3svc_call_counters, .vs_dispatch = nlmsvc_dispatch, .vs_xdrsize =sizeof (union nlmsvc_xdrstore ), };
fs/nfs/callback_xdr.c +0 -6
@@ -1090,26 +1090,20 @@ static const struct svc_procedure nfs4_callback_procedures1[] = { } }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfs4_callback_count1[ARRAY_SIZE(nfs4_callback_procedures1)]);const struct svc_version nfs4_callback_version1 = { .vs_vers =1 , .vs_nproc =ARRAY_SIZE (nfs4_callback_procedures1), .vs_proc = nfs4_callback_procedures1, - .vs_count = nfs4_callback_count1, .vs_xdrsize =NFS4_CALLBACK_XDRSIZE , .vs_dispatch = nfs_callback_dispatch, .vs_hidden = true, .vs_need_cong_ctrl = true, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfs4_callback_count4[ARRAY_SIZE(nfs4_callback_procedures1)]);const struct svc_version nfs4_callback_version4 = { .vs_vers =4 , .vs_nproc =ARRAY_SIZE (nfs4_callback_procedures1), .vs_proc = nfs4_callback_procedures1, - .vs_count = nfs4_callback_count4, .vs_xdrsize =NFS4_CALLBACK_XDRSIZE , .vs_dispatch = nfs_callback_dispatch, .vs_hidden = true,
fs/nfsd/localio.c +0 -3
@@ -210,14 +210,11 @@ static const struct svc_procedure localio_procedures1[] = { };#define LOCALIO_NR_PROCEDURES ARRAY_SIZE(localio_procedures1) -static DEFINE_PER_CPU_ALIGNED(unsigned long, - localio_count[LOCALIO_NR_PROCEDURES]);const struct svc_version localio_version1 = { .vs_vers =1 , .vs_nproc =LOCALIO_NR_PROCEDURES , .vs_proc = localio_procedures1, .vs_dispatch = nfsd_dispatch, - .vs_count = localio_count, .vs_xdrsize =XDR_QUADLEN (UUID_SIZE ), .vs_hidden = true, };
fs/nfsd/nfs2acl.c +0 -3
@@ -388,13 +388,10 @@ static const struct svc_procedure nfsd_acl_procedures2[5] = { }, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfsd_acl_count2[ARRAY_SIZE(nfsd_acl_procedures2)]);const struct svc_version nfsd_acl_version2 = { .vs_vers =2 , .vs_nproc =ARRAY_SIZE (nfsd_acl_procedures2), .vs_proc = nfsd_acl_procedures2, - .vs_count = nfsd_acl_count2, .vs_dispatch = nfsd_dispatch, .vs_xdrsize =NFS3_SVC_XDRSIZE , };
fs/nfsd/nfs3acl.c +0 -3
@@ -278,13 +278,10 @@ static const struct svc_procedure nfsd_acl_procedures3[3] = { }, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfsd_acl_count3[ARRAY_SIZE(nfsd_acl_procedures3)]);const struct svc_version nfsd_acl_version3 = { .vs_vers =3 , .vs_nproc =ARRAY_SIZE (nfsd_acl_procedures3), .vs_proc = nfsd_acl_procedures3, - .vs_count = nfsd_acl_count3, .vs_dispatch = nfsd_dispatch, .vs_xdrsize =NFS3_SVC_XDRSIZE , };
fs/nfsd/nfs3proc.c +0 -3
@@ -1108,13 +1108,10 @@ static const struct svc_procedure nfsd_procedures3[22] = { }, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfsd_count3[ARRAY_SIZE(nfsd_procedures3)]);const struct svc_version nfsd_version3 = { .vs_vers =3 , .vs_nproc =ARRAY_SIZE (nfsd_procedures3), .vs_proc = nfsd_procedures3, .vs_dispatch = nfsd_dispatch, - .vs_count = nfsd_count3, .vs_xdrsize =NFS3_SVC_XDRSIZE , };
fs/nfsd/nfs4proc.c +0 -3
@@ -4262,13 +4262,10 @@ static const struct svc_procedure nfsd_procedures4[2] = { }, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfsd_count4[ARRAY_SIZE(nfsd_procedures4)]);const struct svc_version nfsd_version4 = { .vs_vers =4 , .vs_nproc =ARRAY_SIZE (nfsd_procedures4), .vs_proc = nfsd_procedures4, - .vs_count = nfsd_count4, .vs_dispatch = nfsd_dispatch, .vs_xdrsize =NFS4_SVC_XDRSIZE , .vs_rpcb_optnl = true,
fs/nfsd/nfsproc.c +0 -3
@@ -845,13 +845,10 @@ static const struct svc_procedure nfsd_procedures2[18] = { }, }; -static DEFINE_PER_CPU_ALIGNED(unsigned long, - nfsd_count2[ARRAY_SIZE(nfsd_procedures2)]);const struct svc_version nfsd_version2 = { .vs_vers =2 , .vs_nproc =ARRAY_SIZE (nfsd_procedures2), .vs_proc = nfsd_procedures2, - .vs_count = nfsd_count2, .vs_dispatch = nfsd_dispatch, .vs_xdrsize =NFS2_SVC_XDRSIZE , };
include/linux/sunrpc/svc.h +0 -1
@@ -407,7 +407,6 @@ struct svc_version {u32 vs_vers;/* version number */ u32 vs_nproc;/* number of procedures */ const struct svc_procedure *vs_proc;/* per-procedure info */ -unsigned long __percpu *vs_count;/* call counts */ u32 vs_xdrsize;/* xdrsize needed for this version */ /* Don't register with rpcbind */
net/sunrpc/svc.c +0 -3
@@ -1345,9 +1345,6 @@ svc_generic_init_request(struct svc_rqst *rqstp,memset (rqstp->rq_argp,0 , procp->pc_argzero);memset (rqstp->rq_resp,0 , procp->pc_ressize); -/* Bump per-procedure stats counter */ -this_cpu_inc (versp->vs_count[rqstp->rq_proc]); -/* Bump per-net per-procedure stats counter */ if (rqstp->rq_server->sv_stats && rqstp->rq_server->sv_stats->program == progp &&
Signature · 1 line
2.55.0
Jeff Layton
maintainerPATCH v7 5/6 nfsd: count NFSv4 callback operations per netns
In reply to Jeff LaytonThe NFS server tracks per-operation call counts for the forward channel (proc4ops) but keeps no statistics for the NFSv4 backchannel (callback) operations it sends to clients.
Add a per-netns array of percpu counters for callback operations, indexed by RFC 8881 callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD), and bump the relevant counter in nfsd4_run_cb(), which is hit exactly once per callback that is actually queued.
CB_GETATTR is sent when a GETATTR conflicts with an outstanding write delegation, which is roughly what the dedicated wdeleg_getattr counter tracked. The two are not identical: the old counter incremented on every such conflict, whereas the CB_GETATTR counter only counts callbacks that are actually queued, so concurrent conflicts that coalesce onto an already in-flight CB_GETATTR are now counted once rather than once per conflict. Report the procfs "wdeleg_getattr" line from the CB_GETATTR counter and drop the now-redundant NFSD_STATS_WDELEG_GETATTR counter, its helper, and its increment site.
--- fs/nfsd/netns.h | 12 +++++++++++- fs/nfsd/nfs4callback.c | 22 +++++++++++++++++++++- fs/nfsd/nfs4state.c | 2 -- fs/nfsd/nfsctl.c | 14 ++++++++++++++ fs/nfsd/stats.c | 2 +- fs/nfsd/stats.h | 5 +++-- 6 files changed, 50 insertions(+), 7 deletions(-)
fs/nfsd/netns.h +11 -1
@@ -53,11 +53,16 @@ enum {NFSD_STATS_FIRST_NFS4_OP ,/* count of individual nfsv4 operations */ NFSD_STATS_LAST_NFS4_OP =NFSD_STATS_FIRST_NFS4_OP +LAST_NFS4_OP , #define NFSD_STATS_NFS4_OP(op) (NFSD_STATS_FIRST_NFS4_OP + (op)) -NFSD_STATS_WDELEG_GETATTR ,/* count of getattr conflict with wdeleg */ #endifNFSD_STATS_COUNTERS_NUM }; +/* +* Per-netns NFSv4 callback (backchannel) per-operation counters, indexed +* directly by RFC 8881 callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD). +*/ +#define NFSD_STATS_CB_OPS_NUM (OP_CB_OFFLOAD + 1) +/* * Represents a nfsd "container". With respect to nfsv4 state tracking, the * fields of interest are the *_id_hashtbls and the *_name_tree. These track
@@ -200,6 +205,11 @@ struct nfsd_net {/* Per-netns stats counters */ struct percpu_counter counter[NFSD_STATS_COUNTERS_NUM ]; +#ifdef CONFIG_NFSD_V4 +/* Per-netns NFSv4 callback (backchannel) per-operation counters */ +struct percpu_counter cb_counter[NFSD_STATS_CB_OPS_NUM ]; +#endif +/* sunrpc svc stats */ struct svc_stat nfsd_svcstats;
fs/nfsd/nfs4callback.c +21 -1
@@ -39,6 +39,7 @@#include "nfsd.h" #include "state.h" #include "netns.h" +#include "stats.h" #include "trace.h" #include "xdr4cb.h" #include "xdr4.h"
@@ -1921,12 +1922,31 @@ void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,bool nfsd4_run_cb (struct nfsd4_callback *cb) {struct nfs4_client *clp = cb->cb_clp; +struct nfsd_net *nn =net_generic (clp->net, nfsd_net_id); +const struct nfsd4_callback_ops *ops = cb->cb_ops; +u32 minorversion = clp->cl_minorversion;bool queued;nfsd41_cb_inflight_begin (clp); queued =nfsd4_queue_cb (cb); -if (!queued) +if (queued) { +if (ops) { +nfsd_stats_cb_op_inc (nn, ops->opcode); +/* +* Minorversion > 0 callbacks prepend a CB_SEQUENCE op +* (see encode_cb_sequence4args()); count it like the +* forechannel counts SEQUENCE, so it isn't perpetually +* reported as zero. CB_NULL probes (ops == NULL) carry +* no CB_SEQUENCE -- and on 4.1+ they are dropped without +* sending any RPC (see nfsd4_run_cb_work()) -- so they +* must not be counted here. +*/ +if (minorversion >0 ) +nfsd_stats_cb_op_inc (nn,OP_CB_SEQUENCE ); + } + }else {nfsd41_cb_inflight_end (clp); + }return queued; }
fs/nfsd/nfs4state.c +0 -2
@@ -10122,7 +10122,6 @@ __be32 nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct dentry *dentry, struct nfs4_delegation **pdp) { - struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id); struct nfsd_thread_local_info *ntli = rqstp->rq_private;struct file_lock_context *ctx;struct nfs4_delegation *dp =NULL ;
@@ -10162,7 +10161,6 @@ nfsd4_deleg_getattr_conflict(struct svc_rqst *rqstp, struct dentry *dentry,return 0 ; } -nfsd_stats_wdeleg_getattr_inc (nn);refcount_inc (&dp->dl_stid.sc_count); ncf = &dp->dl_cb_fattr;nfs4_cb_getattr (&dp->dl_cb_fattr);
fs/nfsd/nfsctl.c +14 -0
@@ -2736,6 +2736,13 @@ static __net_init int nfsd_net_init(struct net *net)if (retval) goto out_repcache_error; +#ifdef CONFIG_NFSD_V4 + retval =percpu_counter_init_many (nn->cb_counter,0 ,GFP_KERNEL , +NFSD_STATS_CB_OPS_NUM ); +if (retval) + goto out_cb_counter_error; +#endif +memset (&nn->nfsd_svcstats,0 ,sizeof (nn->nfsd_svcstats)); nn->nfsd_svcstats.program = &nfsd_programs[0 ]; retval =svc_stat_alloc_counts (&nn->nfsd_svcstats);
@@ -2764,6 +2771,10 @@ static __net_init int nfsd_net_init(struct net *net) out_svcstats_error:svc_stat_free_counts (&nn->nfsd_svcstats); out_proc_error: +#ifdef CONFIG_NFSD_V4 +percpu_counter_destroy_many (nn->cb_counter,NFSD_STATS_CB_OPS_NUM ); +out_cb_counter_error: +#endif percpu_counter_destroy_many (nn->counter,NFSD_STATS_COUNTERS_NUM ); out_repcache_error:nfsd_idmap_shutdown (net);
@@ -2804,6 +2815,9 @@ static __net_exit void nfsd_net_exit(struct net *net)nfsd_net_cb_shutdown (nn);nfsd_proc_stat_shutdown (net);svc_stat_free_counts (&nn->nfsd_svcstats); +#ifdef CONFIG_NFSD_V4 +percpu_counter_destroy_many (nn->cb_counter,NFSD_STATS_CB_OPS_NUM ); +#endif percpu_counter_destroy_many (nn->counter,NFSD_STATS_COUNTERS_NUM );nfsd_idmap_shutdown (net);nfsd_export_shutdown (net);
fs/nfsd/stats.c +1 -1
@@ -65,7 +65,7 @@ static int nfsd_show(struct seq_file *seq, void *v) percpu_counter_sum_positive(&nn->counter[NFSD_STATS_NFS4_OP(i)])); }seq_printf (seq,"\nwdeleg_getattr %lld" , -percpu_counter_sum_positive (&nn->counter[NFSD_STATS_WDELEG_GETATTR ])); +percpu_counter_sum_positive (&nn->cb_counter[OP_CB_GETATTR ]));seq_putc (seq,'\n' );#endif
fs/nfsd/stats.h +3 -2
@@ -71,9 +71,10 @@ static inline void nfsd_stats_drc_mem_usage_sub(struct nfsd_net *nn, s64 amount) }#ifdef CONFIG_NFSD_V4 -static inline void nfsd_stats_wdeleg_getattr_inc (struct nfsd_net *nn) +static inline void nfsd_stats_cb_op_inc (struct nfsd_net *nn,u32 opcode) { -percpu_counter_inc (&nn->counter[NFSD_STATS_WDELEG_GETATTR ]); +if (opcode >=OP_CB_GETATTR && opcode <=OP_CB_OFFLOAD ) +percpu_counter_inc (&nn->cb_counter[opcode]); }#endif #endif /* _NFSD_STATS_H */
Signature · 1 line
2.55.0
Jeff Layton
maintainerPATCH v7 6/6 nfsd: export NFSv4 callback op stats via netlink
In reply to Jeff LaytonAdd a proc4cb-ops nested attribute to the server-stats netlink dump, reusing the existing server-proc-entry (op/count) layout. The dump gains a callback section that emits one entry per callback opcode (OP_CB_GETATTR..OP_CB_OFFLOAD) from the per-netns callback counters, paged across messages like the other per-operation sections.
This lets nfsstat report NFSv4 backchannel operation counts over netlink, including CB_GETATTR which corresponds to the procfs wdeleg_getattr line.
--- Documentation/netlink/specs/nfsd.yaml | 6 ++++++ fs/nfsd/nfsctl.c | 27 +++++++++++++++++++++++++++ include/uapi/linux/nfsd_netlink.h | 1 + 3 files changed, 34 insertions(+)
Documentation/netlink/specs/nfsd.yaml +6 -0
@@ -410,6 +410,11 @@ attribute-sets: type: nest nested-attributes: server-proc-entry multi-attr: true + - + name: proc4cb-ops + type: nest + nested-attributes: server-proc-entry + multi-attr: true operations: list:
@@ -621,6 +626,7 @@ operations: - proc3-ops - proc4-ops - proc4ops-ops + - proc4cb-ops mcast-groups: list:
fs/nfsd/nfsctl.c +27 -0
@@ -2451,6 +2451,29 @@ static int nfsd_nl_server_stats_nfs4ops(struct sk_buff *skb, return 0; } + +/* +* Emit NFSv4 callback (backchannel) per-operation counts, resuming at *idx, +* which counts from OP_CB_GETATTR. Same return convention as +* nfsd_nl_server_stats_proc(). +*/ +static int nfsd_nl_server_stats_cbops (struct sk_buff *skb, +struct nfsd_net *nn,int *idx) +{ +int op; + +for (op =OP_CB_GETATTR + *idx; op <=OP_CB_OFFLOAD ; op++, (*idx)++) { +u64 cnt =percpu_counter_sum_positive (&nn->cb_counter[op]); + +if (!cnt) +continue ; +if (nfsd_nl_put_proc_entry (skb,NFSD_A_SERVER_STATS_PROC4CB_OPS , + op, cnt)) +return -EMSGSIZE ; + } + +return 0 ; +}#endif /* Sections of the server-stats dump, emitted in order across messages. */
@@ -2459,6 +2482,7 @@ enum {NFSD_SERVER_STATS_PROC2 ,NFSD_SERVER_STATS_PROC3 ,NFSD_SERVER_STATS_PROC4 , +NFSD_SERVER_STATS_PROC4CB ,NFSD_SERVER_STATS_PROC4OPS , NFSD_SERVER_STATS_DONE, };
@@ -2527,6 +2551,9 @@ int nfsd_nl_server_stats_get_dumpit(struct sk_buff *skb, NFSD_A_SERVER_STATS_PROC4_OPS, &idx);break ;#ifdef CONFIG_NFSD_V4 +case NFSD_SERVER_STATS_PROC4CB : + ret =nfsd_nl_server_stats_cbops (skb, nn, &idx); +break ;case NFSD_SERVER_STATS_PROC4OPS : ret =nfsd_nl_server_stats_nfs4ops (skb, nn, &idx);break ;
include/uapi/linux/nfsd_netlink.h +1 -0
@@ -254,6 +254,7 @@ enum {NFSD_A_SERVER_STATS_PROC3_OPS ,NFSD_A_SERVER_STATS_PROC4_OPS ,NFSD_A_SERVER_STATS_PROC4OPS_OPS , +NFSD_A_SERVER_STATS_PROC4CB_OPS , __NFSD_A_SERVER_STATS_MAX,NFSD_A_SERVER_STATS_MAX = (__NFSD_A_SERVER_STATS_MAX -1 )
Signature · 1 line
2.55.0
Chuck Lever
maintainerQuoted 11 lines · Jeff Layton
I last sent this series about a month ago, and dropped the ball on sending out a newer version afterward. This version fixes some minor issues that Chuck and Sashiko pointed out in review.
The nfsstat tool currently scrapes
/proc/net/rpc/nfsdfor server statistics. This procfs interface has several limitations: the counters are global (not network-namespace-aware), the format is fragile to parse, and it cannot be extended without breaking existing parsers.[...]
Applied to nfsd-testing, thanks!
[1/6] sunrpc: add per-netns per-procedure call counts to svc_stat
commit: 99d92415bd4a11218abe303fc9e6c12fef392d87
[2/6] sunrpc: use per-net counts in svc_seq_show()
commit: 07d348fc22493ddf7f195903b929ddae1dd0f533
[3/6] nfsd: implement server-stats-get netlink handler
commit: 58656de1ad40362db64f5ac6debd2646b956fe1a
[4/6] sunrpc: remove unused svc_version vs_count field
commit: 09048531826cc3a5b0baca029ce3e3af432029aa
[5/6] nfsd: count NFSv4 callback operations per netns
commit: 5ce4d8f4b06b8f987b1cc88f13e8c1a1cf29db37
[6/6] nfsd: export NFSv4 callback op stats via netlink
commit: 134cf2c7e0bc44e69d52d955bac1c2348a86832f--
Chuck Lever