Skip to contentSkip to conversation

Improvements to nfsd stats

PATCH In review Amir Goldstein 3 participants 9 messages Jan 6, 2021 – Jan 22, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover Improvements to nfsd stats

Hi Bruce,

Per your request, I added a cleanup patch for unused counter.

Replaced the hacky counters array "union" with proper array and added helpers to update the counters to avoid human mistakes related to counter indices.

Thanks,
Amir.

Changes since v1:
- Cleanup unused stats counters (Bruce)
- Replace counters array hack with proper array (Chuck)
- Helpers to update both global and per-export stats

Amir Goldstein (3):
nfsd: remove unused stats counters
nfsd: protect concurrent access to nfsd stats counters
nfsd: report per-export stats

 fs/nfsd/export.c   |  68 +++++++++++++++++++++++----
 fs/nfsd/export.h   |  15 ++++++
 fs/nfsd/netns.h    |  23 +++++----
 fs/nfsd/nfs4proc.c |   2 +-
 fs/nfsd/nfscache.c |  52 +++++++++++++++------
 fs/nfsd/nfsctl.c   |   8 +++-
 fs/nfsd/nfsd.h     |   2 +-
 fs/nfsd/nfsfh.c    |   4 +-
 fs/nfsd/stats.c    | 114 +++++++++++++++++++++++++++++++--------------
 fs/nfsd/stats.h    |  96 +++++++++++++++++++++++++++++---------
 fs/nfsd/vfs.c      |   4 +-
 11 files changed, 292 insertions(+), 96 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/3 nfsd: remove unused stats counters

Commit 501cb1849f86 ("nfsd: rip out the raparms cache") removed the code that updates read-ahead cache stats counters,
commit 8bbfa9f3889b ("knfsd: remove the nfsd thread busy histogram") removed code that updates the thread busy stats counters back in 2009 and code that updated filehandle cache stats was removed back in 2002.

Remove the unused stats counters from nfsd_stats struct and print hardcoded zeros in /proc/net/rpc/nfsd.

---
 fs/nfsd/stats.c | 41 ++++++++++++++++-------------------------
 fs/nfsd/stats.h | 10 ----------
 2 files changed, 16 insertions(+), 35 deletions(-)
fs/nfsd/stats.c +16 -25
index b1bc582b0493..e928e224205a 100644
@@ -7,16 +7,14 @@
  * Format:
  *	rc <hits> <misses> <nocache>
  *			Statistsics for the reply cache
- *	fh <stale> <total-lookups> <anonlookups> <dir-not-in-dcache> <nondir-not-in-dcache>
+ *	fh <stale> <deprecated filehandle cache stats>
  *			statistics for filehandle lookup
  *	io <bytes-read> <bytes-written>
  *			statistics for IO throughput
- *	th <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%> 
- *			time (seconds) when nfsd thread usage above thresholds
- *			and number of times that all threads were in use
- *	ra cache-size  <10%  <20%  <30% ... <100% not-found
- *			number of times that read-ahead entry was found that deep in
- *			the cache.
+ *	th <threads> <deprecated thread usage histogram stats>
+ *			number of threads
+ *	ra <deprecated ra-cache stats>
+ *
  *	plus generic RPC stats (see net/sunrpc/stats.c)
  *
  * Copyright (C) 1995, 1996, 1997 Olaf Kirch <[email protected]>
@@ -38,31 +36,24 @@ static int nfsd_proc_show(struct seq_file *seq, void *v)
 {
 	int i;
 
-	seq_printf(seq, "rc %u %u %u\nfh %u %u %u %u %u\nio %u %u\n",
+	seq_printf(seq, "rc %u %u %u\nfh %u 0 0 0 0\nio %u %u\n",
 		      nfsdstats.rchits,
 		      nfsdstats.rcmisses,
 		      nfsdstats.rcnocache,
 		      nfsdstats.fh_stale,
-		      nfsdstats.fh_lookup,
-		      nfsdstats.fh_anon,
-		      nfsdstats.fh_nocache_dir,
-		      nfsdstats.fh_nocache_nondir,
 		      nfsdstats.io_read,
 		      nfsdstats.io_write);
+
 	/* thread usage: */
-	seq_printf(seq, "th %u %u", nfsdstats.th_cnt, nfsdstats.th_fullcnt);
-	for (i=0; i<10; i++) {
-		unsigned int jifs = nfsdstats.th_usage[i];
-		unsigned int sec = jifs / HZ, msec = (jifs % HZ)*1000/HZ;
-		seq_printf(seq, " %u.%03u", sec, msec);
-	}
-
-	/* newline and ra-cache */
-	seq_printf(seq, "\nra %u", nfsdstats.ra_size);
-	for (i=0; i<11; i++)
-		seq_printf(seq, " %u", nfsdstats.ra_depth[i]);
-	seq_putc(seq, '\n');
-	
+	seq_printf(seq, "th %u 0", nfsdstats.th_cnt);
+
+	/* deprecated thread usage histogram stats */
+	for (i = 0; i < 10; i++)
+		seq_puts(seq, " 0.000");
+
+	/* deprecated ra-cache stats */
+	seq_puts(seq, "\nra 0 0 0 0 0 0 0 0 0 0 0 0\n");
+
 	/* show my rpc info */
 	svc_seq_show(seq, &nfsd_svcstats);
 
fs/nfsd/stats.h +0 -10
index b23fdac69820..5e3cdf21556a 100644
@@ -15,19 +15,9 @@ struct nfsd_stats {
 	unsigned int	rcmisses;	/* repcache hits */
 	unsigned int	rcnocache;	/* uncached reqs */
 	unsigned int	fh_stale;	/* FH stale error */
-	unsigned int	fh_lookup;	/* dentry cached */
-	unsigned int	fh_anon;	/* anon file dentry returned */
-	unsigned int	fh_nocache_dir;	/* filehandle not found in dcache */
-	unsigned int	fh_nocache_nondir;	/* filehandle not found in dcache */
 	unsigned int	io_read;	/* bytes returned to read requests */
 	unsigned int	io_write;	/* bytes passed in write requests */
 	unsigned int	th_cnt;		/* number of available threads */
-	unsigned int	th_usage[10];	/* number of ticks during which n perdeciles
-					 * of available threads were in use */
-	unsigned int	th_fullcnt;	/* number of times last free thread was used */
-	unsigned int	ra_size;	/* size of ra cache */
-	unsigned int	ra_depth[11];	/* number of times ra entry was found that deep
-					 * in the cache (10percentiles). [10] = not found */
 #ifdef CONFIG_NFSD_V4
 	unsigned int	nfs4_opcount[LAST_NFS4_OP + 1];	/* count of individual nfsv4 operations */
 #endif
Signature · 1 line
2.17.1
#2 lore raw

PATCH v2 2/3 nfsd: protect concurrent access to nfsd stats counters

In reply to Amir Goldstein

nfsd stats counters can be updated by concurrent nfsd threads without any protection.

Convert some nfsd_stats and nfsd_net struct members to use percpu counters.

The longest_chain* members of struct nfsd_net remain unprotected.

---
 fs/nfsd/netns.h    | 23 +++++++------
 fs/nfsd/nfs4proc.c |  2 +-
 fs/nfsd/nfscache.c | 52 +++++++++++++++++++++---------
 fs/nfsd/nfsctl.c   |  5 ++-
 fs/nfsd/nfsfh.c    |  2 +-
 fs/nfsd/stats.c    | 77 ++++++++++++++++++++++++++++++++++++--------
 fs/nfsd/stats.h    | 80 +++++++++++++++++++++++++++++++++++++++-------
 fs/nfsd/vfs.c      |  4 +--
 8 files changed, 192 insertions(+), 53 deletions(-)
fs/nfsd/netns.h +14 -9
index 7346acda9d76..c330f5bd0cf3 100644
@@ -10,6 +10,7 @@
 
 #include <net/net_namespace.h>
 #include <net/netns/generic.h>
+#include <linux/percpu_counter.h>
 
 /* Hash tables for nfs4_clientid state */
 #define CLIENT_HASH_BITS                 4
@@ -21,6 +22,14 @@
 struct cld_net;
 struct nfsd4_client_tracking_ops;
 
+enum {
+	/* cache misses due only to checksum comparison failures */
+	NFSD_NET_PAYLOAD_MISSES,
+	/* amount of memory (in bytes) currently consumed by the DRC */
+	NFSD_NET_DRC_MEM_USAGE,
+	NFSD_NET_COUNTERS_NUM
+};
+
 /*
  * Represents a nfsd "container". With respect to nfsv4 state tracking, the
  * fields of interest are the *_id_hashtbls and the *_name_tree. These track
@@ -149,20 +158,16 @@ struct nfsd_net {
 
 	/*
 	 * Stats and other tracking of on the duplicate reply cache.
-	 * These fields and the "rc" fields in nfsdstats are modified
-	 * with only the per-bucket cache lock, which isn't really safe
-	 * and should be fixed if we want the statistics to be
-	 * completely accurate.
+	 * The longest_chain* fields are modified with only the per-bucket
+	 * cache lock, which isn't really safe and should be fixed if we want
+	 * these statistics to be completely accurate.
 	 */
 
 	/* total number of entries */
 	atomic_t                 num_drc_entries;
 
-	/* cache misses due only to checksum comparison failures */
-	unsigned int             payload_misses;
-
-	/* amount of memory (in bytes) currently consumed by the DRC */
-	unsigned int             drc_mem_usage;
+	/* Per-netns stats counters */
+	struct percpu_counter    counter[NFSD_NET_COUNTERS_NUM];
 
 	/* longest hash chain seen */
 	unsigned int             longest_chain;
fs/nfsd/nfs4proc.c +1 -1
index 4727b7f03c5b..8e6507104a38 100644
@@ -2169,7 +2169,7 @@ nfsd4_proc_null(struct svc_rqst *rqstp)
 static inline void nfsd4_increment_op_stats(u32 opnum)
 {
 	if (opnum >= FIRST_NFS4_OP && opnum <= LAST_NFS4_OP)
-		nfsdstats.nfs4_opcount[opnum]++;
+		percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_NFS4_OP(opnum)]);
 }
 
 static const struct nfsd4_operation nfsd4_ops[];
fs/nfsd/nfscache.c +37 -15

134 lines not on this page · raw message

fs/nfsd/nfsctl.c +4 -1
index f6d5d783f4a4..258605ee49b8 100644
@@ -1534,7 +1534,9 @@ static int __init init_nfsd(void)
 	retval = nfsd4_init_pnfs();
 	if (retval)
 		goto out_free_slabs;
-	nfsd_stat_init();	/* Statistics */
+	retval = nfsd_stat_init();	/* Statistics */
+	if (retval)
+		goto out_free_pnfs;
 	retval = nfsd_drc_slab_create();
 	if (retval)
 		goto out_free_stat;
@@ -1554,6 +1556,7 @@ static int __init init_nfsd(void)
 	nfsd_drc_slab_free();
 out_free_stat:
 	nfsd_stat_shutdown();
+out_free_pnfs:
 	nfsd4_exit_pnfs();
 out_free_slabs:
 	nfsd4_free_slabs();
fs/nfsd/nfsfh.c +1 -1
index 66f2ef67792a..9e31b2b5c6d2 100644
@@ -422,7 +422,7 @@ fh_verify(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, int access)
 	}
 out:
 	if (error == nfserr_stale)
-		nfsdstats.fh_stale++;
+		nfsd_stats_fh_stale_inc();
 	return error;
 }
 
fs/nfsd/stats.c +64 -13

99 lines not on this page · raw message

fs/nfsd/stats.h +69 -11

96 lines not on this page · raw message

fs/nfsd/vfs.c +2 -2
index 04937e51de56..d560c1bb2ec2 100644
@@ -889,7 +889,7 @@ static __be32 nfsd_finish_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			       unsigned long *count, u32 *eof, ssize_t host_err)
 {
 	if (host_err >= 0) {
-		nfsdstats.io_read += host_err;
+		nfsd_stats_io_read_add(host_err);
 		*eof = nfsd_eof_on_read(file, offset, host_err, *count);
 		*count = host_err;
 		fsnotify_access(file);
@@ -1040,7 +1040,7 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct nfsd_file *nf,
 		goto out_nfserr;
 	}
 	*cnt = host_err;
-	nfsdstats.io_write += *cnt;
+	nfsd_stats_io_write_add(*cnt);
 	fsnotify_modify(file);
 
 	if (stable && use_wgather) {
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v2 2/3] nfsd: protect concurrent access to nfsd stats counters

These three patches seem fine. To bikeshed just a bit more:

fs/nfsd/nfscache.c int nfsd_cache_lookup(struct svc_rqst *rqstp) · line 466

Quoted 3 lines · Amir Goldstein
fs/nfsd/nfscache.c +1 -1
 	/* We found a matching entry which is either in progress or done. */
-	nfsdstats.rchits++;
+	nfsd_stats_rc_hits_inc();

Maybe make that something like

	nfsd_stats_inc(NFSD_STATS_RC_HITS);

and then we could avoid boilerplate like the below?

--b.

Quoted 44 lines
fs/nfsd/stats.h +44 -0
+static inline void nfsd_stats_rc_hits_inc(void)
+{
+	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_HITS]);
+}
+
+static inline void nfsd_stats_rc_misses_inc(void)
+{
+	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_MISSES]);
+}
+
+static inline void nfsd_stats_rc_nocache_inc(void)
+{
+	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_NOCACHE]);
+}
+
+static inline void nfsd_stats_fh_stale_inc(void)
+{
+	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_FH_STALE]);
+}
+
+static inline void nfsd_stats_io_read_add(s64 amount)
+{
+	percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_READ], amount);
+}
+
+static inline void nfsd_stats_io_write_add(s64 amount)
+{
+	percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_WRITE], amount);
+}
+
+static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
+{
+	percpu_counter_inc(&nn->counter[NFSD_NET_PAYLOAD_MISSES]);
+}
+
+static inline void nfsd_stats_drc_mem_usage_add(struct nfsd_net *nn, s64 amount)
+{
+	percpu_counter_add(&nn->counter[NFSD_NET_DRC_MEM_USAGE], amount);
+}
+
+static inline void nfsd_stats_drc_mem_usage_sub(struct nfsd_net *nn, s64 amount)
+{
+	percpu_counter_sub(&nn->counter[NFSD_NET_DRC_MEM_USAGE], amount);
+}
#4 lore raw

Re: [PATCH v2 2/3] nfsd: protect concurrent access to nfsd stats counters

Quoted 8 lines · J . Bruce Fields

These three patches seem fine. To bikeshed just a bit more:

fs/nfsd/nfscache.c int nfsd_cache_lookup(struct svc_rqst *rqstp) · line 466

Quoted 3 lines · Amir Goldstein
      /* We found a matching entry which is either in progress or done. */
fs/nfsd/nfscache.c +1 -1
-     nfsdstats.rchits++;
+     nfsd_stats_rc_hits_inc();
Quoted 6 lines

Maybe make that something like

        nfsd_stats_inc(NFSD_STATS_RC_HITS);

and then we could avoid boilerplate like the below?

It looks like boilerplate, but every helper is a bit different, so we would have to introduce several variants like:

         nfsd_net_stats_inc(nn, NFSD_NET_PAYLOAD_MISSES);

Which is the way I started, but realized it opens a big hole for human mistakes in the form of:

         nfsd_net_stats_inc(nn, NFSD_STATS_RC_HITS);
         nfsd_stats_inc(NFSD_NET_PAYLOAD_MISSES);

And we have no way to detect those errors at compile time because enum types are not strict types.

Also, in the next patch, three more helpers become unique
as they are made into helpers to account for both global and per-export stats (fh_stale, io_read, io_write).
Making those helpers generic would require aligning the start of global stats enum values with the per-export enum values and that is a source of more human errors.

See the end result of stats.h. All the helpers are unique and the only ones that remain generic are nfsd_stats_rc_*.
Making those generic would also require checking the range of the index value is in the NFSD_STATS_RC_* range.

I also tried a version with boilerplate defining macros, i.e.:

NFSD_STATS_FUNCS(rc_hits, RC_HITS)
NFSD_STATS_FUNCS(fh_stale, FH_STALE)
NFSD_STATS_FUNCS(io_read, IO_READ)

But when I realized in the end it can only serve the rc_* counters, I dropped it.

Thoughts?

Thanks,
Amir.

Quoted 46 lines

--b.

Quoted 44 lines
fs/nfsd/stats.h +44 -0
+static inline void nfsd_stats_rc_hits_inc(void)
+{
+     percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_HITS]);
+}
+
+static inline void nfsd_stats_rc_misses_inc(void)
+{
+     percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_MISSES]);
+}
+
+static inline void nfsd_stats_rc_nocache_inc(void)
+{
+     percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_NOCACHE]);
+}
+
+static inline void nfsd_stats_fh_stale_inc(void)
+{
+     percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_FH_STALE]);
+}
+
+static inline void nfsd_stats_io_read_add(s64 amount)
+{
+     percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_READ], amount);
+}
+
+static inline void nfsd_stats_io_write_add(s64 amount)
+{
+     percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_WRITE], amount);
+}
+
+static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
+{
+     percpu_counter_inc(&nn->counter[NFSD_NET_PAYLOAD_MISSES]);
+}
+
+static inline void nfsd_stats_drc_mem_usage_add(struct nfsd_net *nn, s64 amount)
+{
+     percpu_counter_add(&nn->counter[NFSD_NET_DRC_MEM_USAGE], amount);
+}
+
+static inline void nfsd_stats_drc_mem_usage_sub(struct nfsd_net *nn, s64 amount)
+{
+     percpu_counter_sub(&nn->counter[NFSD_NET_DRC_MEM_USAGE], amount);
+}
#5 lore raw

Re: [PATCH v2 2/3] nfsd: protect concurrent access to nfsd stats counters

Quoted 42 lines · Amir Goldstein
Quoted 8 lines · J . Bruce Fields

These three patches seem fine. To bikeshed just a bit more:

fs/nfsd/nfscache.c int nfsd_cache_lookup(struct svc_rqst *rqstp) · line 466

Quoted 3 lines · Amir Goldstein
      /* We found a matching entry which is either in progress or done. */
fs/nfsd/nfscache.c +1 -1
-     nfsdstats.rchits++;
+     nfsd_stats_rc_hits_inc();
Quoted 6 lines

Maybe make that something like

        nfsd_stats_inc(NFSD_STATS_RC_HITS);

and then we could avoid boilerplate like the below?

It looks like boilerplate, but every helper is a bit different, so we would have to introduce several variants like:

         nfsd_net_stats_inc(nn, NFSD_NET_PAYLOAD_MISSES);

Which is the way I started, but realized it opens a big hole for human mistakes in the form of:

         nfsd_net_stats_inc(nn, NFSD_STATS_RC_HITS);
         nfsd_stats_inc(NFSD_NET_PAYLOAD_MISSES);

And we have no way to detect those errors at compile time because enum types are not strict types.

Also, in the next patch, three more helpers become unique
as they are made into helpers to account for both global and per-export stats (fh_stale, io_read, io_write).
Making those helpers generic would require aligning the start of global stats enum values with the per-export enum values and that is a source of more human errors.

See the end result of stats.h. All the helpers are unique and the only ones that remain generic are nfsd_stats_rc_*.
Making those generic would also require checking the range of the index value is in the NFSD_STATS_RC_* range.

Quoted 8 lines

I also tried a version with boilerplate defining macros, i.e.:

NFSD_STATS_FUNCS(rc_hits, RC_HITS)
NFSD_STATS_FUNCS(fh_stale, FH_STALE)
NFSD_STATS_FUNCS(io_read, IO_READ)

But when I realized in the end it can only serve the rc_* counters, I dropped it.

Yeah, OK. I'm not a fan of that kind of macro use, either. It's made me waste time when "grep" doesn't turn up a function definition, for example.

Quoted 1 line

Thoughts?

I've got no clever alternative, and you've tried a lot of alternatives, so I'll trust your judgement. Thanks for the explanation.

--b.

#6 lore raw

PATCH v2 3/3 nfsd: report per-export stats

In reply to Amir Goldstein

Collect some nfsd stats per export in addition to the global stats.

A new nfsdfs export_stats file is created. It uses the same ops as the exports file to iterate the export entries and we use the file's name to determine the reported info per export. For example:

 $ cat /proc/fs/nfsd/export_stats
 # Version 1.1
 # Path Client Start-time
 #	Stats
 /test	localhost	92
	fh_stale: 0
	io_read: 9
	io_write: 1

Every export entry reports the start time when stats collection started, so stats collecting scripts can know if stats where reset between samples.

---
 fs/nfsd/export.c | 68 ++++++++++++++++++++++++++++++++++++++++++------
 fs/nfsd/export.h | 15 +++++++++++
 fs/nfsd/nfsctl.c |  3 +++
 fs/nfsd/nfsd.h   |  2 +-
 fs/nfsd/nfsfh.c  |  4 +--
 fs/nfsd/stats.h  | 12 ++++++---
 fs/nfsd/vfs.c    |  4 +--
 7 files changed, 92 insertions(+), 16 deletions(-)
fs/nfsd/export.c +60 -8
index 81e7bb12aca6..7c863f2c21e0 100644
@@ -331,12 +331,29 @@ static void nfsd4_fslocs_free(struct nfsd4_fs_locations *fsloc)
 	fsloc->locations = NULL;
 }
 
+static int export_stats_init(struct export_stats *stats)
+{
+	stats->start_time = ktime_get_seconds();
+	return nfsd_percpu_counters_init(stats->counter, EXP_STATS_COUNTERS_NUM);
+}
+
+static void export_stats_reset(struct export_stats *stats)
+{
+	nfsd_percpu_counters_reset(stats->counter, EXP_STATS_COUNTERS_NUM);
+}
+
+static void export_stats_destroy(struct export_stats *stats)
+{
+	nfsd_percpu_counters_destroy(stats->counter, EXP_STATS_COUNTERS_NUM);
+}
+
 static void svc_export_put(struct kref *ref)
 {
 	struct svc_export *exp = container_of(ref, struct svc_export, h.ref);
 	path_put(&exp->ex_path);
 	auth_domain_put(exp->ex_client);
 	nfsd4_fslocs_free(&exp->ex_fslocs);
+	export_stats_destroy(&exp->ex_stats);
 	kfree(exp->ex_uuid);
 	kfree_rcu(exp, ex_rcu);
 }
@@ -692,22 +709,47 @@ static void exp_flags(struct seq_file *m, int flag, int fsid,
 		kuid_t anonu, kgid_t anong, struct nfsd4_fs_locations *fslocs);
 static void show_secinfo(struct seq_file *m, struct svc_export *exp);
 
+static int is_export_stats_file(struct seq_file *m)
+{
+	/*
+	 * The export_stats file uses the same ops as the exports file.
+	 * We use the file's name to determine the reported info per export.
+	 * There is no rename in nsfdfs, so d_name.name is stable.
+	 */
+	return !strcmp(m->file->f_path.dentry->d_name.name, "export_stats");
+}
+
 static int svc_export_show(struct seq_file *m,
 			   struct cache_detail *cd,
 			   struct cache_head *h)
 {
-	struct svc_export *exp ;
+	struct svc_export *exp;
+	bool export_stats = is_export_stats_file(m);
 
-	if (h ==NULL) {
-		seq_puts(m, "#path domain(flags)\n");
+	if (h == NULL) {
+		if (export_stats)
+			seq_puts(m, "#path domain start-time\n#\tstats\n");
+		else
+			seq_puts(m, "#path domain(flags)\n");
 		return 0;
 	}
 	exp = container_of(h, struct svc_export, h);
 	seq_path(m, &exp->ex_path, " \t\n\\");
 	seq_putc(m, '\t');
 	seq_escape(m, exp->ex_client->name, " \t\n\\");
+	if (export_stats) {
+		seq_printf(m, "\t%lld\n", exp->ex_stats.start_time);
+		seq_printf(m, "\tfh_stale: %lld\n",
+			   percpu_counter_sum_positive(&exp->ex_stats.counter[EXP_STATS_FH_STALE]));
+		seq_printf(m, "\tio_read: %lld\n",
+			   percpu_counter_sum_positive(&exp->ex_stats.counter[EXP_STATS_IO_READ]));
+		seq_printf(m, "\tio_write: %lld\n",
+			   percpu_counter_sum_positive(&exp->ex_stats.counter[EXP_STATS_IO_WRITE]));
+		seq_putc(m, '\n');
+		return 0;
+	}
 	seq_putc(m, '(');
-	if (test_bit(CACHE_VALID, &h->flags) && 
+	if (test_bit(CACHE_VALID, &h->flags) &&
 	    !test_bit(CACHE_NEGATIVE, &h->flags)) {
 		exp_flags(m, exp->ex_flags, exp->ex_fsid,
 			  exp->ex_anon_uid, exp->ex_anon_gid, &exp->ex_fslocs);
@@ -748,6 +790,7 @@ static void svc_export_init(struct cache_head *cnew, struct cache_head *citem)
 	new->ex_layout_types = 0;
 	new->ex_uuid = NULL;
 	new->cd = item->cd;
+	export_stats_reset(&new->ex_stats);
 }
 
 static void export_update(struct cache_head *cnew, struct cache_head *citem)
@@ -780,10 +823,15 @@ static void export_update(struct cache_head *cnew, struct cache_head *citem)
 static struct cache_head *svc_export_alloc(void)
 {
 	struct svc_export *i = kmalloc(sizeof(*i), GFP_KERNEL);
-	if (i)
-		return &i->h;
-	else
+	if (!i)
+		return NULL;
+
+	if (export_stats_init(&i->ex_stats)) {
+		kfree(i);
 		return NULL;
+	}
+
+	return &i->h;
 }
 
 static const struct cache_detail svc_export_cache_template = {
@@ -1245,10 +1293,14 @@ static int e_show(struct seq_file *m, void *p)
 	struct cache_head *cp = p;
 	struct svc_export *exp = container_of(cp, struct svc_export, h);
 	struct cache_detail *cd = m->private;
+	bool export_stats = is_export_stats_file(m);
 
 	if (p == SEQ_START_TOKEN) {
 		seq_puts(m, "# Version 1.1\n");
-		seq_puts(m, "# Path Client(Flags) # IPs\n");
+		if (export_stats)
+			seq_puts(m, "# Path Client Start-time\n#\tStats\n");
+		else
+			seq_puts(m, "# Path Client(Flags) # IPs\n");
 		return 0;
 	}
 
fs/nfsd/export.h +15 -0
index e7daa1f246f0..ee0e3aba4a6e 100644
@@ -6,6 +6,7 @@
 #define NFSD_EXPORT_H
 
 #include <linux/sunrpc/cache.h>
+#include <linux/percpu_counter.h>
 #include <uapi/linux/nfsd/export.h>
 #include <linux/nfs4.h>
 
@@ -46,6 +47,19 @@ struct exp_flavor_info {
 	u32	flags;
 };
 
+/* Per-export stats */
+enum {
+	EXP_STATS_FH_STALE,
+	EXP_STATS_IO_READ,
+	EXP_STATS_IO_WRITE,
+	EXP_STATS_COUNTERS_NUM
+};
+
+struct export_stats {
+	time64_t		start_time;
+	struct percpu_counter	counter[EXP_STATS_COUNTERS_NUM];
+};
+
 struct svc_export {
 	struct cache_head	h;
 	struct auth_domain *	ex_client;
@@ -62,6 +76,7 @@ struct svc_export {
 	struct nfsd4_deviceid_map *ex_devid_map;
 	struct cache_detail	*cd;
 	struct rcu_head		ex_rcu;
+	struct export_stats	ex_stats;
 };
 
 /* an "export key" (expkey) maps a filehandlefragement to an
fs/nfsd/nfsctl.c +3 -0
index 258605ee49b8..4f6e514192bd 100644
@@ -32,6 +32,7 @@
 enum {
 	NFSD_Root = 1,
 	NFSD_List,
+	NFSD_Export_Stats,
 	NFSD_Export_features,
 	NFSD_Fh,
 	NFSD_FO_UnlockIP,
@@ -1348,6 +1349,8 @@ static int nfsd_fill_super(struct super_block *sb, struct fs_context *fc)
 
 	static const struct tree_descr nfsd_files[] = {
 		[NFSD_List] = {"exports", &exports_nfsd_operations, S_IRUGO},
+		/* Per-export io stats use same ops as exports file */
+		[NFSD_Export_Stats] = {"export_stats", &exports_nfsd_operations, S_IRUGO},
 		[NFSD_Export_features] = {"export_features",
 					&export_features_operations, S_IRUGO},
 		[NFSD_FO_UnlockIP] = {"unlock_ip",
fs/nfsd/nfsd.h +1 -1
index d63cf8196fed..8bdc37aa2c2e 100644
@@ -24,8 +24,8 @@
 #include <uapi/linux/nfsd/debug.h>
 
 #include "netns.h"
-#include "stats.h"
 #include "export.h"
+#include "stats.h"
 
 #undef ifdebug
 #ifdef CONFIG_SUNRPC_DEBUG
fs/nfsd/nfsfh.c +2 -2
index 9e31b2b5c6d2..4744a276058d 100644
@@ -349,7 +349,7 @@ static __be32 nfsd_set_fh_dentry(struct svc_rqst *rqstp, struct svc_fh *fhp)
 __be32
 fh_verify(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, int access)
 {
-	struct svc_export *exp;
+	struct svc_export *exp = NULL;
 	struct dentry	*dentry;
 	__be32		error;
 
@@ -422,7 +422,7 @@ fh_verify(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, int access)
 	}
 out:
 	if (error == nfserr_stale)
-		nfsd_stats_fh_stale_inc();
+		nfsd_stats_fh_stale_inc(exp);
 	return error;
 }
 
fs/nfsd/stats.h +9 -3
index 87c3150c200f..51ecda852e23 100644
@@ -59,19 +59,25 @@ static inline void nfsd_stats_rc_nocache_inc(void)
 	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_RC_NOCACHE]);
 }
 
-static inline void nfsd_stats_fh_stale_inc(void)
+static inline void nfsd_stats_fh_stale_inc(struct svc_export *exp)
 {
 	percpu_counter_inc(&nfsdstats.counter[NFSD_STATS_FH_STALE]);
+	if (exp)
+		percpu_counter_inc(&exp->ex_stats.counter[EXP_STATS_FH_STALE]);
 }
 
-static inline void nfsd_stats_io_read_add(s64 amount)
+static inline void nfsd_stats_io_read_add(struct svc_export *exp, s64 amount)
 {
 	percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_READ], amount);
+	if (exp)
+		percpu_counter_add(&exp->ex_stats.counter[EXP_STATS_IO_READ], amount);
 }
 
-static inline void nfsd_stats_io_write_add(s64 amount)
+static inline void nfsd_stats_io_write_add(struct svc_export *exp, s64 amount)
 {
 	percpu_counter_add(&nfsdstats.counter[NFSD_STATS_IO_WRITE], amount);
+	if (exp)
+		percpu_counter_add(&exp->ex_stats.counter[EXP_STATS_IO_WRITE], amount);
 }
 
 static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
fs/nfsd/vfs.c +2 -2
index d560c1bb2ec2..d316e11923c5 100644
@@ -889,7 +889,7 @@ static __be32 nfsd_finish_read(struct svc_rqst *rqstp, struct svc_fh *fhp,
 			       unsigned long *count, u32 *eof, ssize_t host_err)
 {
 	if (host_err >= 0) {
-		nfsd_stats_io_read_add(host_err);
+		nfsd_stats_io_read_add(fhp->fh_export, host_err);
 		*eof = nfsd_eof_on_read(file, offset, host_err, *count);
 		*count = host_err;
 		fsnotify_access(file);
@@ -1040,7 +1040,7 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct nfsd_file *nf,
 		goto out_nfserr;
 	}
 	*cnt = host_err;
-	nfsd_stats_io_write_add(*cnt);
+	nfsd_stats_io_write_add(exp, *cnt);
 	fsnotify_modify(file);
 
 	if (stable && use_wgather) {
Signature · 1 line
2.17.1
#7 lore raw
In reply to Amir Goldstein

Hello Amir!

Quoted 9 lines · Amir Goldstein

Hi Bruce,

Per your request, I added a cleanup patch for unused counter.

Replaced the hacky counters array "union" with proper array and added helpers to update the counters to avoid human mistakes related to counter indices.

Thanks for your patches. v2 of your series has been committed to the for-next branch at

git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

in preparation for the v5.12 merge window.

Quoted 13 lines

Thanks,
Amir.

Changes since v1:
- Cleanup unused stats counters (Bruce)
- Replace counters array hack with proper array (Chuck)
- Helpers to update both global and per-export stats

Amir Goldstein (3):
nfsd: remove unused stats counters
nfsd: protect concurrent access to nfsd stats counters
nfsd: report per-export stats

Quoted 16 lines
fs/nfsd/export.c   |  68 +++++++++++++++++++++++----
fs/nfsd/export.h   |  15 ++++++
fs/nfsd/netns.h    |  23 +++++----
fs/nfsd/nfs4proc.c |   2 +-
fs/nfsd/nfscache.c |  52 +++++++++++++++------
fs/nfsd/nfsctl.c   |   8 +++-
fs/nfsd/nfsd.h     |   2 +-
fs/nfsd/nfsfh.c    |   4 +-
fs/nfsd/stats.c    | 114 +++++++++++++++++++++++++++++++--------------
fs/nfsd/stats.h    |  96 +++++++++++++++++++++++++++++---------
fs/nfsd/vfs.c      |   4 +-
11 files changed, 292 insertions(+), 96 deletions(-)
Signature · 1 line
2.17.1

--
Chuck Lever

#8 lore raw
Quoted 13 lines · Chuck Lever

Hello Amir!

Quoted 9 lines · Amir Goldstein

Hi Bruce,

Per your request, I added a cleanup patch for unused counter.

Replaced the hacky counters array "union" with proper array and added helpers to update the counters to avoid human mistakes related to counter indices.

Quoted 7 lines

Thanks for your patches. v2 of your series has been committed to the for-next branch at

git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git

in preparation for the v5.12 merge window.

Great!

Thanks for the notice,
Amir.