Skip to contentSkip to conversation

page_pool: Add page_pool stat counters

PATCH Superseded Joe Damato 2 participants 12 messages Jan 29, 2022 – Jan 30, 2022

Subsystems (3)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Joe Damato

author
#0 lore raw

v2 cover page_pool: Add page_pool stat counters

Greetings:

This series adds some stat counters for the page_pool allocation path which help to track:

- fast path allocations
- slow path order-0 allocations
- slow path high order allocations
- refills which failed due to an empty ptr ring, forcing a slow path allocation
- allocations fulfilled via successful refill
- pages which cannot be added to the cache because of numa mismatch (i.e. waived)

This v2 series includes some major changes from the original, namely:

1. A new kernel config option has been added, which defaults to N, preventing this code from being compiled in by default
2. The stats structure has been converted to a per-cpu structure
3. The stats are now exported via proc (/proc/net/page_pool_stat)

The main advantage of the v2 over the original approach is that no modifications to drivers are required and no new external APIs are introduced.

I benchmarked the code with the stats enabled and again with them disabled using the netoptimizer/prototype-kernel benchmark programs [1], as Jesper suggested.

Test system:
- 2x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
- 2 NUMA zones, with 18 cores per zone and 2 threads per core

bench_page_pool_simple results:
test name stats enabled stats disabled
cycles nanosec cycles nanosec

for_loop 0 0.337 0 0.334
atomic_inc 13 6.021 13 6.022
lock 31 13.545 31 13.846

no-softirq-page_pool01 45 20.040 44 19.388
no-softirq-page_pool02 46 20.295 43 19.073
no-softirq-page_pool03 110 48.053 122 53.405

tasklet_page_pool01_fast_path 14 6.126 12 5.640
tasklet_page_pool02_ptr_ring 42 18.334 40 17.695
tasklet_page_pool03_slow 109 47.854 108 47.355

bench_page_pool_cross_cpu results:
test name stats enabled stats disabled
cycles nanosec cycles nanosec

page_pool_cross_cpu CPU(0) 2246 979.007 2136 931.075
page_pool_cross_cpu CPU(1) 2240 976.289 2145 934.924
page_pool_cross_cpu CPU(2) 1123 489.511 1072 467.474

page_pool_cross_cpu average 1886 - 1784 -

Thanks.

[1]:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/lib

Joe Damato (10):
page_pool: kconfig: Add flag for page pool stats
page_pool: Add per-cpu page_pool_stats struct
page_pool: Add a macro for incrementing stats
page_pool: Add stat tracking fast path allocations
page_pool: Add slow path order 0 allocation stat
page_pool: Add slow path high order allocation stat
page_pool: Add stat tracking empty ring
page_pool: Add stat tracking cache refill
page_pool: Add a stat tracking waived pages
net-procfs: Show page pool stats in proc

 include/net/page_pool.h | 20 ++++++++++++++
 net/Kconfig             | 12 +++++++++
 net/core/net-procfs.c   | 69 ++++++++++++++++++++++++++++++++++++++++++++++++-
 net/core/page_pool.c    | 28 +++++++++++++++++---
 4 files changed, 125 insertions(+), 4 deletions(-)
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH v2 1/10 page_pool: kconfig: Add flag for page pool stats

Control enabling / disabling page_pool_stats with a kernel config option. Option is defaulted to N.

---
 net/Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)
net/Kconfig +12 -0
index 8a1f9d0..feeca42 100644
@@ -434,6 +434,18 @@ config NET_DEVLINK
 config PAGE_POOL
 	bool
 
+config PAGE_POOL_STATS
+	default n
+	bool "Page pool stats"
+	depends on PAGE_POOL
+	help
+	  Enable page pool statistics to track allocations. Stats are exported
+	  to the file /proc/net/page_pool_stat. Users can examine these
+	  stats to better understand how their drivers and the kernel's
+	  page allocator, and the page pool interact with each other.
+
+	  If unsure, say N.
+
 config FAILOVER
 	tristate "Generic failover module"
 	help
Signature · 1 line
2.7.4
author
#2 lore raw

PATCH v2 2/10 page_pool: Add per-cpu page_pool_stats struct

In reply to Joe Damato

A per-cpu (empty) page_pool_stats struct has been added as a place holder.

---
 include/net/page_pool.h | 10 ++++++++++
 net/core/page_pool.c    |  5 +++++
 2 files changed, 15 insertions(+)
include/net/page_pool.h +10 -0
index 79a8055..dae65f2 100644
@@ -137,6 +137,16 @@ struct page_pool {
 	u64 destroy_cnt;
 };
 
+#ifdef CONFIG_PAGE_POOL_STATS
+/*
+ * stats for tracking page_pool events.
+ */
+struct page_pool_stats {
+};
+
+DECLARE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);
+#endif
+
 struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
 
 static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
net/core/page_pool.c +5 -0
index bd62c01..7e33590 100644
@@ -26,6 +26,11 @@
 
 #define BIAS_MAX	LONG_MAX
 
+#ifdef CONFIG_PAGE_POOL_STATS
+DEFINE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);
+EXPORT_PER_CPU_SYMBOL(page_pool_stats);
+#endif
+
 static int page_pool_init(struct page_pool *pool,
 			  const struct page_pool_params *params)
 {
Signature · 1 line
2.7.4
author
#3 lore raw

PATCH v2 3/10 page_pool: Add a macro for incrementing stats

In reply to Joe Damato

Add simple wrapper macro for incrementing page pool stats. This wrapper is intended to be used in softirq context.

---
 net/core/page_pool.c | 9 +++++++++
 1 file changed, 9 insertions(+)
net/core/page_pool.c +9 -0
index 7e33590..b1a2599 100644
@@ -29,6 +29,15 @@
 #ifdef CONFIG_PAGE_POOL_STATS
 DEFINE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);
 EXPORT_PER_CPU_SYMBOL(page_pool_stats);
+
+#define page_pool_stat_alloc_inc(__stat)					\
+	do {									\
+		struct page_pool_stats *pps = this_cpu_ptr(&page_pool_stats);	\
+		pps->alloc.__stat++;						\
+	} while (0)
+
+#else
+#define page_pool_stat_alloc_inc(stat)
 #endif
 
 static int page_pool_init(struct page_pool *pool,
Signature · 1 line
2.7.4
author
#4 lore raw

PATCH v2 4/10 page_pool: Add stat tracking fast path allocations

In reply to Joe Damato

Add a counter to track successful fast-path allocations.

---
 include/net/page_pool.h | 3 +++
 net/core/page_pool.c    | 1 +
 2 files changed, 4 insertions(+)
include/net/page_pool.h +3 -0
index dae65f2..96949ad 100644
@@ -142,6 +142,9 @@ struct page_pool {
  * stats for tracking page_pool events.
  */
 struct page_pool_stats {
+	struct {
+		u64 fast; /* fast path allocations */
+	} alloc;
 };
 
 DECLARE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);
net/core/page_pool.c +1 -0
index b1a2599..6f692d9 100644
@@ -180,6 +180,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)
 	if (likely(pool->alloc.count)) {
 		/* Fast-path */
 		page = pool->alloc.cache[--pool->alloc.count];
+		page_pool_stat_alloc_inc(fast);
 	} else {
 		page = page_pool_refill_alloc_cache(pool);
 	}
Signature · 1 line
2.7.4
author
#5 lore raw

PATCH v2 5/10 page_pool: Add slow path order 0 allocation stat

In reply to Joe Damato

Track order 0 allocations in the slow path which cause an interaction with the buddy allocator.

---
 include/net/page_pool.h | 1 +
 net/core/page_pool.c    | 6 ++++--
 2 files changed, 5 insertions(+), 2 deletions(-)
include/net/page_pool.h +1 -0
index 96949ad..ab67e86 100644
@@ -144,6 +144,7 @@ struct page_pool {
 struct page_pool_stats {
 	struct {
 		u64 fast; /* fast path allocations */
+		u64 slow; /* slow-path order-0 allocations */
 	} alloc;
 };
 
net/core/page_pool.c +4 -2
index 6f692d9..554a40e 100644
@@ -308,10 +308,12 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
 	}
 
 	/* Return last page */
-	if (likely(pool->alloc.count > 0))
+	if (likely(pool->alloc.count > 0)) {
 		page = pool->alloc.cache[--pool->alloc.count];
-	else
+		page_pool_stat_alloc_inc(slow);
+	} else {
 		page = NULL;
+	}
 
 	/* When page just alloc'ed is should/must have refcnt 1. */
 	return page;
Signature · 1 line
2.7.4
author
#6 lore raw

PATCH v2 6/10 page_pool: Add slow path high order allocation stat

In reply to Joe Damato

Track high order allocations in the slow path which cause an interaction with the buddy allocator.

---
 include/net/page_pool.h | 1 +
 net/core/page_pool.c    | 1 +
 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
index ab67e86..f59b8a9 100644
@@ -145,6 +145,7 @@ struct page_pool_stats {
 	struct {
 		u64 fast; /* fast path allocations */
 		u64 slow; /* slow-path order-0 allocations */
+		u64 slow_high_order; /* slow-path high order allocations */
 	} alloc;
 };
 
net/core/page_pool.c +1 -0
index 554a40e..24306d6 100644
@@ -254,6 +254,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
 		return NULL;
 	}
 
+	page_pool_stat_alloc_inc(slow_high_order);
 	page_pool_set_pp_info(pool, page);
 
 	/* Track how many pages are held 'in-flight' */
Signature · 1 line
2.7.4
author
#7 lore raw

PATCH v2 7/10 page_pool: Add stat tracking empty ring

In reply to Joe Damato

Add a stat tracking when the ptr ring is empty. When this occurs, the cache could not be refilled and a slow path allocation was forced.

---
 include/net/page_pool.h | 3 +++
 net/core/page_pool.c    | 4 +++-
 2 files changed, 6 insertions(+), 1 deletion(-)
include/net/page_pool.h +3 -0
index f59b8a9..ed2bc73 100644
@@ -146,6 +146,9 @@ struct page_pool_stats {
 		u64 fast; /* fast path allocations */
 		u64 slow; /* slow-path order-0 allocations */
 		u64 slow_high_order; /* slow-path high order allocations */
+		u64 empty; /* failed refills due to empty ptr ring, forcing
+			    * slow path allocation
+			    */
 	} alloc;
 };
 
net/core/page_pool.c +3 -1
index 24306d6..9d20b12 100644
@@ -131,8 +131,10 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)
 	int pref_nid; /* preferred NUMA node */
 
 	/* Quicker fallback, avoid locks when ring is empty */
-	if (__ptr_ring_empty(r))
+	if (__ptr_ring_empty(r)) {
+		page_pool_stat_alloc_inc(empty);
 		return NULL;
+	}
 
 	/* Softirq guarantee CPU and thus NUMA node is stable. This,
 	 * assumes CPU refilling driver RX-ring will also run RX-NAPI.
Signature · 1 line
2.7.4
author
#8 lore raw

PATCH v2 8/10 page_pool: Add stat tracking cache refill

In reply to Joe Damato

Add a stat tracking succesfull allocations which triggered a refill.

---
 include/net/page_pool.h | 1 +
 net/core/page_pool.c    | 1 +
 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
index ed2bc73..4991109 100644
@@ -149,6 +149,7 @@ struct page_pool_stats {
 		u64 empty; /* failed refills due to empty ptr ring, forcing
 			    * slow path allocation
 			    */
+		u64 refill; /* allocations via successful refill */
 	} alloc;
 };
 
net/core/page_pool.c +1 -0
index 9d20b12..ffb68b8 100644
@@ -185,6 +185,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)
 		page_pool_stat_alloc_inc(fast);
 	} else {
 		page = page_pool_refill_alloc_cache(pool);
+		page_pool_stat_alloc_inc(refill);
 	}
 
 	return page;
Signature · 1 line
2.7.4
author
#9 lore raw

PATCH v2 9/10 page_pool: Add a stat tracking waived pages

In reply to Joe Damato

Track how often pages obtained from the ring cannot be added to the cache because of a NUMA mismatch.

---
 include/net/page_pool.h | 1 +
 net/core/page_pool.c    | 1 +
 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
index 4991109..e411ef6 100644
@@ -150,6 +150,7 @@ struct page_pool_stats {
 			    * slow path allocation
 			    */
 		u64 refill; /* allocations via successful refill */
+		u64 waive;  /* failed refills due to numa zone mismatch */
 	} alloc;
 };
 
net/core/page_pool.c +1 -0
index ffb68b8..c6f31c5 100644
@@ -161,6 +161,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)
 			 * This limit stress on page buddy alloactor.
 			 */
 			page_pool_return_page(pool, page);
+			page_pool_stat_alloc_inc(waive);
 			page = NULL;
 			break;
 		}
Signature · 1 line
2.7.4
author
#10 lore raw

PATCH v2 10/10 net-procfs: Show page pool stats in proc

In reply to Joe Damato

Per-cpu page pool allocation stats are exported in the file /proc/net/page_pool_stat allowing users to better understand the interaction between their drivers and kernel memory allocation.

---
 net/core/net-procfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 68 insertions(+), 1 deletion(-)
net/core/net-procfs.c +68 -1
index 88cc0ad..3d3f3e8 100644
@@ -4,6 +4,10 @@
 #include <linux/seq_file.h>
 #include <net/wext.h>
 
+#ifdef CONFIG_PAGE_POOL_STATS
+#include <net/page_pool.h>
+#endif
+
 #define BUCKET_SPACE (32 - NETDEV_HASHBITS - 1)
 
 #define get_bucket(x) ((x) >> BUCKET_SPACE)
@@ -310,6 +314,58 @@ static const struct seq_operations ptype_seq_ops = {
 	.show  = ptype_seq_show,
 };
 
+#ifdef CONFIG_PAGE_POOL_STATS
+static struct page_pool_stats *page_pool_stat_get_online(loff_t *pos)
+{
+	struct page_pool_stats *pp_stat = NULL;
+
+	while (*pos < nr_cpu_ids) {
+		if (cpu_online(*pos)) {
+			pp_stat = per_cpu_ptr(&page_pool_stats, *pos);
+			break;
+		}
+
+		++*pos;
+	}
+
+	return pp_stat;
+}
+
+static void *page_pool_seq_start(struct seq_file *seq, loff_t *pos)
+{
+	return page_pool_stat_get_online(pos);
+}
+
+static void *page_pool_seq_next(struct seq_file *seq, void *v, loff_t *pos)
+{
+	++*pos;
+	return page_pool_stat_get_online(pos);
+}
+
+static void page_pool_seq_stop(struct seq_file *seq, void *v)
+{
+}
+
+static int page_pool_seq_show(struct seq_file *seq, void *v)
+{
+	struct page_pool_stats *pp_stat = v;
+
+	seq_printf(seq, "%08llx %08llx %08llx %08llx %08llx %08llx %08llx %08llx\n",
+		   seq->index, pp_stat->alloc.fast,
+		   pp_stat->alloc.slow, pp_stat->alloc.slow_high_order,
+		   pp_stat->alloc.empty, pp_stat->alloc.refill,
+		   pp_stat->alloc.refill, pp_stat->alloc.waive);
+	return 0;
+}
+
+static const struct seq_operations page_pool_seq_ops = {
+	.start = page_pool_seq_start,
+	.next = page_pool_seq_next,
+	.stop = page_pool_seq_stop,
+	.show = page_pool_seq_show,
+};
+#endif
+
 static int __net_init dev_proc_net_init(struct net *net)
 {
 	int rc = -ENOMEM;
@@ -323,12 +379,23 @@ static int __net_init dev_proc_net_init(struct net *net)
 	if (!proc_create_net("ptype", 0444, net->proc_net, &ptype_seq_ops,
 			sizeof(struct seq_net_private)))
 		goto out_softnet;
+#ifdef CONFIG_PAGE_POOL_STATS
+	if (!proc_create_seq("page_pool_stat", 0444, net->proc_net,
+			     &page_pool_seq_ops))
+		goto out_ptype;
+#endif
 
 	if (wext_proc_init(net))
-		goto out_ptype;
+		goto out_page_pool;
 	rc = 0;
 out:
 	return rc;
+
+out_page_pool:
+#ifdef CONFIG_PAGE_POOL_STATS
+	remove_proc_entry("page_pool_stat", net->proc_net);
+#endif
+
 out_ptype:
 	remove_proc_entry("ptype", net->proc_net);
 out_softnet:
Signature · 1 line
2.7.4
#11 lore raw

Re: [net-next v2 10/10] net-procfs: Show page pool stats in proc

Thank you for the patch! Perhaps something to improve:

Hi Joe,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on net-next/master]

url:    https://github.com/0day-ci/linux/commits/Joe-Damato/page_pool-Add-page_pool-stat-counters/20220130-074147
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git ff58831fa02deb42fd731f830d8d9ec545573c7c
config: arm-randconfig-r016-20220130 (https://download.01.org/0day-ci/archive/20220130/202201300928.UoQgv8PS-lkp(a)intel.com/config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 33b45ee44b1f32ffdbc995e6fec806271b4b3ba4)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/0day-ci/linux/commit/5f0fd838269d51e7a97662eaf54868a248b8bd42
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Joe-Damato/page_pool-Add-page_pool-stat-counters/20220130-074147 git checkout 5f0fd838269d51e7a97662eaf54868a248b8bd42
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash net/core/

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> net/core/net-procfs.c:399:1: warning: unused label 'out_ptype' [-Wunused-label]

out_ptype:
^~~~~~~~~~
1 warning generated.

vim +/out_ptype +399 net/core/net-procfs.c

5f0fd838269d51 Joe Damato 2022-01-29 398

900ff8c6321418 Cong Wang  2013-02-18 @399  out_ptype:
900ff8c6321418 Cong Wang  2013-02-18  400  	remove_proc_entry("ptype", net->proc_net);
900ff8c6321418 Cong Wang  2013-02-18  401  out_softnet:
900ff8c6321418 Cong Wang  2013-02-18  402  	remove_proc_entry("softnet_stat", net->proc_net);
900ff8c6321418 Cong Wang  2013-02-18  403  out_dev:
900ff8c6321418 Cong Wang  2013-02-18  404  	remove_proc_entry("dev", net->proc_net);
900ff8c6321418 Cong Wang  2013-02-18  405  	goto out;
900ff8c6321418 Cong Wang  2013-02-18  406  }
900ff8c6321418 Cong Wang  2013-02-18  407  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org