page_pool: Add page_pool stat counters
Subsystems (3)
v2 cover page_pool: Add page_pool stat counters
Greetings:
This series adds some stat counters for the page_pool allocation path which help to track:
- fast path allocations
- slow path order-0 allocations
- slow path high order allocations
- refills which failed due to an empty ptr ring, forcing a slow path allocation
- allocations fulfilled via successful refill
- pages which cannot be added to the cache because of numa mismatch (i.e. waived)
This v2 series includes some major changes from the original, namely:
1. A new kernel config option has been added, which defaults to N, preventing this code from being compiled in by default
2. The stats structure has been converted to a per-cpu structure
3. The stats are now exported via proc (/proc/net/page_pool_stat)
The main advantage of the v2 over the original approach is that no modifications to drivers are required and no new external APIs are introduced.
I benchmarked the code with the stats enabled and again with them disabled using the netoptimizer/prototype-kernel benchmark programs [1], as Jesper suggested.
Test system:
- 2x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
- 2 NUMA zones, with 18 cores per zone and 2 threads per core
bench_page_pool_simple results:
test name stats enabled stats disabled
cycles nanosec cycles nanosec
for_loop 0 0.337 0 0.334atomic_inc 13 6.021 13 6.022
lock 31 13.545 31 13.846
no-softirq-page_pool01 45 20.040 44 19.388
no-softirq-page_pool02 46 20.295 43 19.073
no-softirq-page_pool03 110 48.053 122 53.405
tasklet_page_pool01_fast_path 14 6.126 12 5.640tasklet_page_pool02_ptr_ring 42 18.334 40 17.695tasklet_page_pool03_slow 109 47.854 108 47.355
bench_page_pool_cross_cpu results:
test name stats enabled stats disabled
cycles nanosec cycles nanosec
page_pool_cross_cpu CPU(0) 2246 979.007 2136 931.075page_pool_cross_cpu CPU(1) 2240 976.289 2145 934.924page_pool_cross_cpu CPU(2) 1123 489.511 1072 467.474
page_pool_cross_cpu average 1886 - 1784 -
Thanks.
[1]:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/lib
Joe Damato (10):
page_pool: kconfig: Add flag for page pool stats
page_pool: Add per-cpu page_pool_stats struct
page_pool: Add a macro for incrementing stats
page_pool: Add stat tracking fast path allocations
page_pool: Add slow path order 0 allocation stat
page_pool: Add slow path high order allocation stat
page_pool: Add stat tracking empty ring
page_pool: Add stat tracking cache refill
page_pool: Add a stat tracking waived pages
net-procfs: Show page pool stats in proc
include/net/page_pool.h | 20 ++++++++++++++ net/Kconfig | 12 +++++++++ net/core/net-procfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++- net/core/page_pool.c | 28 +++++++++++++++++--- 4 files changed, 125 insertions(+), 4 deletions(-)
Signature · 1 line
2.7.4
PATCH v2 1/10 page_pool: kconfig: Add flag for page pool stats
Control enabling / disabling page_pool_stats with a kernel config option. Option is defaulted to N.
--- net/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+)
net/Kconfig +12 -0
@@ -434,6 +434,18 @@ config NET_DEVLINK config PAGE_POOL bool +config PAGE_POOL_STATS + default n + bool "Page pool stats" + depends on PAGE_POOL + help + Enable page pool statistics to track allocations. Stats are exported + to the file /proc/net/page_pool_stat. Users can examine these + stats to better understand how their drivers and the kernel's + page allocator, and the page pool interact with each other. + + If unsure, say N. + config FAILOVER tristate "Generic failover module" help
Signature · 1 line
2.7.4
PATCH v2 2/10 page_pool: Add per-cpu page_pool_stats struct
In reply to Joe DamatoA per-cpu (empty) page_pool_stats struct has been added as a place holder.
--- include/net/page_pool.h | 10 ++++++++++ net/core/page_pool.c | 5 +++++ 2 files changed, 15 insertions(+)
include/net/page_pool.h +10 -0
@@ -137,6 +137,16 @@ struct page_pool {u64 destroy_cnt; }; +#ifdef CONFIG_PAGE_POOL_STATS +/* +* stats for tracking page_pool events. +*/ +struct page_pool_stats { +}; + +DECLARE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats); +#endif +struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);static inline struct page *page_pool_dev_alloc_pages (struct page_pool *pool)
net/core/page_pool.c +5 -0
@@ -26,6 +26,11 @@#define BIAS_MAX LONG_MAX +#ifdef CONFIG_PAGE_POOL_STATS +DEFINE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats); +EXPORT_PER_CPU_SYMBOL (page_pool_stats); +#endif +static int page_pool_init (struct page_pool *pool,const struct page_pool_params *params) {
Signature · 1 line
2.7.4
PATCH v2 3/10 page_pool: Add a macro for incrementing stats
In reply to Joe DamatoAdd simple wrapper macro for incrementing page pool stats. This wrapper is intended to be used in softirq context.
--- net/core/page_pool.c | 9 +++++++++ 1 file changed, 9 insertions(+)
net/core/page_pool.c +9 -0
@@ -29,6 +29,15 @@#ifdef CONFIG_PAGE_POOL_STATS DEFINE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);EXPORT_PER_CPU_SYMBOL (page_pool_stats); + +#define page_pool_stat_alloc_inc (__stat) \ + do { \ + struct page_pool_stats *pps = this_cpu_ptr(&page_pool_stats); \ + pps->alloc.__stat++; \ + } while (0) + +#else +#define page_pool_stat_alloc_inc (stat)#endif static int page_pool_init(struct page_pool *pool,
Signature · 1 line
2.7.4
PATCH v2 4/10 page_pool: Add stat tracking fast path allocations
In reply to Joe DamatoAdd a counter to track successful fast-path allocations.
--- include/net/page_pool.h | 3 +++ net/core/page_pool.c | 1 + 2 files changed, 4 insertions(+)
include/net/page_pool.h +3 -0
@@ -142,6 +142,9 @@ struct page_pool {* stats for tracking page_pool events. */ struct page_pool_stats { +struct { +u64 fast;/* fast path allocations */ + } alloc; }; DECLARE_PER_CPU_ALIGNED(struct page_pool_stats, page_pool_stats);
net/core/page_pool.c +1 -0
@@ -180,6 +180,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)if (likely (pool->alloc.count)) {/* Fast-path */ page = pool->alloc.cache[--pool->alloc.count]; +page_pool_stat_alloc_inc (fast); }else { page =page_pool_refill_alloc_cache (pool); }
Signature · 1 line
2.7.4
PATCH v2 5/10 page_pool: Add slow path order 0 allocation stat
In reply to Joe DamatoTrack order 0 allocations in the slow path which cause an interaction with the buddy allocator.
--- include/net/page_pool.h | 1 + net/core/page_pool.c | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-)
include/net/page_pool.h +1 -0
@@ -144,6 +144,7 @@ struct page_pool {struct page_pool_stats {struct {u64 fast;/* fast path allocations */ +u64 slow;/* slow-path order-0 allocations */ } alloc; };
net/core/page_pool.c +4 -2
@@ -308,10 +308,12 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, }/* Return last page */ -if (likely (pool->alloc.count >0 )) +if (likely (pool->alloc.count >0 )) { page = pool->alloc.cache[--pool->alloc.count]; -else +page_pool_stat_alloc_inc (slow); + }else { page =NULL ; + }/* When page just alloc'ed is should/must have refcnt 1. */ return page;
Signature · 1 line
2.7.4
PATCH v2 6/10 page_pool: Add slow path high order allocation stat
In reply to Joe DamatoTrack high order allocations in the slow path which cause an interaction with the buddy allocator.
--- include/net/page_pool.h | 1 + net/core/page_pool.c | 1 + 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
@@ -145,6 +145,7 @@ struct page_pool_stats {struct {u64 fast;/* fast path allocations */ u64 slow;/* slow-path order-0 allocations */ +u64 slow_high_order;/* slow-path high order allocations */ } alloc; };
net/core/page_pool.c +1 -0
@@ -254,6 +254,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool,return NULL ; } +page_pool_stat_alloc_inc (slow_high_order);page_pool_set_pp_info (pool, page);/* Track how many pages are held 'in-flight' */
Signature · 1 line
2.7.4
PATCH v2 7/10 page_pool: Add stat tracking empty ring
In reply to Joe DamatoAdd a stat tracking when the ptr ring is empty. When this occurs, the cache could not be refilled and a slow path allocation was forced.
--- include/net/page_pool.h | 3 +++ net/core/page_pool.c | 4 +++- 2 files changed, 6 insertions(+), 1 deletion(-)
include/net/page_pool.h +3 -0
@@ -146,6 +146,9 @@ struct page_pool_stats {u64 fast;/* fast path allocations */ u64 slow;/* slow-path order-0 allocations */ u64 slow_high_order;/* slow-path high order allocations */ +u64 empty;/* failed refills due to empty ptr ring, forcing +* slow path allocation +*/ } alloc; };
net/core/page_pool.c +3 -1
@@ -131,8 +131,10 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)int pref_nid;/* preferred NUMA node */ /* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) +if (__ptr_ring_empty (r)) { +page_pool_stat_alloc_inc (empty);return NULL ; + }/* Softirq guarantee CPU and thus NUMA node is stable. This, * assumes CPU refilling driver RX-ring will also run RX-NAPI.
Signature · 1 line
2.7.4
PATCH v2 8/10 page_pool: Add stat tracking cache refill
In reply to Joe DamatoAdd a stat tracking succesfull allocations which triggered a refill.
--- include/net/page_pool.h | 1 + net/core/page_pool.c | 1 + 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
@@ -149,6 +149,7 @@ struct page_pool_stats {u64 empty;/* failed refills due to empty ptr ring, forcing * slow path allocation */ +u64 refill;/* allocations via successful refill */ } alloc; };
net/core/page_pool.c +1 -0
@@ -185,6 +185,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)page_pool_stat_alloc_inc (fast); }else { page =page_pool_refill_alloc_cache (pool); +page_pool_stat_alloc_inc (refill); }return page;
Signature · 1 line
2.7.4
PATCH v2 9/10 page_pool: Add a stat tracking waived pages
In reply to Joe DamatoTrack how often pages obtained from the ring cannot be added to the cache because of a NUMA mismatch.
--- include/net/page_pool.h | 1 + net/core/page_pool.c | 1 + 2 files changed, 2 insertions(+)
include/net/page_pool.h +1 -0
@@ -150,6 +150,7 @@ struct page_pool_stats {* slow path allocation */ u64 refill;/* allocations via successful refill */ +u64 waive;/* failed refills due to numa zone mismatch */ } alloc; };
net/core/page_pool.c +1 -0
@@ -161,6 +161,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)* This limit stress on page buddy alloactor. */ page_pool_return_page (pool, page); +page_pool_stat_alloc_inc (waive); page =NULL ;break ; }
Signature · 1 line
2.7.4
PATCH v2 10/10 net-procfs: Show page pool stats in proc
In reply to Joe DamatoPer-cpu page pool allocation stats are exported in the file /proc/net/page_pool_stat allowing users to better understand the interaction between their drivers and kernel memory allocation.
--- net/core/net-procfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-)
net/core/net-procfs.c +68 -1
@@ -4,6 +4,10 @@#include <linux/seq_file.h> #include <net/wext.h> +#ifdef CONFIG_PAGE_POOL_STATS +#include <net/page_pool.h> +#endif +#define BUCKET_SPACE (32 - NETDEV_HASHBITS - 1)#define get_bucket (x) ((x) >> BUCKET_SPACE)
@@ -310,6 +314,58 @@ static const struct seq_operations ptype_seq_ops = { .show = ptype_seq_show, }; +#ifdef CONFIG_PAGE_POOL_STATS +static struct page_pool_stats *page_pool_stat_get_online (loff_t *pos) +{ +struct page_pool_stats *pp_stat =NULL ; + +while (*pos < nr_cpu_ids) { +if (cpu_online (*pos)) { + pp_stat =per_cpu_ptr (&page_pool_stats, *pos); +break ; + } + + ++*pos; + } + +return pp_stat; +} + +static void *page_pool_seq_start (struct seq_file *seq,loff_t *pos) +{ +return page_pool_stat_get_online (pos); +} + +static void *page_pool_seq_next (struct seq_file *seq,void *v,loff_t *pos) +{ + ++*pos; +return page_pool_stat_get_online (pos); +} + +static void page_pool_seq_stop (struct seq_file *seq,void *v) +{ +} + +static int page_pool_seq_show (struct seq_file *seq,void *v) +{ +struct page_pool_stats *pp_stat = v; + +seq_printf (seq,"%08llx %08llx %08llx %08llx %08llx %08llx %08llx %08llx\n" , + seq->index, pp_stat->alloc.fast, + pp_stat->alloc.slow, pp_stat->alloc.slow_high_order, + pp_stat->alloc.empty, pp_stat->alloc.refill, + pp_stat->alloc.refill, pp_stat->alloc.waive); +return 0 ; +} + +static const struct seq_operations page_pool_seq_ops = { + .start = page_pool_seq_start, + .next = page_pool_seq_next, + .stop = page_pool_seq_stop, + .show = page_pool_seq_show, +}; +#endif +static int __net_init dev_proc_net_init (struct net *net) {int rc = -ENOMEM ;
@@ -323,12 +379,23 @@ static int __net_init dev_proc_net_init(struct net *net)if (!proc_create_net ("ptype" ,0444 , net->proc_net, &ptype_seq_ops,sizeof (struct seq_net_private ))) goto out_softnet; +#ifdef CONFIG_PAGE_POOL_STATS +if (!proc_create_seq ("page_pool_stat" ,0444 , net->proc_net, + &page_pool_seq_ops)) + goto out_ptype; +#endif if (wext_proc_init (net)) - goto out_ptype; + goto out_page_pool; rc =0 ; out:return rc; + +out_page_pool: +#ifdef CONFIG_PAGE_POOL_STATS +remove_proc_entry ("page_pool_stat" , net->proc_net); +#endif + out_ptype:remove_proc_entry ("ptype" , net->proc_net); out_softnet:
Signature · 1 line
2.7.4
kernel test robot
build botRe: [net-next v2 10/10] net-procfs: Show page pool stats in proc
Thank you for the patch! Perhaps something to improve:
Hi Joe,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on net-next/master]
url: https://github.com/0day-ci/linux/commits/Joe-Damato/page_pool-Add-page_pool-stat-counters/20220130-074147 base: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git ff58831fa02deb42fd731f830d8d9ec545573c7c config: arm-randconfig-r016-20220130 (https://download.01.org/0day-ci/archive/20220130/202201300928.UoQgv8PS-lkp(a)intel.com/config) compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 33b45ee44b1f32ffdbc995e6fec806271b4b3ba4)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install arm cross compiling tool for clang build
# apt-get install binutils-arm-linux-gnueabi
# https://github.com/0day-ci/linux/commit/5f0fd838269d51e7a97662eaf54868a248b8bd42
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Joe-Damato/page_pool-Add-page_pool-stat-counters/20220130-074147 git checkout 5f0fd838269d51e7a97662eaf54868a248b8bd42
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=arm SHELL=/bin/bash net/core/
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
>> net/core/net-procfs.c:399:1: warning: unused label 'out_ptype' [-Wunused-label]
out_ptype:
^~~~~~~~~~
1 warning generated.
vim +/out_ptype +399 net/core/net-procfs.c
5f0fd838269d51 Joe Damato 2022-01-29 398
900ff8c6321418 Cong Wang 2013-02-18 @399 out_ptype:
900ff8c6321418 Cong Wang 2013-02-18 400 remove_proc_entry("ptype", net->proc_net);
900ff8c6321418 Cong Wang 2013-02-18 401 out_softnet:
900ff8c6321418 Cong Wang 2013-02-18 402 remove_proc_entry("softnet_stat", net->proc_net);
900ff8c6321418 Cong Wang 2013-02-18 403 out_dev:
900ff8c6321418 Cong Wang 2013-02-18 404 remove_proc_entry("dev", net->proc_net);
900ff8c6321418 Cong Wang 2013-02-18 405 goto out;
900ff8c6321418 Cong Wang 2013-02-18 406 }
900ff8c6321418 Cong Wang 2013-02-18 407 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org