page_pool: Add page_pool stat counters
Subsystems (3)
v5 cover page_pool: Add page_pool stat counters
Greetings:
Welcome to v5.
This revision contains some minor changes as suggested by Ilias (summarized below). These changes do not impact performance, so I've copied the performance results from v4 along with the note about my thoughts on the API I've exposed for use by drivers.
Since v4, this code exposes a single API: page_pool_get_stats which fills in a caller provided struct page_pool_stats with the summed per-cpu stats for the specified pool. The intention is that drivers will use this API to access pool stats so that they can be exposed to the user via ethtool, debugfs, etc. I have modified a driver to use this API. If this v5 is accepted, I will submit the driver changes in a follow-up to the appropriate list.
I chose this design of filling in a struct because pulling struct page_pool_stats into page_pool.c would require exposing a large number of APIs for accessing individual field values. The obvious downside of this is that changing the struct fields could break driver code. I don't have a strong sense for what the page_pool maintainers would prefer, so I chose the simplest option.
I re-ran the benchmarks using a larger number of loops and cpus, as documented below. I've summarized the results below; links to the raw output with this series applied with stats off [1] and stats on [2] are included for examination.
Test system:
- 2x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
- 2 NUMA zones, with 18 cores per zone and 2 threads per core
bench_page_pool_simple results, loops=200000000
test name stats enabled stats disabled
cycles nanosec cycles nanosec
for_loop 0 0.335 0 0.336atomic_inc 14 6.209 13 6.027
lock 31 13.571 31 13.935
no-softirq-page_pool01 70 30.783 68 29.959
no-softirq-page_pool02 73 32.124 72 31.412
no-softirq-page_pool03 112 49.060 108 47.373
tasklet_page_pool01_fast_path 14 6.413 13 5.833tasklet_page_pool02_ptr_ring 41 17.939 39 17.370tasklet_page_pool03_slow 110 48.179 108 47.173
bench_page_pool_cross_cpu results, loops=20000000 returning_cpus=4:
test name stats enabled stats disabled
cycles nanosec cycles nanosec
page_pool_cross_cpu CPU(0) 4101 1787.639 3990 1739.178page_pool_cross_cpu CPU(1) 4107 1790.281 3987 1737.756page_pool_cross_cpu CPU(2) 4102 1787.777 3991 1739.731page_pool_cross_cpu CPU(3) 4108 1790.455 3991 1739.703page_pool_cross_cpu CPU(4) 1027 447.614 997 434.933
page_pool_cross_cpu average 3489 - 3391 -
Thanks.
[1]: https://gist.githubusercontent.com/jdamato-fsly/343814d33ed75372e98782b796b607a4/raw/9f440a0b38f3eff37b76b914934620625172a0f4/v4%2520with%2520stats%2520off
[2]: https://gist.githubusercontent.com/jdamato-fsly/343814d33ed75372e98782b796b607a4/raw/9f440a0b38f3eff37b76b914934620625172a0f4/v4%2520with%2520stats%2520on
v4 -> v5:
- Fixed the description of the kernel option in Kconfig.
- Squashed commits 1-10 from v4 into a single commit for easier review.
- Changed the comment style of the comment for
the this_cpu_inc_alloc_stat macro.
- Changed the return type of page_pool_get_stats from struct page_pool_stat * to bool.
v3 -> v4:
- Restructured stats to be per-cpu per-pool.
- Global stats and proc file were removed.
- Exposed an API (page_pool_get_stats) for batching the pool stats.
v2 -> v3:
- patch 8/10 ("Add stat tracking cache refill") fixed placement of counter increment.
- patch 10/10 ("net-procfs: Show page pool stats in proc") updated:
- fix unused label warning from kernel test robot,
- fixed page_pool_seq_show to only display the refill stat once,
- added a remove_proc_entry for page_pool_stat to dev_proc_net_exit.
v1 -> v2:
- A new kernel config option has been added, which defaults to N, preventing this code from being compiled in by default
- The stats structure has been converted to a per-cpu structure
- The stats are now exported via proc (/proc/net/page_pool_stat)
Joe Damato (2):
page_pool: Add page_pool stat counters
page_pool: Add function to batch and return stats
include/net/page_pool.h | 27 +++++++++++++++++++++ net/Kconfig | 13 +++++++++++ net/core/page_pool.c | 62 +++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 98 insertions(+), 4 deletions(-)
Signature · 1 line
2.7.4
PATCH v5 1/2 page_pool: Add page_pool stat counters
Add per-cpu per-pool statistics counters for the allocation path of a page pool.
This code is disabled by default and a kernel config option is provided for users who wish to enable them.
The statistics added are:
- fast: successful fast path allocations
- slow: slow path order-0 allocations
- slow_high_order: slow path high order allocations
- empty: ptr ring is empty, so a slow path allocation was forced.
- refill: an allocation which triggered a refill of the cache
- waive: pages obtained from the ptr ring that cannot be added to the cache due to a NUMA mismatch.
--- include/net/page_pool.h | 18 ++++++++++++++++++ net/Kconfig | 13 +++++++++++++ net/core/page_pool.c | 37 +++++++++++++++++++++++++++++++++---- 3 files changed, 64 insertions(+), 4 deletions(-)
include/net/page_pool.h +18 -0
@@ -135,7 +135,25 @@ struct page_pool {refcount_t user_cnt;u64 destroy_cnt; +#ifdef CONFIG_PAGE_POOL_STATS +struct page_pool_stats __percpu *stats; +#endif +}; + +#ifdef CONFIG_PAGE_POOL_STATS +struct page_pool_stats { +struct { +u64 fast;/* fast path allocations */ +u64 slow;/* slow-path order 0 allocations */ +u64 slow_high_order;/* slow-path high order allocations */ +u64 empty;/* failed refills due to empty ptr ring, forcing +* slow path allocation +*/ +u64 refill;/* allocations via successful refill */ +u64 waive;/* failed refills due to numa zone mismatch */ + } alloc; }; +#endif struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);
net/Kconfig +13 -0
@@ -434,6 +434,19 @@ config NET_DEVLINK config PAGE_POOL bool +config PAGE_POOL_STATS + default n + bool "Page pool stats" + depends on PAGE_POOL + help + Enable page pool statistics to track allocations. This option + incurs additional CPU cost in allocation paths and additional + memory cost to store the statistics. These statistics are only + available if this option is enabled and if the driver using + the page pool supports exporting this data. + + If unsure, say N. + config FAILOVER tristate "Generic failover module" help
net/core/page_pool.c +33 -4
@@ -26,6 +26,17 @@#define BIAS_MAX LONG_MAX +#ifdef CONFIG_PAGE_POOL_STATS +/* this_cpu_inc_alloc_stat is intended to be used in softirq context */ +#define this_cpu_inc_alloc_stat (pool, __stat) \ + do { \ + struct page_pool_stats __percpu *s = pool->stats; \ + __this_cpu_inc(s->alloc.__stat); \ + } while (0) +#else +#define this_cpu_inc_alloc_stat (pool, __stat) +#endif +static int page_pool_init (struct page_pool *pool,const struct page_pool_params *params) {
@@ -73,6 +84,12 @@ static int page_pool_init(struct page_pool *pool, pool->p.flags & PP_FLAG_PAGE_FRAG)return -EINVAL ; +#ifdef CONFIG_PAGE_POOL_STATS + pool->stats = alloc_percpu(struct page_pool_stats); +if (!pool->stats) +return -ENOMEM ; +#endif +if (ptr_ring_init (&pool->ring, ring_qsize,GFP_KERNEL ) <0 )return -ENOMEM ;
@@ -117,8 +134,10 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)int pref_nid;/* preferred NUMA node */ /* Quicker fallback, avoid locks when ring is empty */ -if (__ptr_ring_empty (r)) +if (__ptr_ring_empty (r)) { +this_cpu_inc_alloc_stat (pool, empty);return NULL ; + }/* Softirq guarantee CPU and thus NUMA node is stable. This, * assumes CPU refilling driver RX-ring will also run RX-NAPI.
@@ -145,14 +164,17 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool)* This limit stress on page buddy alloactor. */ page_pool_return_page (pool, page); +this_cpu_inc_alloc_stat (pool, waive); page =NULL ;break ; } }while (pool->alloc.count <PP_ALLOC_CACHE_REFILL );/* Return last page */ -if (likely (pool->alloc.count >0 )) +if (likely (pool->alloc.count >0 )) { page = pool->alloc.cache[--pool->alloc.count]; +this_cpu_inc_alloc_stat (pool, refill); + }return page; }
@@ -166,6 +188,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)if (likely (pool->alloc.count)) {/* Fast-path */ page = pool->alloc.cache[--pool->alloc.count]; +this_cpu_inc_alloc_stat (pool, fast); }else { page =page_pool_refill_alloc_cache (pool); }
@@ -239,6 +262,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool,return NULL ; } +this_cpu_inc_alloc_stat (pool, slow_high_order);page_pool_set_pp_info (pool, page);/* Track how many pages are held 'in-flight' */
@@ -293,10 +317,12 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, }/* Return last page */ -if (likely (pool->alloc.count >0 )) +if (likely (pool->alloc.count >0 )) { page = pool->alloc.cache[--pool->alloc.count]; -else +this_cpu_inc_alloc_stat (pool, slow); + }else { page =NULL ; + }/* When page just alloc'ed is should/must have refcnt 1. */ return page;
@@ -620,6 +646,9 @@ static void page_pool_free(struct page_pool *pool)if (pool->p.flags &PP_FLAG_DMA_MAP )put_device (pool->p.dev); +#ifdef CONFIG_PAGE_POOL_STATS +free_percpu (pool->stats); +#endif kfree (pool); }
Signature · 1 line
2.7.4
Quoted 27 lines · Joe Damato
Add per-cpu per-pool statistics counters for the allocation path of a page pool.
This code is disabled by default and a kernel config option is provided for users who wish to enable them.
The statistics added are:
- fast: successful fast path allocations
- slow: slow path order-0 allocations
-slow_high_order: slow path high order allocations
- empty: ptr ring is empty, so a slow path allocation was forced.
- refill: an allocation which triggered a refill of the cache
- waive: pages obtained from the ptr ring that cannot be added to the cache due to a NUMA mismatch.Signed-off-by Joe Damato <[email protected]>--- include/net/page_pool.h | 18 ++++++++++++++++++ net/Kconfig | 13 +++++++++++++ net/core/page_pool.c | 37 +++++++++++++++++++++++++++++++++---- 3 files changed, 64 insertions(+), 4 deletions(-)include/net/page_pool.h +0
-0
include/net/page_pool.h struct page_pool { · line 141
Quoted 7 lines
include/net/page_pool.h +4
-0@@ -135,7 +135,25 @@ struct page_pool {refcount_t user_cnt;u64 destroy_cnt; +#ifdef CONFIG_PAGE_POOL_STATS +struct page_pool_stats __percpu *stats; +#endif +};
You still have to consider cache-line locality, as I have pointed out before. This placement is wrong!
Output from pahole:
/* --- cacheline 23 boundary (1472 bytes) --- */
atomic_t pages_state_release_cnt; /* 1472 4 */ refcount_t user_cnt; /* 1476 4 */ u64 destroy_cnt; /* 1480 8 */
Your *stats pointer end-up on a cache-line that "remote" CPUs will write into (pages_state_release_cnt). This is why we see a slowdown to the 'bench_page_pool_cross_cpu' test.
--Jesper
Quoted 32 lines · Jesper Dangaard Brouer
Quoted 29 lines · Joe Damato
Add per-cpu per-pool statistics counters for the allocation path of a page pool.
This code is disabled by default and a kernel config option is provided for users who wish to enable them.
The statistics added are:
- fast: successful fast path allocations
- slow: slow path order-0 allocations
-slow_high_order: slow path high order allocations
- empty: ptr ring is empty, so a slow path allocation was forced.
- refill: an allocation which triggered a refill of the cache
- waive: pages obtained from the ptr ring that cannot be added to the cache due to a NUMA mismatch.Signed-off-by Joe Damato <[email protected]>--- include/net/page_pool.h | 18 ++++++++++++++++++ net/Kconfig | 13 +++++++++++++ net/core/page_pool.c | 37 +++++++++++++++++++++++++++++++++---- 3 files changed, 64 insertions(+), 4 deletions(-)include/net/page_pool.h +0
-0@@ -135,7 +135,25 @@ struct page_pool {refcount_t user_cnt;
Quoted 8 lines
include/net/page_pool.h struct page_pool { · line 141
Quoted 5 lines
include/net/page_pool.h +4
-0u64 destroy_cnt; +#ifdef CONFIG_PAGE_POOL_STATS +struct page_pool_stats __percpu *stats; +#endif +};You still have to consider cache-line locality, as I have pointed out before.
You are right, I forgot to include that in this revision. Sorry about that and thanks for the review and response.
Quoted 4 lines
This placement is wrong!
Output from pahole:
Quoted 8 lines
/* --- cacheline 23 boundary (1472 bytes) --- */
atomic_t pages_state_release_cnt; /* 1472 4 */ refcount_t user_cnt; /* 1476 4 */ u64 destroy_cnt; /* 1480 8 */Your *stats pointer end-up on a cache-line that "remote" CPUs will write into (
pages_state_release_cnt).
This is why we see a slowdown to the 'bench_page_pool_cross_cpu' test.
If I give *stats its own cache-line by adding ____cacheline_aligned_in_smp (but leaving the placement at the end of the page_pool struct), pahole reports:
/* --- cacheline 24 boundary (1536 bytes) --- */
atomic_t pages_state_release_cnt; /* 1536 4 */ refcount_t user_cnt; /* 1540 4 */ u64 destroy_cnt; /* 1544 8 */
/* XXX 48 bytes hole, try to pack */
/* --- cacheline 25 boundary (1600 bytes) --- */
struct page_pool_stats * stats; /* 1600 8 */
Re-running bench_page_pool_cross_cpu loops=20000000 returning_cpus=4 still shows a fairly large variation in cycles (measurement period time of 34.128419339 sec) from run to run; roughly a delta of 287 cycles in the runs I just performed back to back.
The best measurements after making the cache-line change described above are faster than compiling the kernel with stats disabled. The worst measurements, however, are very close to the data I submit in the cover letter for this revision.
As far as I can tell xdp_mem_id is not written to particularly often - only when RX rings are configured in the driver - so I also tried moving *stats above xdp_mem_id so that they share a cache-line and reduce the size of the hole between xdp_mem_id and pp_alloc_cache.
pahole reports:
/* --- cacheline 3 boundary (192 bytes) --- */
struct page_pool_stats * stats; /* 192 8 */ u32 xdp_mem_id; /* 200 4 */
Results of bench_page_pool_cross_cpu loops=20000000 returning_cpus=4 are the same as above -- the best measurements are faster than stats disabled, the worst are very close to what I mentioned in the cover letter for this v5.
I am happy to submit a v6 with *stats placed in either location; on its own cache-line at the expense of a larger page_pool struct, or placed near xdp_mem_id to consume some of the hole between xdp_mem_id and pp_alloc_cache.
Do you have a preference on giving the stats pointer its own cache-line vs sharing a line with xdp_mem_id?
In either case: the benchmarks don't seem to show a consistent significant improvement on my test hardware. Is there another benchmark or a different set of arguments you think I should use when running this test?
My apologies if I am missing something obvious here.
Thanks,
Joe
PATCH v5 2/2 page_pool: Add function to batch and return stats
In reply to Joe DamatoAdds a function page_pool_get_stats which can be used by drivers to obtain the batched stats for a specified page pool.
--- include/net/page_pool.h | 9 +++++++++ net/core/page_pool.c | 25 +++++++++++++++++++++++++ 2 files changed, 34 insertions(+)
include/net/page_pool.h +9 -0
@@ -153,6 +153,15 @@ struct page_pool_stats { u64 waive; /* failed refills due to numa zone mismatch */ } alloc; }; + +/* +* Drivers that wish to harvest page pool stats and report them to users +* (perhaps via ethtool, debugfs, or another mechanism) can allocate a +* struct page_pool_stats and call page_pool_get_stats to get the batched pcpu +* stats. +*/ +bool page_pool_get_stats (struct page_pool *pool, +struct page_pool_stats *stats);#endif struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);
net/core/page_pool.c +25 -0
@@ -33,6 +33,31 @@ struct page_pool_stats __percpu *s = pool->stats; \__this_cpu_inc (s->alloc.__stat); \ } while (0) + +bool page_pool_get_stats(struct page_pool *pool, + struct page_pool_stats *stats) +{ + int cpu = 0; + +if (!stats) +return false; + +for_each_possible_cpu (cpu) { +const struct page_pool_stats *pcpu = +per_cpu_ptr (pool->stats, cpu); + + stats->alloc.fast += pcpu->alloc.fast; + stats->alloc.slow += pcpu->alloc.slow; + stats->alloc.slow_high_order += + pcpu->alloc.slow_high_order; + stats->alloc.empty += pcpu->alloc.empty; + stats->alloc.refill += pcpu->alloc.refill; + stats->alloc.waive += pcpu->alloc.waive; + } + + return true; +} +EXPORT_SYMBOL (page_pool_get_stats);#else #define this_cpu_inc_alloc_stat (pool, __stat)#endif
Signature · 1 line
2.7.4