page_pool cleanups
Subsystems (6)
cover page_pool cleanups
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts with AF_XDP.
- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
- Statistics and cleanup for page pool.
Jonathan Lemon (5):
page_pool: Add page_pool_keep_page
page_pool: allow configurable linear cache size
page_pool: Add statistics
net/mlx5: Add page_pool stats to the Mellanox driver
page_pool: Cleanup and rename page_pool functions.
Saeed Mahameed (2):
page_pool: Add API to update numa node and flush page caches net/mlx5e: Rx, Update page pool numa node when changed
Tariq Toukan (3):
net/mlx5e: RX, Remove RX page-cache net/mlx5e: RX, Manage RX pages only via page pool API net/mlx5e: RX, Internal DMA mapping in page_pool
drivers/net/ethernet/mellanox/mlx5/core/en.h | 18 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 12 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 19 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 128 ++-------- .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++-- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +- include/net/page_pool.h | 216 +++++++++-------- net/core/page_pool.c | 221 +++++++++++------- 8 files changed, 319 insertions(+), 353 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/10 net/mlx5e: RX, Remove RX page-cache
From: Tariq Toukan <[email protected]>
Obsolete the RX page-cache layer, pages are now recycled in page_pool.
This patch introduce a temporary degradation as recycling does not keep the pages DMA-mapped. That is fixed in a downstream patch.
Issue: 1487631
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 13 ---- .../net/ethernet/mellanox/mlx5/core/en_main.c | 16 ----- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 67 ++----------------- 3 files changed, 4 insertions(+), 92 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -13
@@ -583,18 +583,6 @@ struct mlx5e_mpw_info {#define MLX5E_MAX_RX_FRAGS 4 -/* a single cache unit is capable to serve one napi call (for non-striding rq) -* or a MPWQE (for striding rq). -*/ -#define MLX5E_CACHE_UNIT (MLX5_MPWRQ_PAGES_PER_WQE > NAPI_POLL_WEIGHT ? \ - MLX5_MPWRQ_PAGES_PER_WQE : NAPI_POLL_WEIGHT) -#define MLX5E_CACHE_SIZE (4 * roundup_pow_of_two(MLX5E_CACHE_UNIT)) -struct mlx5e_page_cache { -u32 head; -u32 tail; -struct mlx5e_dma_info page_cache[MLX5E_CACHE_SIZE ]; -}; -struct mlx5e_rq ;typedef void (*mlx5e_fp_handle_rx_cqe )(struct mlx5e_rq *,struct mlx5_cqe64 *); typedef struct sk_buff *
@@ -658,7 +646,6 @@ struct mlx5e_rq {struct mlx5e_rq_stats *stats;struct mlx5e_cq cq;struct mlx5e_cq_decomp cqd; -struct mlx5e_page_cache page_cache;struct hwtstamp_config *tstamp;struct mlx5_clock *clock;
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0 -16
@@ -612,9 +612,6 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->dim.mode =DIM_CQ_PERIOD_MODE_START_FROM_EQE ; } - rq->page_cache.head =0 ; - rq->page_cache.tail =0 ; -return 0 ; err_free:
@@ -640,8 +637,6 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,static void mlx5e_free_rq (struct mlx5e_rq *rq) { -int i; -if (rq->xdp_prog)bpf_prog_put (rq->xdp_prog);
@@ -655,17 +650,6 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)mlx5e_free_di_list (rq); } -for (i = rq->page_cache.head; i != rq->page_cache.tail; - i = (i +1 ) & (MLX5E_CACHE_SIZE -1 )) { -struct mlx5e_dma_info *dma_info = &rq->page_cache.page_cache[i]; - -/* With AF_XDP, page_cache is not used, so this loop is not -* entered, and it's safe to call mlx5e_page_release_dynamic -* directly. -*/ -mlx5e_page_release_dynamic (rq, dma_info, false); - } -xdp_rxq_info_unreg (&rq->xdp_rxq);page_pool_destroy (rq->page_pool);mlx5_wq_destroy (&rq->wq_ctrl);
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +4 -63
97 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH 01/10 net-next] net/mlx5e: RX, Remove RX page-cache
Quoted 10 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
Obsolete the RX page-cache layer, pages are now recycled in
page_pool.This patch introduce a temporary degradation as recycling does not keep the pages DMA-mapped. That is fixed in a downstream patch.
Issue: 1487631
Please drop these Issue identifiers.
Quoted 2 lines
Signed-off-by Tariq Toukan <[email protected]>
And no empty lines between tags.
Quoted 1 line
Signed-off-by Jonathan Lemon <[email protected]>
Re: [PATCH 01/10 net-next] net/mlx5e: RX, Remove RX page-cache
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
Obsolete the RX page-cache layer, pages are now recycled in
page_pool.This patch introduce a temporary degradation as recycling does not keep the pages DMA-mapped. That is fixed in a downstream patch.
Quoted 10 lines
Quoted 1 line
Issue: 1487631
Please drop these Issue identifiers.
Quoted 2 lines
Signed-off-by Tariq Toukan <[email protected]>And no empty lines between tags.
Quoted 1 line
Signed-off-by Jonathan Lemon <[email protected]>
Will fix.
Signature · 1 line
Jonathan
Re: [PATCH 01/10 net-next] net/mlx5e: RX, Remove RX page-cache
In reply to Jonathan LemonQuoted 8 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
Obsolete the RX page-cache layer, pages are now recycled in
page_pool.This patch introduce a temporary degradation as recycling does not keep the pages DMA-mapped. That is fixed in a downstream patch.
Tariq,
i think we need to have performance numbers here to show degradation. i am sure that non XDP traffic TCP/UDP performance will be hit.
Quoted 211 lines
Issue: 1487631
Signed-off-by Tariq Toukan <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 13 ---- .../net/ethernet/mellanox/mlx5/core/en_main.c | 16 ----- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 67 ++--------------- -- 3 files changed, 4 insertions(+), 92 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/en.h +0
-6@@ -583,18 +583,6 @@ struct mlx5e_mpw_info {#define MLX5E_MAX_RX_FRAGS 4 -/* a single cache unit is capable to serve one napi call (for non- striding rq) -* or a MPWQE (for striding rq). -*/ -#define MLX5E_CACHE_UNIT (MLX5_MPWRQ_PAGES_PER_WQE > NAPI_POLL_WEIGHT ? \ - MLX5_MPWRQ_PAGES_PER_WQE : NAPI_POLL_WEIGHT) -#define MLX5E_CACHE_SIZE (4 *
roundup_pow_of_two(MLX5E_CACHE_UNIT))drivers/net/ethernet/mellanox/mlx5/core/en.h +0
-6-struct mlx5e_page_cache { -u32 head; -u32 tail; -struct mlx5e_dma_info page_cache[MLX5E_CACHE_SIZE ]; -}; -struct mlx5e_rq ; typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, structmlx5_cqe64*);
typedef structsk_buff*drivers/net/ethernet/mellanox/mlx5/core/en.h +0
-1@@ -658,7 +646,6 @@ struct mlx5e_rq {struct mlx5e_rq_stats *stats;struct mlx5e_cq cq;struct mlx5e_cq_decomp cqd; -struct mlx5e_page_cache page_cache;struct hwtstamp_config *tstamp;struct mlx5_clock *clock;drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-3@@ -612,9 +612,6 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, rq->dim.mode =DIM_CQ_PERIOD_MODE_START_FROM_EQE ; } - rq->page_cache.head =0 ; - rq->page_cache.tail =0 ; -return 0 ;
err_free:drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-7@@ -640,8 +637,6 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, static void mlx5e_free_rq(struct mlx5e_rq *rq) { -int i; -if (rq->xdp_prog)bpf_prog_put (rq->xdp_prog);@@ -655,17 +650,6 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)mlx5e_free_di_list (rq); } -for (i = rq->page_cache.head; i != rq->page_cache.tail; - i = (i +1 ) & (MLX5E_CACHE_SIZE -1 )) { -struct mlx5e_dma_info *dma_info = &rq- >page_cache.page_cache[i]; - -/* With AF_XDP, page_cache is not used, so this loop is not
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-1-* entered, and it's safe to call
mlx5e_page_release_dynamicdrivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-5-* directly. -*/ -mlx5e_page_release_dynamic (rq, dma_info, false); - } -xdp_rxq_info_unreg (&rq->xdp_rxq);page_pool_destroy (rq->page_pool);mlx5_wq_destroy (&rq->wq_ctrl);drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +0
-56@@ -184,65 +184,9 @@ static inline u32 mlx5e_decompress_cqes_start(struct mlx5e_rq *rq, return mlx5e_decompress_cqes_cont(rq, wq, 1, budget_rem) - 1; } -static inline bool mlx5e_page_is_reserved (struct page *page) -{ -return page_is_pfmemalloc (page) ||page_to_nid (page) !=numa_mem_id (); -} - -static inline bool mlx5e_rx_cache_put (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info) -{ -struct mlx5e_page_cache *cache = &rq->page_cache; -u32 tail_next = (cache->tail +1 ) & (MLX5E_CACHE_SIZE -1 ); -struct mlx5e_rq_stats *stats = rq->stats; - -if (tail_next == cache->head) { - stats->cache_full++; -return false; - } - -if (unlikely (mlx5e_page_is_reserved (dma_info->page))) { - stats->cache_waive++; -return false; - } - - cache->page_cache[cache->tail] = *dma_info; - cache->tail = tail_next; -return true; -} - -static inline bool mlx5e_rx_cache_get (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info) -{ -struct mlx5e_page_cache *cache = &rq->page_cache; -struct mlx5e_rq_stats *stats = rq->stats; - -if (unlikely (cache->head == cache->tail)) { - stats->cache_empty++; -return false; - } - -if (page_ref_count (cache->page_cache[cache->head].page) !=1 ) { - stats->cache_busy++; -return false; - } - - *dma_info = cache->page_cache[cache->head]; - cache->head = (cache->head +1 ) & (MLX5E_CACHE_SIZE -1 ); - stats->cache_reuse++; - -dma_sync_single_for_device (rq->pdev, dma_info->addr, -PAGE_SIZE , -DMA_FROM_DEVICE ); -return true; -} -static inline int mlx5e_page_alloc_pool (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) { -if (mlx5e_rx_cache_get (rq, dma_info)) -return 0 ; -drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +2
-5dma_info->page =page_pool_dev_alloc_pages (rq->page_pool);if (unlikely (!dma_info->page))return -ENOMEM ;@@ -276,14 +220,11 @@ void mlx5e_page_release_dynamic(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info, bool recycle) { -if (likely (recycle)) { -if (mlx5e_rx_cache_put (rq, dma_info)) -return ; +mlx5e_page_dma_unmap (rq, dma_info); -mlx5e_page_dma_unmap (rq, dma_info); +if (likely (recycle)) {page_pool_recycle_direct (rq->page_pool, dma_info- >page); }else { -mlx5e_page_dma_unmap (rq, dma_info);page_pool_release_page (rq->page_pool, dma_info->page);drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1
-1put_page (dma_info->page); }@@ -1167,7 +1108,7 @@ void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)if (!skb) {/* probably for XDP */ if (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq- >flags)) { -/* do not return page to cache, +/* do not return page to pool, * it will be returned on XDP_TX completion. drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1
-1*/ goto wq_cyc_pop;@@ -1210,7 +1151,7 @@ void mlx5e_handle_rx_cqe_rep(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)if (!skb) {/* probably for XDP */ if (__test_and_clear_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq- >flags)) { -/* do not return page to cache, +/* do not return page to pool, * it will be returned on XDP_TX completion. drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +0
-0*/ goto wq_cyc_pop;
Re: [PATCH 01/10 net-next] net/mlx5e: RX, Remove RX page-cache
I was running the updated patches on machines with various workloads, and have a bunch of different results.
For the following numbers,
Effective = hit / (hit + empty + stall) * 100
In other words, show the hit rate for for every trip to the cache, and the cache full stat is ignored.
On a webserver:
[web] # ./eff
('rx_pool_cache_hit:', '360127643')
('rx_pool_cache_full:', '0')
('rx_pool_cache_empty:', '6455735977')
('rx_pool_ring_produce:', '474958')
('rx_pool_ring_consume:', '0')
('rx_pool_ring_return:', '474958')
('rx_pool_flush:', '144')
('rx_pool_node_change:', '0')
cache effectiveness: 5.28
On a proxygen:
# ethtool -S eth0 | grep rx_pool
rx_pool_cache_hit: 1646798
rx_pool_cache_full: 0
rx_pool_cache_empty: 15723566
rx_pool_ring_produce: 474958
rx_pool_ring_consume: 0
rx_pool_ring_return: 474958
rx_pool_flush: 144
rx_pool_node_change: 0
cache effectiveness: 9.48On both of these, only pages with refcount = 1 are being kept.
I changed things around in the page pool so:
1) the cache behaves like a ring instead of a stack, this sacrifices temporal locality.
2) it caches all pages returned regardless of refcount, but only returns pages with refcount=1.
This is the same behavior as the mlx5 cache. Some gains would come about if the sojourn time though the cache is greater than the lifetime of the page usage by the networking stack, as it provides a fixed working set of mapped pages.
On the web server, this is a net loss:
[web] # ./eff
('rx_pool_cache_hit:', '6052662')
('rx_pool_cache_full:', '156355415')
('rx_pool_cache_empty:', '409600')
('rx_pool_cache_stall:', '302787473')
('rx_pool_ring_produce:', '156633847')
('rx_pool_ring_consume:', '9925520')
('rx_pool_ring_return:', '278788')
('rx_pool_flush:', '96')
('rx_pool_node_change:', '0')
cache effectiveness: 1.95720846778
For proxygen on the other hand, it's a win:
[proxy] # ./eff
('rx_pool_cache_hit:', '69235177')
('rx_pool_cache_full:', '35404387')
('rx_pool_cache_empty:', '460800')
('rx_pool_cache_stall:', '42932530')
('rx_pool_ring_produce:', '35717618')
('rx_pool_ring_consume:', '27879469')
('rx_pool_ring_return:', '404800')
('rx_pool_flush:', '108')
('rx_pool_node_change:', '0')
cache effectiveness: 61.4721608624
So the correct behavior isn't quite clear cut here - caching a working set of mapped pages is beneficial in spite of the HOL blocking stalls for some workloads, but I'm sure that it wouldn't be too difficult to exceed the WS size.
Thoughts?
Signature · 1 line
Jonathan
Re: [PATCH 01/10 net-next] net/mlx5e: RX, Remove RX page-cache
Quoted 76 lines · Jonathan Lemon
I was running the updated patches on machines with various workloads, and have a bunch of different results.
For the following numbers,
Effective = hit / (hit + empty + stall) * 100In other words, show the hit rate for for every trip to the cache, and the cache full stat is ignored.
On a webserver:
[web] # ./eff
('rx_pool_cache_hit:', '360127643')
('rx_pool_cache_full:', '0')
('rx_pool_cache_empty:', '6455735977')
('rx_pool_ring_produce:', '474958')
('rx_pool_ring_consume:', '0')
('rx_pool_ring_return:', '474958')
('rx_pool_flush:', '144')
('rx_pool_node_change:', '0')
cache effectiveness: 5.28On a proxygen:
# ethtool -S eth0 | grep rx_pool rx_pool_cache_hit: 1646798 rx_pool_cache_full: 0 rx_pool_cache_empty: 15723566 rx_pool_ring_produce: 474958 rx_pool_ring_consume: 0 rx_pool_ring_return: 474958 rx_pool_flush: 144 rx_pool_node_change: 0 cache effectiveness: 9.48On both of these, only pages with refcount = 1 are being kept.
I changed things around in the page pool so:
1) the cache behaves like a ring instead of a stack, this sacrifices temporal locality.
2) it caches all pages returned regardless of refcount, but only returns pages with refcount=1.
This is the same behavior as the mlx5 cache. Some gains would come about if the sojourn time though the cache is greater than the lifetime of the page usage by the networking stack, as it provides a fixed working set of mapped pages.
On the web server, this is a net loss:
[web] # ./eff
('rx_pool_cache_hit:', '6052662')
('rx_pool_cache_full:', '156355415')
('rx_pool_cache_empty:', '409600')
('rx_pool_cache_stall:', '302787473')
('rx_pool_ring_produce:', '156633847')
('rx_pool_ring_consume:', '9925520')
('rx_pool_ring_return:', '278788')
('rx_pool_flush:', '96')
('rx_pool_node_change:', '0')
cache effectiveness: 1.95720846778For proxygen on the other hand, it's a win:
[proxy] # ./eff
('rx_pool_cache_hit:', '69235177')
('rx_pool_cache_full:', '35404387')
('rx_pool_cache_empty:', '460800')
('rx_pool_cache_stall:', '42932530')
('rx_pool_ring_produce:', '35717618')
('rx_pool_ring_consume:', '27879469')
('rx_pool_ring_return:', '404800')
('rx_pool_flush:', '108')
('rx_pool_node_change:', '0')
cache effectiveness: 61.4721608624
Quoted 7 lines
So the correct behavior isn't quite clear cut here - caching a working set of mapped pages is beneficial in spite of the HOL blocking stalls for some workloads, but I'm sure that it wouldn't be too difficult to exceed the WS size.
Thoughts?
We have a WIP in which we avoid the HOL block, by having pages returned to the available-queue only when their refcnt reaches back to 1. This requires catching this case in the page/skb release path.
See:
https://github.com/xdp-project/xdp-project/tree/master/areas/mem
PATCH 2/10 net/mlx5e: RX, Manage RX pages only via page pool API
In reply to Jonathan LemonFrom: Tariq Toukan <[email protected]>
Do not directly call page allocator functions, but use page pool API only.
Obsolete the recycle param.
Issue: 1487631
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 9 ++-- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 46 +++++++------------ 3 files changed, 23 insertions(+), 36 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +2 -2
@@ -928,8 +928,8 @@ bool mlx5e_striding_rq_possible(struct mlx5_core_dev *mdev,void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_page_release_dynamic (struct mlx5e_rq *rq, -struct mlx5e_dma_info *dma_info, -bool recycle); +struct mlx5e_dma_info *dma_info); +void mlx5e_page_release (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_handle_rx_cqe (struct mlx5e_rq *rq,struct mlx5_cqe64 *cqe);void mlx5e_handle_rx_cqe_mpwrq (struct mlx5e_rq *rq,struct mlx5_cqe64 *cqe);bool mlx5e_post_rx_wqes (struct mlx5e_rq *rq);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +4 -5
@@ -354,8 +354,7 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,static void mlx5e_free_xdpsq_desc (struct mlx5e_xdpsq *sq,struct mlx5e_xdp_wqe_info *wi, -u32 *xsk_frames, -bool recycle) +u32 *xsk_frames) {struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo;u16 i;
@@ -372,7 +371,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,break ;case MLX5E_XDP_XMIT_MODE_PAGE :/* XDP_TX from the regular RQ */ -mlx5e_page_release_dynamic (xdpi.page.rq, &xdpi.page.di, recycle); +mlx5e_page_release_dynamic (xdpi.page.rq, &xdpi.page.di);break ;case MLX5E_XDP_XMIT_MODE_XSK :/* AF_XDP send */
@@ -430,7 +429,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq) sqcc += wi->num_wqebbs; -mlx5e_free_xdpsq_desc (sq, wi, &xsk_frames, true); +mlx5e_free_xdpsq_desc (sq, wi, &xsk_frames); }while (!last_wqe); }while ((++i <MLX5E_TX_CQ_POLL_BUDGET ) && (cqe =mlx5_cqwq_get_cqe (&cq->wq)));
@@ -461,7 +460,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq) sq->cc += wi->num_wqebbs; -mlx5e_free_xdpsq_desc (sq, wi, &xsk_frames, false); +mlx5e_free_xdpsq_desc (sq, wi, &xsk_frames); }if (xsk_frames)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +17 -29
127 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 3/10 net/mlx5e: RX, Internal DMA mapping in page_pool
In reply to Jonathan LemonFrom: Tariq Toukan <[email protected]>
After RX page-cache is removed in previous patch, let the page_pool be responsible for the DMA mapping.
Issue: 1487631
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 -- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 +-- .../net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 16 +--------------- 4 files changed, 3 insertions(+), 20 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +0 -2
@@ -926,10 +926,8 @@ bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev);bool mlx5e_striding_rq_possible (struct mlx5_core_dev *mdev,struct mlx5e_params *params); -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info); -void mlx5e_page_release (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_handle_rx_cqe (struct mlx5e_rq *rq,struct mlx5_cqe64 *cqe);void mlx5e_handle_rx_cqe_mpwrq (struct mlx5e_rq *rq,struct mlx5_cqe64 *cqe);bool mlx5e_post_rx_wqes (struct mlx5e_rq *rq);
drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +1 -2
@@ -161,8 +161,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di, goto xdp_abort;__set_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags);__set_bit (MLX5E_RQ_FLAG_XDP_REDIRECT , rq->flags); -if (!xsk) -mlx5e_page_dma_unmap (rq, di); +/* xdp maps call xdp_release_frame() if needed */ rq->stats->xdp_redirect++;return true;default :
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -1
@@ -546,7 +546,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, }else {/* Create a page_pool and register it with rxq */ pp_params.order =0 ; - pp_params.flags =0 ;/* No-internal DMA mapping in page_pool */ + pp_params.flags =PP_FLAG_DMA_MAP ; pp_params.pool_size = pool_size; pp_params.nid =cpu_to_node (c->cpu); pp_params.dev = c->pdev;
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1 -15
@@ -190,14 +190,7 @@ static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq, dma_info->page =page_pool_dev_alloc_pages (rq->page_pool);if (unlikely (!dma_info->page))return -ENOMEM ; - - dma_info->addr =dma_map_page (rq->pdev, dma_info->page,0 , -PAGE_SIZE , rq->buff.map_dir); -if (unlikely (dma_mapping_error (rq->pdev, dma_info->addr))) { -page_pool_recycle_direct (rq->page_pool, dma_info->page); - dma_info->page =NULL ; -return -ENOMEM ; - } + dma_info->addr =page_pool_get_dma_addr (dma_info->page);return 0 ; }
@@ -211,16 +204,9 @@ static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,return mlx5e_page_alloc_pool (rq, dma_info); } -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) -{ -dma_unmap_page (rq->pdev, dma_info->addr,PAGE_SIZE , rq->buff.map_dir); -} -void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) { -mlx5e_page_dma_unmap (rq, dma_info); -page_pool_recycle_direct (rq->page_pool, dma_info->page); }
Signature · 1 line
2.17.1
Re: [PATCH 03/10 net-next] net/mlx5e: RX, Internal DMA mapping in page_pool
Quoted 9 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
After RX page-cache is removed in previous patch, let the
page_poolbe responsible for the DMA mapping.Issue: 1487631
Signed-off-by Tariq Toukan <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>
IIUC you'll need to add DMA syncs here, no? map/unmap has syncing as side effect.
Re: [PATCH 03/10 net-next] net/mlx5e: RX, Internal DMA mapping in page_pool
Quoted 6 lines · Jakub Kicinski
Quoted 5 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
After RX page-cache is removed in previous patch, let the
page_poolbe responsible for the DMA mapping.
Quoted 7 lines
Quoted 4 lines
Issue: 1487631
Signed-off-by Tariq Toukan <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>IIUC you'll need to add DMA syncs here, no? map/unmap has syncing as side effect.
The driver still needs to do DMA sync for data transfers, this only covers the initial mapping.
Signature · 1 line
Jonathan
Re: [PATCH 03/10 net-next] net/mlx5e: RX, Internal DMA mapping in page_pool
In reply to Jonathan LemonQuoted 76 lines · Jonathan Lemon
From: Tariq Toukan <[email protected]>
After RX page-cache is removed in previous patch, let the
page_poolbe responsible for the DMA mapping.Issue: 1487631
Signed-off-by Tariq Toukan <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 -- drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 +-- .../net/ethernet/mellanox/mlx5/core/en_main.c | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 16 +------------- -- 4 files changed, 3 insertions(+), 20 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/en.h +0
-2@@ -926,10 +926,8 @@ bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev);bool mlx5e_striding_rq_possible (struct mlx5_core_dev *mdev,struct mlx5e_params *params); -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info); -void mlx5e_page_release (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info);void mlx5e_handle_rx_cqe(struct
mlx5e_rq*rq, structmlx5_cqe64
*cqe);
void mlx5e_handle_rx_cqe_mpwrq(structmlx5e_rq*rq, structmlx5_cqe64*cqe);
boolmlx5e_post_rx_wqes(struct mlx5e_rq *rq);drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +1
-2@@ -161,8 +161,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, structmlx5e_dma_info *di, goto xdp_abort;__set_bit (MLX5E_RQ_FLAG_XDP_XMIT , rq->flags);__set_bit (MLX5E_RQ_FLAG_XDP_REDIRECT , rq->flags); -if (!xsk) -mlx5e_page_dma_unmap (rq, di); +/* xdp maps call xdp_release_frame() if needed */ rq->stats->xdp_redirect++;return true;drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +0
-0default :drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-1@@ -546,7 +546,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, }else {/* Create a page_pool and register it with rxq */ pp_params.order =0 ; - pp_params.flags =0 ;/* No-internal DMA mapping in
page_pool*/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1
-0+ pp_params.flags =PP_FLAG_DMA_MAP ; pp_params.pool_size = pool_size; pp_params.nid =cpu_to_node (c->cpu); pp_params.dev = c->pdev;drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +0
-0
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c static inline int mlx5e_page_alloc_pool(struct · line 193
Quoted 4 lines
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +0
-0@@ -190,14 +190,7 @@ static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq, dma_info->page =page_pool_dev_alloc_pages (rq->page_pool);if (unlikely (!dma_info->page))return -ENOMEM ;
Jakub is right, you need to dma_sync for device after allocating .. pages recycled from cache can be synced for CPU .. you need to sync them for device here.
Quoted 33 lines
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +1
-8- - dma_info->addr =dma_map_page (rq->pdev, dma_info->page,0 , -PAGE_SIZE , rq->buff.map_dir); -if (unlikely (dma_mapping_error (rq->pdev, dma_info->addr))) { -page_pool_recycle_direct (rq->page_pool, dma_info- >page); - dma_info->page =NULL ; -return -ENOMEM ; - } + dma_info->addr =page_pool_get_dma_addr (dma_info->page);drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +0
-7return 0 ; }@@ -211,16 +204,9 @@ static inline int mlx5e_page_alloc(structmlx5e_rq *rq, return mlx5e_page_alloc_pool(rq, dma_info); } -void mlx5e_page_dma_unmap (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) -{ -dma_unmap_page (rq->pdev, dma_info->addr,PAGE_SIZE , rq- >buff.map_dir); -} -void mlx5e_page_release_dynamic (struct mlx5e_rq *rq,struct mlx5e_dma_info *dma_info) { -mlx5e_page_dma_unmap (rq, dma_info); -
page_pool_recycle_direct(rq->page_pool, dma_info->page);
}
PATCH 4/10 page_pool: Add API to update numa node and flush page caches
In reply to Jonathan LemonFrom: Saeed Mahameed <[email protected]>
Add page_pool_update_nid() to be called from drivers when they detect numa node changes.
It will do:
1) Flush the pool's page cache and ptr_ring.
2) Update page pool nid value to start allocating from the new numa node.
--- include/net/page_pool.h | 10 ++++++++++ net/core/page_pool.c | 16 +++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)
include/net/page_pool.h +10 -0
@@ -226,4 +226,14 @@ static inline bool page_pool_put(struct page_pool *pool) return refcount_dec_and_test(&pool->user_cnt); } +/* Only safe from napi context or when user guarantees it is thread safe */ +void __page_pool_flush (struct page_pool *pool); +static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) +{ +if (unlikely (pool->p.nid != new_nid)) { +/* TODO: Add statistics/trace */ +__page_pool_flush (pool); + pool->p.nid = new_nid; + } +}#endif /* _NET_PAGE_POOL_H */
net/core/page_pool.c +11 -5
@@ -373,16 +373,13 @@ void __page_pool_free(struct page_pool *pool) }EXPORT_SYMBOL (__page_pool_free); -/* Request to shutdown: release pages cached by page_pool, and check -* for in-flight pages -*/ -bool __page_pool_request_shutdown (struct page_pool *pool) +void __page_pool_flush (struct page_pool *pool) {struct page *page;/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be -* call concurrently. +* called concurrently. */ while (pool->alloc.count) { page = pool->alloc.cache[--pool->alloc.count];
@@ -393,6 +390,15 @@ bool __page_pool_request_shutdown(struct page_pool *pool)* be in-flight. */ __page_pool_empty_ring (pool); +} +EXPORT_SYMBOL (__page_pool_flush); + +/* Request to shutdown: release pages cached by page_pool, and check +* for in-flight pages +*/ +bool __page_pool_request_shutdown (struct page_pool *pool) +{ +__page_pool_flush (pool);return __page_pool_safe_to_destroy (pool); }
Signature · 1 line
2.17.1
Ilias Apalodimas
maintainerRe: [PATCH 04/10 net-next] page_pool: Add API to update numa node and flush page caches
Hi Saeed,
Quoted 22 lines · Jonathan Lemon
From: Saeed Mahameed <[email protected]>
Add
page_pool_update_nid()to be called from drivers when they detect numa node changes.It will do:
1) Flush the pool's page cache andptr_ring.
2) Update page pool nid value to start allocating from the new numa node.Signed-off-by Saeed Mahameed <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 10 ++++++++++ net/core/page_pool.c | 16 +++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)include/net/page_pool.h +0
-0
include/net/page_pool.h static inline bool page_pool_put(struct page_pool *pool) · line 230
Quoted 5 lines
include/net/page_pool.h +2
-0@@ -226,4 +226,14 @@ static inline bool page_pool_put(struct page_pool *pool)return refcount_dec_and_test (&pool->user_cnt); } +/* Only safe from napi context or when user guarantees it is thread safe */ +void __page_pool_flush (struct page_pool *pool);
This should be called per packet right? Any noticeable impact on performance?
Quoted 33 lines
include/net/page_pool.h +8
-0+static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) +{ +if (unlikely (pool->p.nid != new_nid)) { +/* TODO: Add statistics/trace */ +__page_pool_flush (pool); + pool->p.nid = new_nid; + } +}#endif /* _NET_PAGE_POOL_H */ net/core/page_pool.c +2
-5@@ -373,16 +373,13 @@ void __page_pool_free(struct page_pool *pool) }EXPORT_SYMBOL (__page_pool_free); -/* Request to shutdown: release pages cached by page_pool, and check -* for in-flight pages -*/ -bool __page_pool_request_shutdown (struct page_pool *pool) +void __page_pool_flush (struct page_pool *pool) {struct page *page;/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be -* call concurrently. +* called concurrently. */ while (pool->alloc.count) { page = pool->alloc.cache[--pool->alloc.count];
net/core/page_pool.c bool __page_pool_request_shutdown(struct page_pool *pool) · line 394
Quoted 5 lines
net/core/page_pool.c +2
-0@@ -393,6 +390,15 @@ bool __page_pool_request_shutdown(struct page_pool *pool)* be in-flight. */ __page_pool_empty_ring (pool); +} +EXPORT_SYMBOL (__page_pool_flush);
A later patch removes this, do we actually need it here?
Quoted 13 lines
net/core/page_pool.c +7
-0+ +/* Request to shutdown: release pages cached by page_pool, and check +* for in-flight pages +*/ +bool __page_pool_request_shutdown (struct page_pool *pool) +{ +__page_pool_flush (pool);return __page_pool_safe_to_destroy (pool); }Signature · 1 line
2.17.1
Thanks
/Ilias
Re: [PATCH 04/10 net-next] page_pool: Add API to update numa node and flush page caches
Quoted 30 lines · Ilias Apalodimas
Hi Saeed,
Quoted 27 lines · Jonathan Lemon
From: Saeed Mahameed <[email protected]>
Add
page_pool_update_nid()to be called from drivers when they detect
numa node changes.It will do:
1) Flush the pool's page cache andptr_ring.
2) Update page pool nid value to start allocating from the new numa node.Signed-off-by Saeed Mahameed <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 10 ++++++++++ net/core/page_pool.c | 16 +++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)include/net/page_pool.h +0
-0@@ -226,4 +226,14 @@ static inline bool page_pool_put(struct page_pool *pool)return refcount_dec_and_test (&pool->user_cnt); }
Quoted 7 lines
Quoted 3 lines
include/net/page_pool.h +1
-0+/* Only safe from napi context or when user guarantees it is thread safe */
+void__page_pool_flush(struct page_pool *pool);This should be called per packet right? Any noticeable impact on performance?
no, once per napi and only if a change in numa node is detected, so very very rare !
Quoted 27 lines
Quoted 27 lines
+static inline void
page_pool_update_nid(struct page_pool *pool, int new_nid)include/net/page_pool.h +7
-0+{ +if (unlikely (pool->p.nid != new_nid)) { +/* TODO: Add statistics/trace */ +__page_pool_flush (pool); + pool->p.nid = new_nid; + } +}#endif /* _NET_PAGE_POOL_H */ net/core/page_pool.c +0
-1@@ -373,16 +373,13 @@ void __page_pool_free(struct page_pool *pool) }EXPORT_SYMBOL (__page_pool_free); -/* Request to shutdown: release pages cached by page_pool, and check
- * for in-flight pages
- */net/core/page_pool.c +1
-1-bool __page_pool_request_shutdown (struct page_pool *pool) +void __page_pool_flush (struct page_pool *pool) {struct page *page;
Quoted 16 lines
Quoted 8 lines
net/core/page_pool.c +1
-1/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be -* call concurrently. +* called concurrently. */ while (pool->alloc.count) { page = pool->alloc.cache[--pool->alloc.count];net/core/page_pool.c bool __page_pool_request_shutdown(struct · line 395
Quoted 6 lines
net/core/page_pool.c +2
-0@@ -393,6 +390,15 @@ bool __page_pool_request_shutdown(structpage_pool *pool) * be in-flight. */ __page_pool_empty_ring (pool); +} +EXPORT_SYMBOL (__page_pool_flush);A later patch removes this, do we actually need it here?
I agree, Jonathan changed the design of my last patch in this series and this became redundant as he is going to do lazy release of unwanted pages, rather than flushing the cache.
Quoted 18 lines
Quoted 15 lines
net/core/page_pool.c +2
-0+ +/* Request to shutdown: release pages cached by page_pool, and check
+ * for in-flight pages
+ */net/core/page_pool.c +3
-0+bool __page_pool_request_shutdown (struct page_pool *pool) +{ +__page_pool_flush (pool);return __page_pool_safe_to_destroy (pool); }Signature · 1 line
2.17.1Thanks
/Ilias
Re: [PATCH 04/10 net-next] page_pool: Add API to update numa node and flush page caches
Quoted 85 lines · Saeed Mahameed
Quoted 30 lines · Ilias Apalodimas
Hi Saeed,
Quoted 27 lines · Jonathan Lemon
From: Saeed Mahameed <[email protected]>
Add
page_pool_update_nid()to be called from drivers when they detect
numa node changes.It will do:
1) Flush the pool's page cache andptr_ring.
2) Update page pool nid value to start allocating from the new numa node.Signed-off-by Saeed Mahameed <[email protected]>Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 10 ++++++++++ net/core/page_pool.c | 16 +++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-)include/net/page_pool.h +0
-0@@ -226,4 +226,14 @@ static inline bool page_pool_put(struct page_pool *pool)return refcount_dec_and_test (&pool->user_cnt); }Quoted 7 lines
Quoted 3 lines
include/net/page_pool.h +1
-0+/* Only safe from napi context or when user guarantees it is thread safe */
+void__page_pool_flush(struct page_pool *pool);This should be called per packet right? Any noticeable impact on performance?
no, once per napi and only if a change in numa node is detected, so very very rare !
Quoted 27 lines
Quoted 27 lines
+static inline void
page_pool_update_nid(struct page_pool *pool, int new_nid)include/net/page_pool.h +7
-0+{ +if (unlikely (pool->p.nid != new_nid)) { +/* TODO: Add statistics/trace */ +__page_pool_flush (pool); + pool->p.nid = new_nid; + } +}#endif /* _NET_PAGE_POOL_H */ net/core/page_pool.c +0
-1@@ -373,16 +373,13 @@ void __page_pool_free(struct page_pool *pool) }EXPORT_SYMBOL (__page_pool_free); -/* Request to shutdown: release pages cached by page_pool, and check
- * for in-flight pages
- */net/core/page_pool.c +1
-1-bool __page_pool_request_shutdown (struct page_pool *pool) +void __page_pool_flush (struct page_pool *pool) {struct page *page;Quoted 16 lines
Quoted 8 lines
net/core/page_pool.c +1
-1/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be -* call concurrently. +* called concurrently. */ while (pool->alloc.count) { page = pool->alloc.cache[--pool->alloc.count];net/core/page_pool.c bool __page_pool_request_shutdown(struct · line 395
Quoted 6 lines
net/core/page_pool.c +2
-0@@ -393,6 +390,15 @@ bool __page_pool_request_shutdown(structpage_pool *pool) * be in-flight. */ __page_pool_empty_ring (pool); +} +EXPORT_SYMBOL (__page_pool_flush);A later patch removes this, do we actually need it here?
Quoted 3 lines
I agree, Jonathan changed the design of my last patch in this series and this became redundant as he is going to do lazy release of unwanted pages, rather than flushing the cache.
Yeah - I didn't want to take the latency hit when the node changed, and would prefer to just amortize the cost over time.
Signature · 1 line
Jonathan
Quoted 19 lines
Quoted 19 lines
Quoted 16 lines
net/core/page_pool.c +2
-0+ +/* Request to shutdown: release pages cached by page_pool, and check
+ * for in-flight pages
+ */net/core/page_pool.c +3
-0+bool __page_pool_request_shutdown (struct page_pool *pool) +{ +__page_pool_flush (pool);return __page_pool_safe_to_destroy (pool); }Signature · 1 line
2.17.1Thanks
/Ilias
PATCH 5/10 net/mlx5e: Rx, Update page pool numa node when changed
In reply to Jonathan LemonFrom: Saeed Mahameed <[email protected]>
On poll rx ring napi cycle check if numa node is different than the page pool nid and update it using page_pool_update_nid();
--- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +3 -0
@@ -1300,6 +1300,9 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)if (unlikely (!test_bit (MLX5E_RQ_STATE_ENABLED , &rq->state)))return 0 ; +if (rq->page_pool) +page_pool_update_nid (rq->page_pool,numa_mem_id ()); +if (rq->cqd.left) work_done +=mlx5e_decompress_cqes_cont (rq, cqwq,0 , budget);
Signature · 1 line
2.17.1
PATCH 6/10 page_pool: Add page_pool_keep_page
In reply to Jonathan LemonWhen releasing a page to the pool, only retain the page if page_pool_keep_page() returns true.
Do not flush the page pool on node changes, but instead lazily discard the pages as they are returned.
--- include/net/page_pool.h | 2 -- net/core/page_pool.c | 39 +++++++++++++++++++++++---------------- 2 files changed, 23 insertions(+), 18 deletions(-)
include/net/page_pool.h +0 -2
@@ -227,12 +227,10 @@ static inline bool page_pool_put(struct page_pool *pool) }/* Only safe from napi context or when user guarantees it is thread safe */ -void __page_pool_flush (struct page_pool *pool);static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) {if (unlikely (pool->p.nid != new_nid)) {/* TODO: Add statistics/trace */ -__page_pool_flush (pool); pool->p.nid = new_nid; } }
net/core/page_pool.c +23 -16
@@ -258,6 +258,7 @@ static bool __page_pool_recycle_into_ring(struct page_pool *pool, struct page *page) {int ret; +/* BH protection not needed if current is serving softirq */ if (in_serving_softirq ()) ret =ptr_ring_produce (&pool->ring, page);
@@ -272,8 +273,8 @@ static bool __page_pool_recycle_into_ring(struct page_pool *pool,* * Caller must provide appropriate safe context. */ -static bool __page_pool_recycle_direct (struct page *page, -struct page_pool *pool) +static bool __page_pool_recycle_into_cache (struct page *page, +struct page_pool *pool) {if (unlikely (pool->alloc.count ==PP_ALLOC_CACHE_SIZE ))return false;
@@ -283,27 +284,35 @@ static bool __page_pool_recycle_direct(struct page *page,return true; } +/* Determine whether this page should be kept or returned +* +* refcnt == 1 means page_pool owns page. +*/ +static bool page_pool_keep_page (struct page_pool *pool,struct page *page) +{ +return page_ref_count (page) ==1 && +page_to_nid (page) == pool->p.nid && + !page_is_pfmemalloc (page); +} +void __page_pool_put_page (struct page_pool *pool,struct page *page,bool allow_direct) {/* This allocator is optimized for the XDP mode that uses * one-frame-per-page, but have fallbacks that act like the * regular page allocator APIs. -* -* refcnt == 1 means page_pool owns page, and can recycle it. */ -if (likely (page_ref_count (page) ==1 )) { +if (likely (page_pool_keep_page (pool, page))) {/* Read barrier done in page_ref_count / READ_ONCE */ if (allow_direct &&in_serving_softirq ()) -if (__page_pool_recycle_direct (page, pool)) +if (__page_pool_recycle_into_cache (page, pool))return ; -if (!__page_pool_recycle_into_ring (pool, page)) { -/* Cache full, fallback to free pages */ -__page_pool_return_page (pool, page); - } -return ; +if (__page_pool_recycle_into_ring (pool, page)) +return ; + +/* Cache full, fallback to return pages */ }/* Fallback/non-XDP mode: API user have elevated refcnt. *
@@ -318,8 +327,7 @@ void __page_pool_put_page(struct page_pool *pool,* doing refcnt based recycle tricks, meaning another process * will be invoking put_page. */ -__page_pool_clean_page (pool, page); -put_page (page); +__page_pool_return_page (pool, page); }EXPORT_SYMBOL (__page_pool_put_page);
@@ -373,7 +381,7 @@ void __page_pool_free(struct page_pool *pool) }EXPORT_SYMBOL (__page_pool_free); -void __page_pool_flush (struct page_pool *pool) +static void page_pool_flush (struct page_pool *pool) {struct page *page;
@@ -391,14 +399,13 @@ void __page_pool_flush(struct page_pool *pool)*/ __page_pool_empty_ring (pool); } -EXPORT_SYMBOL (__page_pool_flush);/* Request to shutdown: release pages cached by page_pool, and check * for in-flight pages */ bool __page_pool_request_shutdown (struct page_pool *pool) { -__page_pool_flush (pool); +page_pool_flush (pool);return __page_pool_safe_to_destroy (pool); }
Signature · 1 line
2.17.1
PATCH 7/10 page_pool: allow configurable linear cache size
In reply to Jonathan LemonSome drivers may utilize more than one page per RX work entry. Allow a configurable cache size, with the same defaults if the size is zero.
Convert magic numbers into descriptive entries.
Re-arrange the page_pool structure for efficiency.
--- include/net/page_pool.h | 50 ++++++++++++++++++++--------------------- net/core/page_pool.c | 49 +++++++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 45 deletions(-)
include/net/page_pool.h +25 -25
@@ -51,41 +51,34 @@* cache is already full (or partly full) then the XDP_DROP recycles * would have to take a slower code path. */ -#define PP_ALLOC_CACHE_SIZE 128#define PP_ALLOC_CACHE_REFILL 64 -struct pp_alloc_cache { -u32 count; -void *cache[PP_ALLOC_CACHE_SIZE ]; -}; +#define PP_ALLOC_CACHE_DEFAULT (2 * PP_ALLOC_CACHE_REFILL) +#define PP_ALLOC_CACHE_LIMIT 512 +#define PP_ALLOC_POOL_DEFAULT 1024 +#define PP_ALLOC_POOL_LIMIT 32768struct page_pool_params {unsigned int flags;unsigned int order;unsigned int pool_size; +unsigned int cache_size;int nid;/* Numa node id to allocate from pages from */ -struct device *dev;/* device, for DMA pre-mapping purposes */ enum dma_data_direction dma_dir;/* DMA mapping direction */ +struct device *dev;/* device, for DMA pre-mapping purposes */ };struct page_pool {struct page_pool_params p; +u32 alloc_count;u32 pages_state_hold_cnt; +atomic_t pages_state_release_cnt; -/* -* Data structure for allocation side -* -* Drivers allocation side usually already perform some kind -* of resource protection. Piggyback on this protection, and -* require driver to protect allocation side. -* -* For NIC drivers this means, allocate a page_pool per -* RX-queue. As the RX-queue is already protected by -* Softirq/BH scheduling and napi_schedule. NAPI schedule -* guarantee that a single napi_struct will only be scheduled -* on a single CPU (see napi_schedule). +/* A page_pool is strictly tied to a single RX-queue being +* protected by NAPI, due to above pp_alloc_cache. This +* refcnt serves purpose is to simplify drivers error handling. */ -struct pp_alloc_cache alloc____cacheline_aligned_in_smp ; +refcount_t user_cnt;/* Data structure for storing recycled pages. *
@@ -100,13 +93,20 @@ struct page_pool {*/ struct ptr_ring ring; -atomic_t pages_state_release_cnt; - -/* A page_pool is strictly tied to a single RX-queue being -* protected by NAPI, due to above pp_alloc_cache. This -* refcnt serves purpose is to simplify drivers error handling. +/* +* Data structure for allocation side +* +* Drivers allocation side usually already perform some kind +* of resource protection. Piggyback on this protection, and +* require driver to protect allocation side. +* +* For NIC drivers this means, allocate a page_pool per +* RX-queue. As the RX-queue is already protected by +* Softirq/BH scheduling and napi_schedule. NAPI schedule +* guarantee that a single napi_struct will only be scheduled +* on a single CPU (see napi_schedule). */ -refcount_t user_cnt; +void *alloc_cache[]; };struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);
net/core/page_pool.c +29 -20
114 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH 07/10 net-next] page_pool: allow configurable linear cache size
Quoted 7 lines · On Wed, 16 Oct 2019 15:50:25 -0700 Jonathan Lemon <[email protected]>…
Some drivers may utilize more than one page per RX work entry. Allow a configurable cache size, with the same defaults if the size is zero.
Convert magic numbers into descriptive entries.
Re-arrange the
page_poolstructure for efficiency.
IMHO the re-arrange you did does not improve efficiency, it kills the efficiency...
Quoted 32 lines
Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 50 ++++++++++++++++++++--------------------- net/core/page_pool.c | 49 +++++++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 45 deletions(-)include/net/page_pool.h +5
-6@@ -51,41 +51,34 @@* cache is already full (or partly full) then the XDP_DROP recycles * would have to take a slower code path. */ -#define PP_ALLOC_CACHE_SIZE 128#define PP_ALLOC_CACHE_REFILL 64 -struct pp_alloc_cache { -u32 count; -void *cache[PP_ALLOC_CACHE_SIZE ]; -}; +#define PP_ALLOC_CACHE_DEFAULT (2 * PP_ALLOC_CACHE_REFILL) +#define PP_ALLOC_CACHE_LIMIT 512 +#define PP_ALLOC_POOL_DEFAULT 1024 +#define PP_ALLOC_POOL_LIMIT 32768struct page_pool_params {unsigned int flags;unsigned int order;unsigned int pool_size; +unsigned int cache_size;int nid;/* Numa node id to allocate from pages from */ -struct device *dev;/* device, for DMA pre-mapping purposes */
include/net/page_pool.h line 75
Quoted 10 lines
include/net/page_pool.h +3
-0enum dma_data_direction dma_dir;/* DMA mapping direction */ +struct device *dev;/* device, for DMA pre-mapping purposes */ };struct page_pool {struct page_pool_params p; +u32 alloc_count;u32 pages_state_hold_cnt; +atomic_t pages_state_release_cnt;
The struct members pages_state_hold_cnt and pages_state_release_cnt, MUST be kept on different cachelines, else the point of tracking inflight pages this way is lost.
Quoted 4 lines
include/net/page_pool.h +0
-4-/* -* Data structure for allocation side -* -* Drivers allocation side usually already perform some kind
include/net/page_pool.h line 78
Quoted 10 lines
include/net/page_pool.h +2
-8-* of resource protection. Piggyback on this protection, and -* require driver to protect allocation side. -* -* For NIC drivers this means, allocate a page_pool per -* RX-queue. As the RX-queue is already protected by -* Softirq/BH scheduling and napi_schedule. NAPI schedule -* guarantee that a single napi_struct will only be scheduled -* on a single CPU (see napi_schedule). +/* A page_pool is strictly tied to a single RX-queue being +* protected by NAPI, due to above pp_alloc_cache. This
^^^^^^^^^^^^^^
You remove the 'pp_alloc_cache' in this patch, and still mention it in the comments.
Quoted 82 lines
include/net/page_pool.h +15
-7+* refcnt serves purpose is to simplify drivers error handling. */ -struct pp_alloc_cache alloc____cacheline_aligned_in_smp ; +refcount_t user_cnt;/* Data structure for storing recycled pages. * @@ -100,13 +93,20 @@ struct page_pool {*/ struct ptr_ring ring; -atomic_t pages_state_release_cnt; - -/* A page_pool is strictly tied to a single RX-queue being -* protected by NAPI, due to above pp_alloc_cache. This -* refcnt serves purpose is to simplify drivers error handling. +/* +* Data structure for allocation side +* +* Drivers allocation side usually already perform some kind +* of resource protection. Piggyback on this protection, and +* require driver to protect allocation side. +* +* For NIC drivers this means, allocate a page_pool per +* RX-queue. As the RX-queue is already protected by +* Softirq/BH scheduling and napi_schedule. NAPI schedule +* guarantee that a single napi_struct will only be scheduled +* on a single CPU (see napi_schedule). */ -refcount_t user_cnt; +void *alloc_cache[]; };struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);net/core/page_pool.c +6
-10@@ -18,22 +18,18 @@#include <trace/events/page_pool.h> -static int page_pool_init (struct page_pool *pool, -const struct page_pool_params *params) +static int page_pool_init (struct page_pool *pool) { -unsigned int ring_qsize =1024 ;/* Default */ - -memcpy (&pool->p, params,sizeof (pool->p));/* Validate only known flags were used */ if (pool->p.flags & ~(PP_FLAG_ALL ))return -EINVAL ; -if (pool->p.pool_size) - ring_qsize = pool->p.pool_size; +if (!pool->p.pool_size) + pool->p.pool_size =PP_ALLOC_POOL_DEFAULT ;/* Sanity limit mem that can be pinned down */ -if (ring_qsize >32768 ) +if (pool->p.pool_size >PP_ALLOC_POOL_LIMIT )return -E2BIG ;/* DMA direction is either DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. @@ -44,7 +40,7 @@ static int page_pool_init(struct page_pool *pool, (pool->p.dma_dir != DMA_BIDIRECTIONAL))return -EINVAL ; -if (ptr_ring_init (&pool->ring, ring_qsize,GFP_KERNEL ) <0 ) +if (ptr_ring_init (&pool->ring, pool->p.pool_size,GFP_KERNEL ) <0 )return -ENOMEM ;atomic_set (&pool->pages_state_release_cnt,0 );@@ -61,13 +57,26 @@ static int page_pool_init(struct page_pool *pool,struct page_pool *page_pool_create (const struct page_pool_params *params) {struct page_pool *pool; +u32 cache_size, size;int err; - pool =kzalloc_node (sizeof (*pool),GFP_KERNEL , params->nid);
net/core/page_pool.c static int page_pool_init(struct page_pool *pool, · line 72
Quoted 10 lines
net/core/page_pool.c +10
-0+ cache_size = params->cache_size; +if (!cache_size) + cache_size =PP_ALLOC_CACHE_DEFAULT ; + +/* Sanity limit mem that can be pinned down */ +if (cache_size >PP_ALLOC_CACHE_LIMIT ) +return ERR_PTR (-E2BIG ); + + size =sizeof (*pool) + cache_size *sizeof (void *); + pool =kzalloc_node (size,GFP_KERNEL , params->nid);
You have now placed alloc_cache at the end of page_pool struct, this allows for dynamic changing the size, that is kind of nice. Before I placed pp_alloc_cache in struct to make sure that we didn't have false-sharing. Now, I'm unsure if false-sharing can happen? (depend on alignment from kzalloc_node).
Quoted 68 lines
net/core/page_pool.c +13
-10if (!pool)return ERR_PTR (-ENOMEM ); - err =page_pool_init (pool, params); +memcpy (&pool->p, params,sizeof (pool->p)); + pool->p.cache_size = cache_size; + + err =page_pool_init (pool);if (err <0 ) {pr_warn ("%s() gave up with errno %d\n" , __func__, err);kfree (pool);@@ -87,9 +96,9 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)/* Test for safe-context, caller should provide this guarantee */ if (likely (in_serving_softirq ())) { -if (likely (pool->alloc.count)) { +if (likely (pool->alloc_count)) {/* Fast-path */ - page = pool->alloc.cache[--pool->alloc.count]; + page = pool->alloc_cache[--pool->alloc_count];return page; } refill = true;@@ -105,8 +114,8 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)spin_lock (&r->consumer_lock); page =__ptr_ring_consume (r);if (refill) - pool->alloc.count =__ptr_ring_consume_batched (r, - pool->alloc.cache, + pool->alloc_count =__ptr_ring_consume_batched (r, + pool->alloc_cache,PP_ALLOC_CACHE_REFILL );spin_unlock (&r->consumer_lock);return page;@@ -276,11 +285,11 @@ static bool __page_pool_recycle_into_ring(struct page_pool *pool,static bool __page_pool_recycle_into_cache (struct page *page,struct page_pool *pool) { -if (unlikely (pool->alloc.count ==PP_ALLOC_CACHE_SIZE )) +if (unlikely (pool->alloc_count == pool->p.cache_size))return false;/* Caller MUST have verified/know (page_ref_count(page) == 1) */ - pool->alloc.cache[pool->alloc.count++] = page; + pool->alloc_cache[pool->alloc_count++] = page;return true; }@@ -365,7 +374,7 @@ void __page_pool_free(struct page_pool *pool)if (!page_pool_put (pool))return ; -WARN (pool->alloc.count,"API usage violation" ); +WARN (pool->alloc_count,"API usage violation" );WARN (!ptr_ring_empty (&pool->ring),"ptr_ring is not empty" );/* Can happen due to forced shutdown */ @@ -389,8 +398,8 @@ static void page_pool_flush(struct page_pool *pool)* no-longer in use, and page_pool_alloc_pages() cannot be * called concurrently. */ -while (pool->alloc.count) { - page = pool->alloc.cache[--pool->alloc.count]; +while (pool->alloc_count) { + page = pool->alloc_cache[--pool->alloc_count];__page_pool_return_page (pool, page); }
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: [PATCH 07/10 net-next] page_pool: allow configurable linear cache size
Quoted 55 lines · Jesper Dangaard Brouer
Quoted 7 lines · On Wed, 16 Oct 2019 15:50:25 -0700 Jonathan Lemon <[email protected]>…
Some drivers may utilize more than one page per RX work entry. Allow a configurable cache size, with the same defaults if the size is zero.
Convert magic numbers into descriptive entries.
Re-arrange the
page_poolstructure for efficiency.IMHO the re-arrange you did does not improve efficiency, it kills the efficiency...
Quoted 41 lines
Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 50 ++++++++++++++++++++--------------------- net/core/page_pool.c | 49 +++++++++++++++++++++++----------------- 2 files changed, 54 insertions(+), 45 deletions(-)include/net/page_pool.h +6
-6@@ -51,41 +51,34 @@* cache is already full (or partly full) then the XDP_DROP recycles * would have to take a slower code path. */ -#define PP_ALLOC_CACHE_SIZE 128#define PP_ALLOC_CACHE_REFILL 64 -struct pp_alloc_cache { -u32 count; -void *cache[PP_ALLOC_CACHE_SIZE ]; -}; +#define PP_ALLOC_CACHE_DEFAULT (2 * PP_ALLOC_CACHE_REFILL) +#define PP_ALLOC_CACHE_LIMIT 512 +#define PP_ALLOC_POOL_DEFAULT 1024 +#define PP_ALLOC_POOL_LIMIT 32768struct page_pool_params {unsigned int flags;unsigned int order;unsigned int pool_size; +unsigned int cache_size;int nid;/* Numa node id to allocate from pages from */ -struct device *dev;/* device, for DMA pre-mapping purposes */ enum dma_data_direction dma_dir;/* DMA mapping direction */ +struct device *dev;/* device, for DMA pre-mapping purposes */ };struct page_pool {struct page_pool_params p;
Quoted 7 lines
include/net/page_pool.h line 75
Quoted 3 lines
include/net/page_pool.h +2
-0+u32 alloc_count;u32 pages_state_hold_cnt; +atomic_t pages_state_release_cnt;The struct members
pages_state_hold_cntandpages_state_release_cnt, MUST be kept on different cachelines, else the point of tracking inflight pages this way is lost.
Will update, but there are other items that make this lost in the noise.
Quoted 15 lines
Quoted 4 lines
include/net/page_pool.h +0
-4-/* -* Data structure for allocation side -* -* Drivers allocation side usually already perform some kind include/net/page_pool.h line 78
Quoted 10 lines
include/net/page_pool.h +2
-8-* of resource protection. Piggyback on this protection, and -* require driver to protect allocation side. -* -* For NIC drivers this means, allocate a page_pool per -* RX-queue. As the RX-queue is already protected by -* Softirq/BH scheduling and napi_schedule. NAPI schedule -* guarantee that a single napi_struct will only be scheduled -* on a single CPU (see napi_schedule). +/* A page_pool is strictly tied to a single RX-queue being +* protected by NAPI, due to above pp_alloc_cache. This ^^^^^^^^^^^^^^
Quoted 2 lines
You remove the '
pp_alloc_cache' in this patch, and still mention it in the comments.
Will fix.
Quoted 183 lines
Quoted 84 lines
include/net/page_pool.h +15
-7+* refcnt serves purpose is to simplify drivers error handling. */ -struct pp_alloc_cache alloc____cacheline_aligned_in_smp ; +refcount_t user_cnt;/* Data structure for storing recycled pages. * @@ -100,13 +93,20 @@ struct page_pool {*/ struct ptr_ring ring; -atomic_t pages_state_release_cnt; - -/* A page_pool is strictly tied to a single RX-queue being -* protected by NAPI, due to above pp_alloc_cache. This -* refcnt serves purpose is to simplify drivers error handling. +/* +* Data structure for allocation side +* +* Drivers allocation side usually already perform some kind +* of resource protection. Piggyback on this protection, and +* require driver to protect allocation side. +* +* For NIC drivers this means, allocate a page_pool per +* RX-queue. As the RX-queue is already protected by +* Softirq/BH scheduling and napi_schedule. NAPI schedule +* guarantee that a single napi_struct will only be scheduled +* on a single CPU (see napi_schedule). */ -refcount_t user_cnt; +void *alloc_cache[]; }; struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_tgfp);
net/core/page_pool.c +6
-10@@ -18,22 +18,18 @@#include <trace/events/page_pool.h> -static int page_pool_init (struct page_pool *pool, -const struct page_pool_params *params) +static int page_pool_init (struct page_pool *pool) { -unsigned int ring_qsize =1024 ;/* Default */ - -memcpy (&pool->p, params,sizeof (pool->p));/* Validate only known flags were used */ if (pool->p.flags & ~(PP_FLAG_ALL ))return -EINVAL ; -if (pool->p.pool_size) - ring_qsize = pool->p.pool_size; +if (!pool->p.pool_size) + pool->p.pool_size =PP_ALLOC_POOL_DEFAULT ;/* Sanity limit mem that can be pinned down */ -if (ring_qsize >32768 ) +if (pool->p.pool_size >PP_ALLOC_POOL_LIMIT )return -E2BIG ;/* DMA direction is either DMA_FROM_DEVICE or DMA_BIDIRECTIONAL. @@ -44,7 +40,7 @@ static int page_pool_init(struct page_pool *pool, (pool->p.dma_dir != DMA_BIDIRECTIONAL))return -EINVAL ; -if (ptr_ring_init (&pool->ring, ring_qsize,GFP_KERNEL ) <0 ) +if (ptr_ring_init (&pool->ring, pool->p.pool_size,GFP_KERNEL ) <0 )return -ENOMEM ;atomic_set (&pool->pages_state_release_cnt,0 );@@ -61,13 +57,26 @@ static int page_pool_init(struct page_pool *pool,struct page_pool *page_pool_create (const struct page_pool_params *params) {struct page_pool *pool; +u32 cache_size, size;int err; - pool =kzalloc_node (sizeof (*pool),GFP_KERNEL , params->nid);net/core/page_pool.c static int page_pool_init(struct page_pool *pool, · line 73
Quoted 10 lines
net/core/page_pool.c +10
-0+ cache_size = params->cache_size; +if (!cache_size) + cache_size =PP_ALLOC_CACHE_DEFAULT ; + +/* Sanity limit mem that can be pinned down */ +if (cache_size >PP_ALLOC_CACHE_LIMIT ) +return ERR_PTR (-E2BIG ); + + size =sizeof (*pool) + cache_size *sizeof (void *); + pool =kzalloc_node (size,GFP_KERNEL , params->nid);You have now placed
alloc_cacheat the end ofpage_poolstruct, this allows for dynamic changing the size, that is kind of nice. Before I placedpp_alloc_cachein struct to make sure that we didn't have false-sharing. Now, I'm unsure if false-sharing can happen? (depend on alignment fromkzalloc_node).Quoted 72 lines
net/core/page_pool.c +4
-1if (!pool)return ERR_PTR (-ENOMEM ); - err =page_pool_init (pool, params); +memcpy (&pool->p, params,sizeof (pool->p)); + pool->p.cache_size = cache_size; + + err =page_pool_init (pool);if (err <0 ) {pr_warn ("%s() gave up with errno %d\n" , __func__, err);net/core/page_pool.c +2
-2kfree (pool);@@ -87,9 +96,9 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)/* Test for safe-context, caller should provide this guarantee */ if (likely (in_serving_softirq ())) { -if (likely (pool->alloc.count)) { +if (likely (pool->alloc_count)) {/* Fast-path */ - page = pool->alloc.cache[--pool->alloc.count]; + page = pool->alloc_cache[--pool->alloc_count];return page; }net/core/page_pool.c +2
-2refill = true;@@ -105,8 +114,8 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) spin_lock(&r->consumer_lock); page =__ptr_ring_consume (r);if (refill) - pool->alloc.count =__ptr_ring_consume_batched (r, - pool->alloc.cache, + pool->alloc_count =__ptr_ring_consume_batched (r, + pool->alloc_cache,PP_ALLOC_CACHE_REFILL );spin_unlock (&r->consumer_lock);net/core/page_pool.c +5
-5return page;@@ -276,11 +285,11 @@ static bool __page_pool_recycle_into_ring(struct page_pool *pool, static bool __page_pool_recycle_into_cache(struct page *page, struct page_pool *pool) { - if (unlikely(pool->alloc.count == PP_ALLOC_CACHE_SIZE)) + if (unlikely(pool->alloc_count == pool->p.cache_size)) return false;/* Caller MUST have verified/know (page_ref_count(page) == 1) */ - pool->alloc.cache[pool->alloc.count++] = page; + pool->alloc_cache[pool->alloc_count++] = page;return true; }@@ -365,7 +374,7 @@ void __page_pool_free(struct page_pool *pool)if (!page_pool_put (pool))return ; -WARN (pool->alloc.count,"API usage violation" ); +WARN (pool->alloc_count,"API usage violation" );WARN (!ptr_ring_empty (&pool->ring),"ptr_ring is not empty" );/* Can happen due to forced shutdown */ @@ -389,8 +398,8 @@ static void page_pool_flush(struct page_pool*pool) * no-longer in use, and page_pool_alloc_pages() cannot be * called concurrently. */ -while (pool->alloc.count) { - page = pool->alloc.cache[--pool->alloc.count]; +while (pool->alloc_count) { + page = pool->alloc_cache[--pool->alloc_count];__page_pool_return_page (pool, page); }Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
PATCH 8/10 page_pool: Add statistics
In reply to Jonathan LemonAdd statistics to the page pool, providing visibility into its operation.
Callers can provide a location where the stats are stored, otherwise the page pool will allocate a statistic area.
--- include/net/page_pool.h | 21 +++++++++++++--- net/core/page_pool.c | 55 +++++++++++++++++++++++++++++++++++------ 2 files changed, 65 insertions(+), 11 deletions(-)
include/net/page_pool.h +18 -3
@@ -34,8 +34,11 @@#include <linux/ptr_ring.h> #include <linux/dma-direction.h> -#define PP_FLAG_DMA_MAP 1/* Should page_pool do the DMA map/unmap */ -#define PP_FLAG_ALL PP_FLAG_DMA_MAP +#define PP_FLAG_DMA_MAP BIT(0)/* page_pool does the DMA map/unmap */ +#define PP_FLAG_ALL (PP_FLAG_DMA_MAP) + +/* internal flags, not expoed to user */ +#define PP_FLAG_INTERNAL_STATS BIT(8)/* * Fast allocation side cache array/stack
@@ -57,6 +60,17 @@#define PP_ALLOC_POOL_DEFAULT 1024#define PP_ALLOC_POOL_LIMIT 32768 +struct page_pool_stats { +u64 cache_hit; +u64 cache_full; +u64 cache_empty; +u64 ring_produce; +u64 ring_consume; +u64 ring_return; +u64 flush; +u64 node_change; +}; +struct page_pool_params {unsigned int flags;unsigned int order;
@@ -65,6 +79,7 @@ struct page_pool_params {int nid;/* Numa node id to allocate from pages from */ enum dma_data_direction dma_dir;/* DMA mapping direction */ struct device *dev;/* device, for DMA pre-mapping purposes */ +struct page_pool_stats *stats;/* pool stats stored externally */ };struct page_pool {
@@ -230,8 +245,8 @@ static inline bool page_pool_put(struct page_pool *pool)static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) {if (unlikely (pool->p.nid != new_nid)) { -/* TODO: Add statistics/trace */ pool->p.nid = new_nid; + pool->p.stats->node_change++; } }#endif /* _NET_PAGE_POOL_H */
net/core/page_pool.c +47 -8
138 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH 08/10 net-next] page_pool: Add statistics
Quoted 30 lines · On Wed, 16 Oct 2019 15:50:26 -0700 Jonathan Lemon <[email protected]>…
Add statistics to the page pool, providing visibility into its operation.
Callers can provide a location where the stats are stored, otherwise the page pool will allocate a statistic area.
Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 21 +++++++++++++--- net/core/page_pool.c | 55 +++++++++++++++++++++++++++++++++++------ 2 files changed, 65 insertions(+), 11 deletions(-)include/net/page_pool.h +5
-2@@ -34,8 +34,11 @@#include <linux/ptr_ring.h> #include <linux/dma-direction.h> -#define PP_FLAG_DMA_MAP 1/* Should page_pool do the DMA map/unmap */ -#define PP_FLAG_ALL PP_FLAG_DMA_MAP +#define PP_FLAG_DMA_MAP BIT(0)/* page_pool does the DMA map/unmap */ +#define PP_FLAG_ALL (PP_FLAG_DMA_MAP) + +/* internal flags, not expoed to user */ +#define PP_FLAG_INTERNAL_STATS BIT(8)/* * Fast allocation side cache array/stack
include/net/page_pool.h line 68
Quoted 9 lines
include/net/page_pool.h +6
-0@@ -57,6 +60,17 @@#define PP_ALLOC_POOL_DEFAULT 1024#define PP_ALLOC_POOL_LIMIT 32768 +struct page_pool_stats { +u64 cache_hit; +u64 cache_full; +u64 cache_empty; +u64 ring_produce; +u64 ring_consume;
You are placing producer and consumer counters on the same cache-line. This is not acceptable.
The page pool and ptr_ring, are specifically designed to avoid cache-line contention between consumers and producers. This patch kills that work.
include/net/page_pool.h line 72
Quoted 4 lines
include/net/page_pool.h +4
-0+u64 ring_return; +u64 flush; +u64 node_change; +};
Another example of carefully avoiding cache-line bouncing is the inflight tracking, e.g. the struct placement of pages_state_release_cnt and pages_state_hold_cnt for inflight accounting.
Quoted 167 lines
include/net/page_pool.h +3
-1+struct page_pool_params {unsigned int flags;unsigned int order;@@ -65,6 +79,7 @@ struct page_pool_params {int nid;/* Numa node id to allocate from pages from */ enum dma_data_direction dma_dir;/* DMA mapping direction */ struct device *dev;/* device, for DMA pre-mapping purposes */ +struct page_pool_stats *stats;/* pool stats stored externally */ };struct page_pool {@@ -230,8 +245,8 @@ static inline bool page_pool_put(struct page_pool *pool)static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) {if (unlikely (pool->p.nid != new_nid)) { -/* TODO: Add statistics/trace */ pool->p.nid = new_nid; + pool->p.stats->node_change++; } }#endif /* _NET_PAGE_POOL_H */ net/core/page_pool.c +45
-8@@ -20,9 +20,10 @@static int page_pool_init (struct page_pool *pool) { +int size;/* Validate only known flags were used */ -if (pool->p.flags & ~(PP_FLAG_ALL )) +if (pool->p.flags & ~PP_FLAG_ALL )return -EINVAL ;if (!pool->p.pool_size)@@ -40,8 +41,16 @@ static int page_pool_init(struct page_pool *pool) (pool->p.dma_dir != DMA_BIDIRECTIONAL))return -EINVAL ; +if (!pool->p.stats) { + size =sizeof (struct page_pool_stats ); + pool->p.stats =kzalloc_node (size,GFP_KERNEL , pool->p.nid); +if (!pool->p.stats) +return -ENOMEM ; + pool->p.flags |=PP_FLAG_INTERNAL_STATS ; + } +if (ptr_ring_init (&pool->ring, pool->p.pool_size,GFP_KERNEL ) <0 ) -return -ENOMEM ; + goto fail;atomic_set (&pool->pages_state_release_cnt,0 );@@ -52,6 +61,12 @@ static int page_pool_init(struct page_pool *pool)get_device (pool->p.dev);return 0 ; + +fail: +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); + +return -ENOMEM ; }struct page_pool *page_pool_create (const struct page_pool_params *params)@@ -98,9 +113,11 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)if (likely (in_serving_softirq ())) {if (likely (pool->alloc_count)) {/* Fast-path */ + pool->p.stats->cache_hit++; page = pool->alloc_cache[--pool->alloc_count];return page; } + pool->p.stats->cache_empty++; refill = true; }@@ -113,10 +130,13 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)*/ spin_lock (&r->consumer_lock); page =__ptr_ring_consume (r); -if (refill) +if (refill) { pool->alloc_count =__ptr_ring_consume_batched (r, pool->alloc_cache,PP_ALLOC_CACHE_REFILL ); + pool->p.stats->ring_consume += pool->alloc_count; + } + pool->p.stats->ring_consume += !!page;spin_unlock (&r->consumer_lock);return page; }@@ -266,15 +286,23 @@ static void __page_pool_return_page(struct page_pool *pool, struct page *page)static bool __page_pool_recycle_into_ring (struct page_pool *pool,struct page *page) { +struct ptr_ring *r = &pool->ring;int ret; -/* BH protection not needed if current is serving softirq */ if (in_serving_softirq ()) - ret =ptr_ring_produce (&pool->ring, page); +spin_lock (&r->producer_lock);else - ret =ptr_ring_produce_bh (&pool->ring, page); +spin_lock_bh (&r->producer_lock); -return (ret ==0 ) ? true : false; + ret =__ptr_ring_produce (r, page); + pool->p.stats->ring_produce++; + +if (in_serving_softirq ()) +spin_unlock (&r->producer_lock); +else +spin_unlock_bh (&r->producer_lock); + +return ret ==0 ; }/* Only allow direct recycling in special circumstances, into the @@ -285,8 +313,10 @@ static bool __page_pool_recycle_into_ring(struct page_pool *pool,static bool __page_pool_recycle_into_cache (struct page *page,struct page_pool *pool) { -if (unlikely (pool->alloc_count == pool->p.cache_size)) +if (unlikely (pool->alloc_count == pool->p.cache_size)) { + pool->p.stats->cache_full++;return false; + }/* Caller MUST have verified/know (page_ref_count(page) == 1) */ pool->alloc_cache[pool->alloc_count++] = page;@@ -343,6 +373,7 @@ EXPORT_SYMBOL(__page_pool_put_page);static void __page_pool_empty_ring (struct page_pool *pool) {struct page *page; +int count =0 ;/* Empty recycle ring */ while ((page =ptr_ring_consume_bh (&pool->ring))) {@@ -351,8 +382,11 @@ static void __page_pool_empty_ring(struct page_pool *pool)pr_crit ("%s() page_pool refcnt %d violation\n" , __func__,page_ref_count (page)); + count++;__page_pool_return_page (pool, page); } + + pool->p.stats->ring_return += count; }static void __warn_in_flight (struct page_pool *pool)@@ -381,6 +415,9 @@ void __page_pool_free(struct page_pool *pool)if (!__page_pool_safe_to_destroy (pool))__warn_in_flight (pool); +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); +ptr_ring_cleanup (&pool->ring,NULL );if (pool->p.flags &PP_FLAG_DMA_MAP )
net/core/page_pool.c static void page_pool_flush(struct page_pool *pool) · line 438
Quoted 8 lines
net/core/page_pool.c +2
-0@@ -394,6 +431,8 @@ static void page_pool_flush(struct page_pool *pool) {struct page *page; + pool->p.stats->flush++; +/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be * called concurrently.
Have you benchmarked the overhead of adding this accounting?
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: [PATCH 08/10 net-next] page_pool: Add statistics
In reply to Jonathan LemonQuoted 125 lines · Jonathan Lemon
Add statistics to the page pool, providing visibility into its operation.
Callers can provide a location where the stats are stored, otherwise the page pool will allocate a statistic area.
Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 21 +++++++++++++--- net/core/page_pool.c | 55 +++++++++++++++++++++++++++++++++++-- ---- 2 files changed, 65 insertions(+), 11 deletions(-)include/net/page_pool.h +1
-2@@ -34,8 +34,11 @@#include <linux/ptr_ring.h> #include <linux/dma-direction.h> -#define PP_FLAG_DMA_MAP 1/* Should page_pool do the DMA map/unmap */ -#define PP_FLAG_ALL PP_FLAG_DMA_MAP +#define PP_FLAG_DMA_MAP BIT(0)/* page_pool does the DMA map/unmap */
include/net/page_pool.h +15
-0+#define PP_FLAG_ALL (PP_FLAG_DMA_MAP) + +/* internal flags, not expoed to user */ +#define PP_FLAG_INTERNAL_STATS BIT(8)/* * Fast allocation side cache array/stack @@ -57,6 +60,17 @@#define PP_ALLOC_POOL_DEFAULT 1024#define PP_ALLOC_POOL_LIMIT 32768 +struct page_pool_stats { +u64 cache_hit; +u64 cache_full; +u64 cache_empty; +u64 ring_produce; +u64 ring_consume; +u64 ring_return; +u64 flush; +u64 node_change; +}; +struct page_pool_params {unsigned int flags;unsigned int order;@@ -65,6 +79,7 @@ struct page_pool_params {int nid;/* Numa node id to allocate from pages from */
include/net/page_pool.h +0
-0enum dma_data_direction dma_dir;/* DMA mapping direction */ struct device *dev;/* device, for DMA pre-mapping purposes */
include/net/page_pool.h +1
-0+struct page_pool_stats *stats;/* pool stats stored externally */
};struct
page_pool{include/net/page_pool.h +1
-1@@ -230,8 +245,8 @@ static inline bool page_pool_put(struct page_pool *pool)static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) {if (unlikely (pool->p.nid != new_nid)) { -/* TODO: Add statistics/trace */ pool->p.nid = new_nid; + pool->p.stats->node_change++; }}
#endif /*_NET_PAGE_POOL_H*/net/core/page_pool.c +11
-2@@ -20,9 +20,10 @@static int page_pool_init (struct page_pool *pool) { +int size;/* Validate only known flags were used */ -if (pool->p.flags & ~(PP_FLAG_ALL )) +if (pool->p.flags & ~PP_FLAG_ALL )return -EINVAL ;if (!pool->p.pool_size)@@ -40,8 +41,16 @@ static int page_pool_init(struct page_pool *pool) (pool->p.dma_dir != DMA_BIDIRECTIONAL))return -EINVAL ; +if (!pool->p.stats) { + size =sizeof (struct page_pool_stats ); + pool->p.stats =kzalloc_node (size,GFP_KERNEL , pool- >p.nid); +if (!pool->p.stats) +return -ENOMEM ; + pool->p.flags |=PP_FLAG_INTERNAL_STATS ; + } +if (ptr_ring_init (&pool->ring, pool->p.pool_size,GFP_KERNEL ) <0 ) -return -ENOMEM ; + goto fail;net/core/page_pool.c +6
-0atomic_set (&pool->pages_state_release_cnt,0 );@@ -52,6 +61,12 @@ static int page_pool_init(struct page_pool *pool)get_device (pool->p.dev);return 0 ; + +fail: +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); + +return -ENOMEM ; } struct page_pool *page_pool_create(const struct page_pool_params*params)
net/core/page_pool.c static struct page *__page_pool_get_cached(struct · line 121
Quoted 9 lines
net/core/page_pool.c +2
-0@@ -98,9 +113,11 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) if (likely(in_serving_softirq())) {if (likely (pool->alloc_count)) {/* Fast-path */ + pool->p.stats->cache_hit++; page = pool->alloc_cache[--pool->alloc_count];return page; } + pool->p.stats->cache_empty++;
this is problematic for 32bit SMP archs, you need to use u64_stats_sync API.
in mlx5 we only support 64bits, unlike page cache which must be protected here.
Quoted 105 lines
net/core/page_pool.c +4
-1refill = true; }@@ -113,10 +130,13 @@ static struct page*__page_pool_get_cached(struct page_pool *pool) */ spin_lock (&r->consumer_lock); page =__ptr_ring_consume (r); -if (refill) +if (refill) { pool->alloc_count =__ptr_ring_consume_batched (r, pool- >alloc_cache,PP_ALLOC_CACHE_ REFILL); + pool->p.stats->ring_consume += pool->alloc_count; + } + pool->p.stats->ring_consume += !!page;net/core/page_pool.c +12
-4spin_unlock (&r->consumer_lock);return page; }@@ -266,15 +286,23 @@ static void __page_pool_return_page(struct page_pool *pool, struct page *page) static bool __page_pool_recycle_into_ring(struct page_pool *pool, struct page *page) { + struct ptr_ring *r = &pool->ring;int ret; -/* BH protection not needed if current is serving softirq */ if (in_serving_softirq ()) - ret =ptr_ring_produce (&pool->ring, page); +spin_lock (&r->producer_lock);else - ret =ptr_ring_produce_bh (&pool->ring, page); +spin_lock_bh (&r->producer_lock); -return (ret ==0 ) ? true : false; + ret =__ptr_ring_produce (r, page); + pool->p.stats->ring_produce++; + +if (in_serving_softirq ()) +spin_unlock (&r->producer_lock); +else +spin_unlock_bh (&r->producer_lock); + +return ret ==0 ; }/* Only allow direct recycling in special circumstances, into the
net/core/page_pool.c +3
-1@@ -285,8 +313,10 @@ static bool __page_pool_recycle_into_ring(structpage_pool *pool, static bool __page_pool_recycle_into_cache(struct page *page, struct page_pool *pool) { - if (unlikely(pool->alloc_count == pool->p.cache_size)) + if (unlikely(pool->alloc_count == pool->p.cache_size)) { + pool->p.stats->cache_full++;return false; + }/* Caller MUST have verified/know (page_ref_count(page) == 1) */
net/core/page_pool.c +4
-0pool->alloc_cache[pool->alloc_count++] = page;@@ -343,6 +373,7 @@ EXPORT_SYMBOL(__page_pool_put_page);static void __page_pool_empty_ring (struct page_pool *pool) {struct page *page; +int count =0 ;/* Empty recycle ring */ while ((page =ptr_ring_consume_bh (&pool->ring))) {@@ -351,8 +382,11 @@ static void __page_pool_empty_ring(struct page_pool *pool) pr_crit("%s() page_pool refcnt %d violation\n", __func__, page_ref_count(page)); + count++;__page_pool_return_page (pool, page); } + + pool->p.stats->ring_return += count; }static void
__warn_in_flight(struct page_pool *pool)net/core/page_pool.c +5
-0@@ -381,6 +415,9 @@ void __page_pool_free(struct page_pool *pool)if (!__page_pool_safe_to_destroy (pool))__warn_in_flight (pool); +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); +ptr_ring_cleanup (&pool->ring,NULL );if (pool->p.flags &PP_FLAG_DMA_MAP )@@ -394,6 +431,8 @@ static void page_pool_flush(struct page_pool *pool) {struct page *page; + pool->p.stats->flush++; +/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be * called concurrently.
Re: [PATCH 08/10 net-next] page_pool: Add statistics
Quoted 132 lines · Saeed Mahameed
Quoted 131 lines · Jonathan Lemon
Add statistics to the page pool, providing visibility into its operation.
Callers can provide a location where the stats are stored, otherwise the page pool will allocate a statistic area.
Signed-off-by Jonathan Lemon <[email protected]>--- include/net/page_pool.h | 21 +++++++++++++--- net/core/page_pool.c | 55 +++++++++++++++++++++++++++++++++++-- ---- 2 files changed, 65 insertions(+), 11 deletions(-)include/net/page_pool.h +1
-2@@ -34,8 +34,11 @@#include <linux/ptr_ring.h> #include <linux/dma-direction.h> -#define PP_FLAG_DMA_MAP 1/* Should page_pool do the DMA map/unmap */ -#define PP_FLAG_ALL PP_FLAG_DMA_MAP +#define PP_FLAG_DMA_MAP BIT(0)/* page_pool does the DMA map/unmap */
include/net/page_pool.h +15
-0+#define PP_FLAG_ALL (PP_FLAG_DMA_MAP) + +/* internal flags, not expoed to user */ +#define PP_FLAG_INTERNAL_STATS BIT(8)/* * Fast allocation side cache array/stack @@ -57,6 +60,17 @@#define PP_ALLOC_POOL_DEFAULT 1024#define PP_ALLOC_POOL_LIMIT 32768 +struct page_pool_stats { +u64 cache_hit; +u64 cache_full; +u64 cache_empty; +u64 ring_produce; +u64 ring_consume; +u64 ring_return; +u64 flush; +u64 node_change; +}; +struct page_pool_params {unsigned int flags;unsigned int order;@@ -65,6 +79,7 @@ struct page_pool_params {int nid;/* Numa node id to allocate from pages from */
include/net/page_pool.h +0
-0enum dma_data_direction dma_dir;/* DMA mapping direction */ struct device *dev;/* device, for DMA pre-mapping purposes */
include/net/page_pool.h +1
-0+struct page_pool_stats *stats;/* pool stats stored externally */
};struct
page_pool{include/net/page_pool.h +1
-1@@ -230,8 +245,8 @@ static inline bool page_pool_put(struct page_pool *pool)static inline void page_pool_update_nid (struct page_pool *pool,int new_nid) {if (unlikely (pool->p.nid != new_nid)) { -/* TODO: Add statistics/trace */ pool->p.nid = new_nid; + pool->p.stats->node_change++; }}
#endif /*_NET_PAGE_POOL_H*/net/core/page_pool.c +11
-2@@ -20,9 +20,10 @@static int page_pool_init (struct page_pool *pool) { +int size;/* Validate only known flags were used */ -if (pool->p.flags & ~(PP_FLAG_ALL )) +if (pool->p.flags & ~PP_FLAG_ALL )return -EINVAL ;if (!pool->p.pool_size)@@ -40,8 +41,16 @@ static int page_pool_init(struct page_pool *pool) (pool->p.dma_dir != DMA_BIDIRECTIONAL))return -EINVAL ; +if (!pool->p.stats) { + size =sizeof (struct page_pool_stats ); + pool->p.stats =kzalloc_node (size,GFP_KERNEL , pool- > p.nid); +if (!pool->p.stats) +return -ENOMEM ; + pool->p.flags |=PP_FLAG_INTERNAL_STATS ; + } +if (ptr_ring_init (&pool->ring, pool->p.pool_size,GFP_KERNEL ) <0 ) -return -ENOMEM ; + goto fail;net/core/page_pool.c +6
-0atomic_set (&pool->pages_state_release_cnt,0 );@@ -52,6 +61,12 @@ static int page_pool_init(struct page_pool *pool)get_device (pool->p.dev);return 0 ; + +fail: +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); + +return -ENOMEM ; } struct page_pool *page_pool_create(const struct page_pool_params*params)
net/core/page_pool.c +1
-0@@ -98,9 +113,11 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) if (likely(in_serving_softirq())) {if (likely (pool->alloc_count)) {/* Fast-path */ + pool->p.stats->cache_hit++; page = pool->alloc_cache[--pool->alloc_count];
Quoted 8 lines
net/core/page_pool.c static struct page *__page_pool_get_cached(struct page_pool *pool) · line 120
Quoted 3 lines
net/core/page_pool.c +1
-0return page; } + pool->p.stats->cache_empty++;this is problematic for 32bit SMP archs, you need to use
u64_stats_syncAPI.
in mlx5 we only support 64bits, unlike page cache which must be protected here.
Oooh, hmm.
I think Apple had the right idea and discarded 32-bits along with the iPhone 5.
Tempted to just make stats a NOP on 32-bit machines.
Signature · 1 line
Jonathan
Quoted 106 lines
Quoted 106 lines
net/core/page_pool.c +4
-1refill = true; }@@ -113,10 +130,13 @@ static struct page*__page_pool_get_cached(struct page_pool *pool) */ spin_lock (&r->consumer_lock); page =__ptr_ring_consume (r); -if (refill) +if (refill) { pool->alloc_count =__ptr_ring_consume_batched (r, pool- > alloc_cache,PP_ALLOC_CACHE_ REFILL); + pool->p.stats->ring_consume += pool->alloc_count; + } + pool->p.stats->ring_consume += !!page;net/core/page_pool.c +12
-4spin_unlock (&r->consumer_lock);return page; }@@ -266,15 +286,23 @@ static void __page_pool_return_page(struct page_pool *pool, struct page *page) static bool __page_pool_recycle_into_ring(struct page_pool *pool, struct page *page) { + struct ptr_ring *r = &pool->ring;int ret; -/* BH protection not needed if current is serving softirq */ if (in_serving_softirq ()) - ret =ptr_ring_produce (&pool->ring, page); +spin_lock (&r->producer_lock);else - ret =ptr_ring_produce_bh (&pool->ring, page); +spin_lock_bh (&r->producer_lock); -return (ret ==0 ) ? true : false; + ret =__ptr_ring_produce (r, page); + pool->p.stats->ring_produce++; + +if (in_serving_softirq ()) +spin_unlock (&r->producer_lock); +else +spin_unlock_bh (&r->producer_lock); + +return ret ==0 ; }/* Only allow direct recycling in special circumstances, into the
net/core/page_pool.c +3
-1@@ -285,8 +313,10 @@ static bool __page_pool_recycle_into_ring(structpage_pool *pool, static bool __page_pool_recycle_into_cache(struct page *page, struct page_pool *pool) { - if (unlikely(pool->alloc_count == pool->p.cache_size)) + if (unlikely(pool->alloc_count == pool->p.cache_size)) { + pool->p.stats->cache_full++;return false; + }/* Caller MUST have verified/know (page_ref_count(page) == 1) */
net/core/page_pool.c +4
-0pool->alloc_cache[pool->alloc_count++] = page;@@ -343,6 +373,7 @@ EXPORT_SYMBOL(__page_pool_put_page);static void __page_pool_empty_ring (struct page_pool *pool) {struct page *page; +int count =0 ;/* Empty recycle ring */ while ((page =ptr_ring_consume_bh (&pool->ring))) {@@ -351,8 +382,11 @@ static void __page_pool_empty_ring(struct page_pool *pool) pr_crit("%s() page_pool refcnt %d violation\n", __func__, page_ref_count(page)); + count++;__page_pool_return_page (pool, page); } + + pool->p.stats->ring_return += count; }static void
__warn_in_flight(struct page_pool *pool)net/core/page_pool.c +5
-0@@ -381,6 +415,9 @@ void __page_pool_free(struct page_pool *pool)if (!__page_pool_safe_to_destroy (pool))__warn_in_flight (pool); +if (pool->p.flags &PP_FLAG_INTERNAL_STATS ) +kfree (pool->p.stats); +ptr_ring_cleanup (&pool->ring,NULL );if (pool->p.flags &PP_FLAG_DMA_MAP )@@ -394,6 +431,8 @@ static void page_pool_flush(struct page_pool *pool) {struct page *page; + pool->p.stats->flush++; +/* Empty alloc cache, assume caller made sure this is * no-longer in use, and page_pool_alloc_pages() cannot be * called concurrently.
PATCH 9/10 net/mlx5: Add page_pool stats to the Mellanox driver
In reply to Jonathan LemonReplace the now deprecated inernal cache stats with the page pool stats.
# ethtool -S eth0 | grep rx_pool
rx_pool_cache_hit: 1646798
rx_pool_cache_full: 0
rx_pool_cache_empty: 15723566
rx_pool_ring_produce: 474958
rx_pool_ring_consume: 0
rx_pool_ring_return: 474958
rx_pool_flush: 144
rx_pool_node_change: 0Showing about a 10% hit rate for the page pool.
--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++++++++++++------- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +++++---- 4 files changed, 35 insertions(+), 25 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en.h +1 -0
@@ -50,6 +50,7 @@#include <net/xdp.h> #include <linux/dim.h> #include <linux/bits.h> +#include <net/page_pool.h> #include "wq.h" #include "mlx5_core.h" #include "en_stats.h"
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1 -0
@@ -551,6 +551,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, pp_params.nid =cpu_to_node (c->cpu); pp_params.dev = c->pdev; pp_params.dma_dir = rq->buff.map_dir; + pp_params.stats = &rq->stats->pool;/* page_pool can be used even when there is no rq->xdp_prog, * given page_pool does not handle DMA mapping there is no
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +24 -15
@@ -102,11 +102,14 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_buff_alloc_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_blks) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_pkts) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_reuse) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_full) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_empty) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_busy) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_waive) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_hit) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_full) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_empty) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_produce) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_consume) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_return) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_flush) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_node_change) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_congst_umr) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_arfs_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_recover) },
@@ -214,11 +217,14 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->rx_buff_alloc_err += rq_stats->buff_alloc_err; s->rx_cqe_compress_blks += rq_stats->cqe_compress_blks; s->rx_cqe_compress_pkts += rq_stats->cqe_compress_pkts; - s->rx_cache_reuse += rq_stats->cache_reuse; - s->rx_cache_full += rq_stats->cache_full; - s->rx_cache_empty += rq_stats->cache_empty; - s->rx_cache_busy += rq_stats->cache_busy; - s->rx_cache_waive += rq_stats->cache_waive; + s->rx_pool_cache_hit += rq_stats->pool.cache_hit; + s->rx_pool_cache_full += rq_stats->pool.cache_full; + s->rx_pool_cache_empty += rq_stats->pool.cache_empty; + s->rx_pool_ring_produce += rq_stats->pool.ring_produce; + s->rx_pool_ring_consume += rq_stats->pool.ring_consume; + s->rx_pool_ring_return += rq_stats->pool.ring_return; + s->rx_pool_flush += rq_stats->pool.flush; + s->rx_pool_node_change += rq_stats->pool.node_change; s->rx_congst_umr += rq_stats->congst_umr; s->rx_arfs_err += rq_stats->arfs_err; s->rx_recover += rq_stats->recover;
@@ -1446,11 +1452,14 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, buff_alloc_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_blks) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_pkts) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_reuse) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_full) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_empty) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_busy) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_waive) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_hit) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_full) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_empty) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_produce) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_consume) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_return) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.flush) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.node_change) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, congst_umr) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, arfs_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, recover) },
drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +9 -10
@@ -109,11 +109,14 @@ struct mlx5e_sw_stats {u64 rx_buff_alloc_err;u64 rx_cqe_compress_blks;u64 rx_cqe_compress_pkts; -u64 rx_cache_reuse; -u64 rx_cache_full; -u64 rx_cache_empty; -u64 rx_cache_busy; -u64 rx_cache_waive; +u64 rx_pool_cache_hit; +u64 rx_pool_cache_full; +u64 rx_pool_cache_empty; +u64 rx_pool_ring_produce; +u64 rx_pool_ring_consume; +u64 rx_pool_ring_return; +u64 rx_pool_flush; +u64 rx_pool_node_change;u64 rx_congst_umr;u64 rx_arfs_err;u64 rx_recover;
@@ -245,14 +248,10 @@ struct mlx5e_rq_stats {u64 buff_alloc_err;u64 cqe_compress_blks;u64 cqe_compress_pkts; -u64 cache_reuse; -u64 cache_full; -u64 cache_empty; -u64 cache_busy; -u64 cache_waive;u64 congst_umr;u64 arfs_err;u64 recover; +struct page_pool_stats pool; };struct mlx5e_sq_stats {
Signature · 1 line
2.17.1
Re: [PATCH 09/10 net-next] net/mlx5: Add page_pool stats to the Mellanox driver
Quoted 1 line · On Wed, 16 Oct 2019 15:50:27 -0700 Jonathan Lemon <[email protected]>…
Replace the now deprecated inernal cache stats with the page pool stats.
I can see that the stats you introduced are useful, but they have to be implemented in way that does not hurt performance.
Quoted 11 lines
# ethtool -S eth0 | grep rx_pool rx_pool_cache_hit: 1646798 rx_pool_cache_full: 0 rx_pool_cache_empty: 15723566 rx_pool_ring_produce: 474958 rx_pool_ring_consume: 0 rx_pool_ring_return: 474958 rx_pool_flush: 144 rx_pool_node_change: 0Showing about a 10% hit rate for the page pool.
What is the workload from above stats?
Quoted 136 lines
Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++++++++++++------- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +++++---- 4 files changed, 35 insertions(+), 25 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/en.h +1
-0@@ -50,6 +50,7 @@#include <net/xdp.h> #include <linux/dim.h> #include <linux/bits.h> +#include <net/page_pool.h> #include "wq.h" #include "mlx5_core.h" #include "en_stats.h" drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1
-0@@ -551,6 +551,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, pp_params.nid =cpu_to_node (c->cpu); pp_params.dev = c->pdev; pp_params.dma_dir = rq->buff.map_dir; + pp_params.stats = &rq->stats->pool;/* page_pool can be used even when there is no rq->xdp_prog, * given page_pool does not handle DMA mapping there is no drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +24
-15@@ -102,11 +102,14 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_buff_alloc_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_blks) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_pkts) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_reuse) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_full) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_empty) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_busy) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_waive) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_hit) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_full) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_empty) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_produce) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_consume) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_return) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_flush) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_node_change) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_congst_umr) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_arfs_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_recover) },@@ -214,11 +217,14 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->rx_buff_alloc_err += rq_stats->buff_alloc_err; s->rx_cqe_compress_blks += rq_stats->cqe_compress_blks; s->rx_cqe_compress_pkts += rq_stats->cqe_compress_pkts; - s->rx_cache_reuse += rq_stats->cache_reuse; - s->rx_cache_full += rq_stats->cache_full; - s->rx_cache_empty += rq_stats->cache_empty; - s->rx_cache_busy += rq_stats->cache_busy; - s->rx_cache_waive += rq_stats->cache_waive; + s->rx_pool_cache_hit += rq_stats->pool.cache_hit; + s->rx_pool_cache_full += rq_stats->pool.cache_full; + s->rx_pool_cache_empty += rq_stats->pool.cache_empty; + s->rx_pool_ring_produce += rq_stats->pool.ring_produce; + s->rx_pool_ring_consume += rq_stats->pool.ring_consume; + s->rx_pool_ring_return += rq_stats->pool.ring_return; + s->rx_pool_flush += rq_stats->pool.flush; + s->rx_pool_node_change += rq_stats->pool.node_change; s->rx_congst_umr += rq_stats->congst_umr; s->rx_arfs_err += rq_stats->arfs_err; s->rx_recover += rq_stats->recover;@@ -1446,11 +1452,14 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, buff_alloc_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_blks) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_pkts) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_reuse) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_full) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_empty) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_busy) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_waive) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_hit) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_full) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_empty) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_produce) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_consume) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_return) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.flush) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.node_change) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, congst_umr) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, arfs_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, recover) },drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +9
-10@@ -109,11 +109,14 @@ struct mlx5e_sw_stats {u64 rx_buff_alloc_err;u64 rx_cqe_compress_blks;u64 rx_cqe_compress_pkts; -u64 rx_cache_reuse; -u64 rx_cache_full; -u64 rx_cache_empty; -u64 rx_cache_busy; -u64 rx_cache_waive; +u64 rx_pool_cache_hit; +u64 rx_pool_cache_full; +u64 rx_pool_cache_empty; +u64 rx_pool_ring_produce; +u64 rx_pool_ring_consume; +u64 rx_pool_ring_return; +u64 rx_pool_flush; +u64 rx_pool_node_change;u64 rx_congst_umr;u64 rx_arfs_err;u64 rx_recover;@@ -245,14 +248,10 @@ struct mlx5e_rq_stats {u64 buff_alloc_err;u64 cqe_compress_blks;u64 cqe_compress_pkts; -u64 cache_reuse; -u64 cache_full; -u64 cache_empty; -u64 cache_busy; -u64 cache_waive;u64 congst_umr;u64 arfs_err;u64 recover; +struct page_pool_stats pool; };struct mlx5e_sq_stats {
Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
Re: [PATCH 09/10 net-next] net/mlx5: Add page_pool stats to the Mellanox driver
Quoted 7 lines · Jesper Dangaard Brouer
Quoted 1 line · On Wed, 16 Oct 2019 15:50:27 -0700 Jonathan Lemon <[email protected]>…
Replace the now deprecated inernal cache stats with the page pool stats.
I can see that the stats you introduced are useful, but they have to be implemented in way that does not hurt performance.
They're not noticeable, but even if they were, they are needed for production, otherwise there's no way to identify problems.
I can separate the ring consume/produce counters so they are always separated by a cache line distance.
Quoted 10 lines
Quoted 10 lines
# ethtool -S eth0 | grep rx_pool rx_pool_cache_hit: 1646798 rx_pool_cache_full: 0 rx_pool_cache_empty: 15723566 rx_pool_ring_produce: 474958 rx_pool_ring_consume: 0 rx_pool_ring_return: 474958 rx_pool_flush: 144 rx_pool_node_change: 0
Quoted 3 lines
Quoted 1 line
Showing about a 10% hit rate for the page pool.
What is the workload from above stats?
Network traffic from a proxygen load balancer. From this, we see the ptr_ring is completely unused except for flushing operations.
Signature · 1 line
Jonathan
Quoted 159 lines
Quoted 151 lines
Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 + .../net/ethernet/mellanox/mlx5/core/en_main.c | 1 + .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++++++++++++------- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +++++---- 4 files changed, 35 insertions(+), 25 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/en.h +1
-0@@ -50,6 +50,7 @@#include <net/xdp.h> #include <linux/dim.h> #include <linux/bits.h> +#include <net/page_pool.h> #include "wq.h" #include "mlx5_core.h" #include "en_stats.h" drivers/net/ethernet/mellanox/mlx5/core/en_main.c +1
-0@@ -551,6 +551,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c, pp_params.nid =cpu_to_node (c->cpu); pp_params.dev = c->pdev; pp_params.dma_dir = rq->buff.map_dir; + pp_params.stats = &rq->stats->pool;/* page_pool can be used even when there is no rq->xdp_prog, drivers/net/ethernet/mellanox/mlx5/core/en_main.c +0
-0* given page_pool does not handle DMA mapping there is no drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +4
-5@@ -102,11 +102,14 @@ static const struct counter_desc sw_stats_desc[] = { { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_buff_alloc_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_blks) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cqe_compress_pkts) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_reuse) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_full) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_empty) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_busy) }, - { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_cache_waive) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_hit) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_full) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_cache_empty) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_produce)},
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +1
-0+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_consume)},
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +11
-5+ { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_ring_return) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_flush) }, + { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_pool_node_change) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_congst_umr) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_arfs_err) }, { MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_recover) },@@ -214,11 +217,14 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv) s->rx_buff_alloc_err += rq_stats->buff_alloc_err; s->rx_cqe_compress_blks += rq_stats->cqe_compress_blks; s->rx_cqe_compress_pkts += rq_stats->cqe_compress_pkts; - s->rx_cache_reuse += rq_stats->cache_reuse; - s->rx_cache_full += rq_stats->cache_full; - s->rx_cache_empty += rq_stats->cache_empty; - s->rx_cache_busy += rq_stats->cache_busy; - s->rx_cache_waive += rq_stats->cache_waive; + s->rx_pool_cache_hit += rq_stats->pool.cache_hit; + s->rx_pool_cache_full += rq_stats->pool.cache_full; + s->rx_pool_cache_empty += rq_stats->pool.cache_empty; + s->rx_pool_ring_produce += rq_stats->pool.ring_produce; + s->rx_pool_ring_consume += rq_stats->pool.ring_consume; + s->rx_pool_ring_return += rq_stats->pool.ring_return; + s->rx_pool_flush += rq_stats->pool.flush; + s->rx_pool_node_change += rq_stats->pool.node_change; s->rx_congst_umr += rq_stats->congst_umr; s->rx_arfs_err += rq_stats->arfs_err;drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +4
-5s->rx_recover += rq_stats->recover;@@ -1446,11 +1452,14 @@ static const struct counter_desc rq_stats_desc[] = { { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, buff_alloc_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_blks) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cqe_compress_pkts) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_reuse) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_full) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_empty) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_busy) }, - { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, cache_waive) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_hit) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_full) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.cache_empty) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_produce)},
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +1
-0+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_consume)},
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +3
-0+ { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.ring_return) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.flush) }, + { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, pool.node_change) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, congst_umr) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, arfs_err) }, { MLX5E_DECLARE_RX_STAT(struct mlx5e_rq_stats, recover) },drivers/net/ethernet/mellanox/mlx5/core/en_stats.h +9
-10@@ -109,11 +109,14 @@ struct mlx5e_sw_stats {u64 rx_buff_alloc_err;u64 rx_cqe_compress_blks;u64 rx_cqe_compress_pkts; -u64 rx_cache_reuse; -u64 rx_cache_full; -u64 rx_cache_empty; -u64 rx_cache_busy; -u64 rx_cache_waive; +u64 rx_pool_cache_hit; +u64 rx_pool_cache_full; +u64 rx_pool_cache_empty; +u64 rx_pool_ring_produce; +u64 rx_pool_ring_consume; +u64 rx_pool_ring_return; +u64 rx_pool_flush; +u64 rx_pool_node_change;u64 rx_congst_umr;u64 rx_arfs_err;u64 rx_recover;@@ -245,14 +248,10 @@ struct mlx5e_rq_stats {u64 buff_alloc_err;u64 cqe_compress_blks;u64 cqe_compress_pkts; -u64 cache_reuse; -u64 cache_full; -u64 cache_empty; -u64 cache_busy; -u64 cache_waive;u64 congst_umr;u64 arfs_err;u64 recover; +struct page_pool_stats pool; };struct mlx5e_sq_stats {Signature · 4 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
PATCH 10/10 page_pool: Cleanup and rename page_pool functions.
In reply to Jonathan LemonFunctions starting with __ usually indicate those which are called while holding a lock, or are exported, but should not be called directly. Internal static functions don't meet either of these criteria, so make it more readable.
Move stub functions to end of file in their own section for readability and ease of maintenance. Add missing page_pool_get() stub function.
Checked by compiling kernel without PAGE_POOL defined.
--- include/net/page_pool.h | 141 ++++++++++++++++++++-------------------- net/core/page_pool.c | 76 ++++++++++------------ 2 files changed, 102 insertions(+), 115 deletions(-)
include/net/page_pool.h +69 -72
@@ -124,6 +124,8 @@ struct page_pool {void *alloc_cache[]; }; +#ifdef CONFIG_PAGE_POOL +struct page *page_pool_alloc_pages (struct page_pool *pool,gfp_t gfp);static inline struct page *page_pool_dev_alloc_pages (struct page_pool *pool)
@@ -133,8 +135,8 @@ static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)return page_pool_alloc_pages (pool, gfp); } -/* get the stored dma direction. A driver might decide to treat this locally and -* avoid the extra cache line from page_pool to determine the direction +/* get the stored dma direction. A driver might decide to treat this locally +* and avoid the extra cache line from page_pool to determine the direction. */ static inline enum dma_data_direction page_pool_get_dma_dir (struct page_pool *pool)
@@ -143,16 +145,36 @@ inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) }struct page_pool *page_pool_create (const struct page_pool_params *params); +void page_pool_free (struct page_pool *pool); +void page_pool_put_page (struct page_pool *pool, +struct page *page,bool allow_direct); -void __page_pool_free (struct page_pool *pool); -static inline void page_pool_free (struct page_pool *pool) +/* Very limited use-cases allow recycle direct */ +static inline void page_pool_recycle_direct (struct page_pool *pool, +struct page *page) { -/* When page_pool isn't compiled-in, net/core/xdp.c doesn't -* allow registering MEM_TYPE_PAGE_POOL, but shield linker. -*/ -#ifdef CONFIG_PAGE_POOL -__page_pool_free (pool); -#endif +page_pool_put_page (pool, page, true); +} + +/* API user MUST have disconnected alloc-side (not allowed to call +* page_pool_alloc_pages()) before calling this. The free-side can +* still run concurrently, to handle in-flight packet-pages. +* +* A request to shutdown can fail (with false) if there are still +* in-flight packet-pages. +*/ +bool page_pool_request_shutdown (struct page_pool *pool); + +/* Disconnects a page (from a page_pool). API users can have a need +* to disconnect a page (from a page_pool), to allow it to be used as +* a regular page (that will eventually be returned to the normal +* page-allocator via put_page). +*/ +void page_pool_release_page (struct page_pool *pool,struct page *page); + +static inline bool is_page_pool_compiled_in (void ) +{ +return true; }/* Drivers use this instead of page_pool_free */
@@ -164,73 +186,11 @@ static inline void page_pool_destroy(struct page_pool *pool)page_pool_free (pool); } -/* Never call this directly, use helpers below */ -void __page_pool_put_page (struct page_pool *pool, -struct page *page,bool allow_direct); - -static inline void page_pool_put_page (struct page_pool *pool, -struct page *page,bool allow_direct) -{ -/* When page_pool isn't compiled-in, net/core/xdp.c doesn't -* allow registering MEM_TYPE_PAGE_POOL, but shield linker. -*/ -#ifdef CONFIG_PAGE_POOL -__page_pool_put_page (pool, page, allow_direct); -#endif -} -/* Very limited use-cases allow recycle direct */ -static inline void page_pool_recycle_direct (struct page_pool *pool, -struct page *page) -{ -__page_pool_put_page (pool, page, true); -} - -/* API user MUST have disconnected alloc-side (not allowed to call -* page_pool_alloc_pages()) before calling this. The free-side can -* still run concurrently, to handle in-flight packet-pages. -* -* A request to shutdown can fail (with false) if there are still -* in-flight packet-pages. -*/ -bool __page_pool_request_shutdown (struct page_pool *pool); -static inline bool page_pool_request_shutdown (struct page_pool *pool) -{ -bool safe_to_remove = false; - -#ifdef CONFIG_PAGE_POOL - safe_to_remove =__page_pool_request_shutdown (pool); -#endif -return safe_to_remove; -} - -/* Disconnects a page (from a page_pool). API users can have a need -* to disconnect a page (from a page_pool), to allow it to be used as -* a regular page (that will eventually be returned to the normal -* page-allocator via put_page). -*/ -void page_pool_unmap_page (struct page_pool *pool,struct page *page); -static inline void page_pool_release_page (struct page_pool *pool, -struct page *page) -{ -#ifdef CONFIG_PAGE_POOL -page_pool_unmap_page (pool, page); -#endif -} -static inline dma_addr_t page_pool_get_dma_addr (struct page *page) {return page->dma_addr; } -static inline bool is_page_pool_compiled_in (void ) -{ -#ifdef CONFIG_PAGE_POOL -return true; -#else -return false; -#endif -} -static inline void page_pool_get (struct page_pool *pool) {refcount_inc (&pool->user_cnt);
@@ -249,4 +209,41 @@ static inline void page_pool_update_nid(struct page_pool *pool, int new_nid) pool->p.stats->node_change++; } } + +#else + +/* When page_pool isn't compiled-in, net/core/xdp.c doesn't +* allow registering MEM_TYPE_PAGE_POOL, but shield linker. +*/ + +static inline void page_pool_free (struct page_pool *pool) +{ +} + +static inline void page_pool_put_page (struct page_pool *pool, +struct page *page,bool allow_direct) +{ +} + +static inline bool page_pool_request_shutdown (struct page_pool *pool) +{ +return false; +} + +static inline void page_pool_release_page (struct page_pool *pool, +struct page *page) +{ +} + +static inline bool is_page_pool_compiled_in (void ) +{ +return false; +} + +static inline void page_pool_get (struct page_pool *pool) +{ +} + +#endif /* CONFIG_PAGE_POOL */ +#endif /* _NET_PAGE_POOL_H */
net/core/page_pool.c +33 -43
200 lines not on this page · raw message
Signature · 1 line
2.17.1
Quoted 10 lines · Jonathan Lemon
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts withAF_XDP.- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
Hi Jonathan, thanks for submitting this,
the patches you have are not up to date, i have new ones with tracing support and some fixes from offlist review iterations, plus performance numbers and a cover letter.
I will send it to you and you can post it as v2 ?
Quoted 30 lines
- Statistics and cleanup for page pool.
Jonathan Lemon (5):
page_pool: Addpage_pool_keep_page
page_pool: allow configurable linear cache size
page_pool: Add statistics
net/mlx5: Addpage_poolstats to the Mellanox driver
page_pool: Cleanup and renamepage_poolfunctions.Saeed Mahameed (2):
page_pool: Add API to update numa node and flush page caches net/mlx5e: Rx, Update page pool numa node when changedTariq Toukan (3):
net/mlx5e: RX, Remove RX page-cache net/mlx5e: RX, Manage RX pages only via page pool API net/mlx5e: RX, Internal DMA mapping inpage_pooldrivers/net/ethernet/mellanox/mlx5/core/en.h | 18 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 12 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 19 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 128 ++-------- .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++-- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +- include/net/page_pool.h | 216 +++++++++------- - net/core/page_pool.c | 221 +++++++++++----- -- 8 files changed, 319 insertions(+), 353 deletions(-)
Quoted 12 lines · Saeed Mahameed
Quoted 10 lines · Jonathan Lemon
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts withAF_XDP.- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
Quoted 7 lines
Hi Jonathan, thanks for submitting this,
the patches you have are not up to date, i have new ones with tracing support and some fixes from offlist review iterations, plus performance
numbers and a cover letter.I will send it to you and you can post it as v2 ?
actually i suggest to take my 3 patches out of this series and submit them as standalone, they are not directly related to the other stuff here, and can perfectly work without them, since my 3 patches are addressing a real issue with page pool numa node migration.
Quoted 34 lines
Quoted 34 lines
- Statistics and cleanup for page pool.
Jonathan Lemon (5):
page_pool: Addpage_pool_keep_page
page_pool: allow configurable linear cache size
page_pool: Add statistics
net/mlx5: Addpage_poolstats to the Mellanox driver
page_pool: Cleanup and renamepage_poolfunctions.Saeed Mahameed (2):
page_pool: Add API to update numa node and flush page caches net/mlx5e: Rx, Update page pool numa node when changedTariq Toukan (3):
net/mlx5e: RX, Remove RX page-cache net/mlx5e: RX, Manage RX pages only via page pool API net/mlx5e: RX, Internal DMA mapping inpage_pooldrivers/net/ethernet/mellanox/mlx5/core/en.h | 18 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 12 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 19 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 128 ++-------- .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++-- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +- include/net/page_pool.h | 216 +++++++++----- -- - net/core/page_pool.c | 221 +++++++++++--- -- -- 8 files changed, 319 insertions(+), 353 deletions(-)
Quoted 12 lines · Saeed Mahameed
Quoted 10 lines · Jonathan Lemon
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts withAF_XDP.- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
Quoted 6 lines
Hi Jonathan, thanks for submitting this,
the patches you have are not up to date, i have new ones with tracing support and some fixes from offlist review iterations, plus performance numbers and a cover letter.I will send it to you and you can post it as v2 ?
Sure, I have some other cleanups to do and have a concern about the cache effectiveness for some workloads.
Signature · 1 line
Jonathan
Quoted 32 lines
Quoted 32 lines
- Statistics and cleanup for page pool.
Jonathan Lemon (5):
page_pool: Addpage_pool_keep_page
page_pool: allow configurable linear cache size
page_pool: Add statistics
net/mlx5: Addpage_poolstats to the Mellanox driver
page_pool: Cleanup and renamepage_poolfunctions.Saeed Mahameed (2):
page_pool: Add API to update numa node and flush page caches net/mlx5e: Rx, Update page pool numa node when changedTariq Toukan (3):
net/mlx5e: RX, Remove RX page-cache net/mlx5e: RX, Manage RX pages only via page pool API net/mlx5e: RX, Internal DMA mapping inpage_pooldrivers/net/ethernet/mellanox/mlx5/core/en.h | 18 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 12 +- .../net/ethernet/mellanox/mlx5/core/en_main.c | 19 +- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 128 ++-------- .../ethernet/mellanox/mlx5/core/en_stats.c | 39 ++-- .../ethernet/mellanox/mlx5/core/en_stats.h | 19 +- include/net/page_pool.h | 216 +++++++++------- - net/core/page_pool.c | 221 +++++++++++----- -- 8 files changed, 319 insertions(+), 353 deletions(-)
Quoted 22 lines · Jonathan Lemon
Quoted 19 lines · Saeed Mahameed
Quoted 10 lines · Jonathan Lemon
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts withAF_XDP.- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
Hi Jonathan, thanks for submitting this,
the patches you have are not up to date, i have new ones with tracing
support and some fixes from offlist review iterations, plus performance
numbers and a cover letter.
Quoted 4 lines
Quoted 1 line
I will send it to you and you can post it as v2 ?
Sure, I have some other cleanups to do and have a concern about the cache effectiveness for some workloads.
Ok then, I will submit the page pool NUMA change patches separately. I will remove the flush mechanism and will add your changes.
for the other patches, mlx5 cache and page pool statistics, i think they need some more work and a lot of pieces are still WIP. I don't want to block the NUMA change API patches.
Quoted 28 lines · Saeed Mahameed
Quoted 22 lines · Jonathan Lemon
Quoted 19 lines · Saeed Mahameed
Quoted 10 lines · Jonathan Lemon
This patch combines work from various people:
- part of Tariq's work to move the DMA mapping from the mlx5 driver into the page pool. This does not include later patches which remove the dma address from the driver, as this conflicts withAF_XDP.- Saeed's changes to check the numa node before including the page in the pool, and flushing the pool on a node change.
Hi Jonathan, thanks for submitting this,
the patches you have are not up to date, i have new ones with tracing
support and some fixes from offlist review iterations, plus performance
numbers and a cover letter.Quoted 4 lines
Quoted 1 line
I will send it to you and you can post it as v2 ?
Sure, I have some other cleanups to do and have a concern about the cache effectiveness for some workloads.
Quoted 6 lines
Ok then, I will submit the page pool NUMA change patches separately. I will remove the flush mechanism and will add your changes.
for the other patches, mlx5 cache and page pool statistics, i think they need some more work and a lot of pieces are still WIP. I don't want to block the NUMA change API patches.
Sounds good - the stats are only really needed once the mlx5 private cache goes away, and it doesn't look like that will happen immediately.
The private cache and the page pool are performing two different functions at the moment.
Signature · 1 line
Jonathan