page pool: not return page to alloc cache during pool destruction
Subsystems (3)
PATCH 1/1 page pool: not return page to alloc cache during pool destruction
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
--- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/page_pool.c +1 -1
@@ -595,7 +595,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,page_pool_dma_sync_for_device (pool, page, dma_sync_size); -if (allow_direct &&in_softirq () && +if (allow_direct &&in_softirq () && !pool->destroy_cnt &&page_pool_recycle_in_cache (page, pool))return NULL ;
Signature · 1 line
2.31.1
Jakub Kicinski
maintainerQuoted 9 lines · On Thu, 15 Jun 2023 09:36:45 +0800 Liang Chen wrote:
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Let's hear from the page pool maintainers but I think the driver is supposed to prevent allocations while pool is getting destroyed. Perhaps we can add DEBUG_NET_WARN_ON_ONCE() for this condition to prevent wasting cycles in production builds?
Quoted 6 lines · Liang Chen
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
It seems there is still page_pool_put[_full]_page() called with allow_direct being true after page_pool_destroy() is call, which is not allowed.
Normally the driver will call napi_disable() before page_pool_destroy() to ensure there is no such page_pool_destroy() calling with allow_direct being true after page_pool_destroy() is called.
Quoted 14 lines
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Signed-off-by Liang Chen <[email protected]>--- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)net/core/page_pool.c +0
-0
net/core/page_pool.c __page_pool_put_page(struct page_pool *pool, struct page *page, · line 598
Quoted 5 lines
net/core/page_pool.c +1
-1@@ -595,7 +595,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,page_pool_dma_sync_for_device (pool, page, dma_sync_size); - if (allow_direct && in_softirq() && + if (allow_direct && in_softirq() && !pool->destroy_cnt &&
The checking seems racy when __page_pool_put_page() and page_pool_destroy() are called concurently.
Quoted 4 lines
net/core/page_pool.c +0
-0page_pool_recycle_in_cache(page, pool))return NULL ;
Quoted 8 lines · Yunsheng Lin
Quoted 6 lines · Liang Chen
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
Quoted 6 lines
It seems there is still page_pool_put[_full]_page() called with
allow_directbeing true afterpage_pool_destroy()is call, which is not allowed.Normally the driver will call
napi_disable()beforepage_pool_destroy()to ensure there is no suchpage_pool_destroy()
no such page_pool_put[_full]_page()
Quoted 34 lines
calling with
allow_directbeing true afterpage_pool_destroy()is called.Quoted 14 lines
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Signed-off-by Liang Chen <[email protected]>--- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)net/core/page_pool.c +0
-0net/core/page_pool.c __page_pool_put_page(struct page_pool *pool, struct page *page, · line 598
Quoted 5 lines
net/core/page_pool.c +1
-1@@ -595,7 +595,7 @@ __page_pool_put_page(struct page_pool *pool, struct page *page,page_pool_dma_sync_for_device (pool, page, dma_sync_size); - if (allow_direct && in_softirq() && + if (allow_direct && in_softirq() && !pool->destroy_cnt &&The checking seems racy when
__page_pool_put_page()andpage_pool_destroy()are called concurently.net/core/page_pool.c __page_pool_put_page(struct page_pool *pool, struct page *page, · line 601
Quoted 4 lines
net/core/page_pool.c +0
-0page_pool_recycle_in_cache(page, pool))return NULL ;.
Ilias Apalodimas
maintainerHi Jakub,
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 15 Jun 2023 09:36:45 +0800 Liang Chen wrote:
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
Quoted 7 lines
Quoted 2 lines
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Let's hear from the page pool maintainers but I think the driver is supposed to prevent allocations while pool is getting destroyed. Perhaps we can add
DEBUG_NET_WARN_ON_ONCE()for this condition to prevent wasting cycles in production builds?
Yes the driver is supposed to do that, but OTOH I generally prefer APIs that don't allow people to shoot themselves in the foot. IIRC this check run in fast path only in XDP mode right? If this doesn't affect performance, I don't have any objections. Jesper was trying to refactor the destruction path, perhaps there's something in this that affects his code?
Thanks
/Ilias
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 15 Jun 2023 09:36:45 +0800 Liang Chen wrote:
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
Quoted 5 lines
Quoted 2 lines
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Let's hear from the page pool maintainers but I think the driver is supposed to prevent allocations while pool is getting destroyed.
Yes, this is a driver API violation. Direct returns (allow_direct) can only happen from drivers RX path, e.g while driver is active processing packets (in NAPI). When driver is shutting down a page_pool, it MUST have stopped RX path and NAPI (napi_disable()) before calling page_pool_destroy() Thus, this situation cannot happen and if it does it is a driver bug.
Quoted 3 lines
Perhaps we can add
DEBUG_NET_WARN_ON_ONCE()for this condition to prevent wasting cycles in production builds?
For this page_pool code path ("allow_direct") it is extremely important we avoid wasting cycles in production. As this is used for XDP_DROP use-cases for 100Gbit/s NICs.
At 100Gbit/s with 64 bytes Ethernet frames (84 on wire), the wirespeed is 148.8Mpps which gives CPU 6.72 nanosec to process each packet. The microbench[1] shows (below signature) that page_pool_alloc_pages() + page_pool_recycle_direct() cost 4.041 ns (or 14 cycles(tsc)). Thus, for this code fast-path every cycle counts.
In practice PCIe transactions/sec seems limit total system to 108Mpps (with multiple RX-queues + descriptor compression) thus 9.26 nanosec to process each packet. Individual hardware RX queues seems be limited to around 36Mpps thus 27.77 nanosec to process each packet.
Adding a DEBUG_NET_WARN_ON_ONCE will be annoying as I like to run my testlab kernels with CONFIG_DEBUG_NET, which will change this extreme fash-path slightly (adding some unlikely's affecting code layout to the mix).
Question to Liang Chen: Did you hit this bug in practice?
--Jesper
CPU E5-1650 v4 @ 3.60GHz
tasklet_page_pool01_fast_path Per elem: 14 cycles(tsc) 4.041 ns tasklet_page_pool02_ptr_ring Per elem: 49 cycles(tsc) 13.622 ns
tasklet_page_pool03_slow Per elem: 162 cycles(tsc) 45.198 ns
[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/lib/bench_page_pool_simple.c
Jakub Kicinski
maintainerQuoted 4 lines · On Thu, 15 Jun 2023 16:00:13 +0200 Jesper Dangaard Brouer wrote:
Adding a
DEBUG_NET_WARN_ON_ONCEwill be annoying as I like to run my testlab kernels withCONFIG_DEBUG_NET, which will change this extreme fash-path slightly (adding some unlikely's affecting code layout to the mix).
That's counter-intuitive - the whole point of DEBUG_NET is to have a place where we can add checks which we don't want in production builds. If we can't use it on the datapath we should just remove it completely and have its checks always enabled..
I do feel your pain of having to test code both with debug enabled and disabled, but we can't have a cookie and eat it too :(
Quoted 44 lines · Jesper Dangaard Brouer
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 15 Jun 2023 09:36:45 +0800 Liang Chen wrote:
When destroying a page pool, the alloc cache and recycle ring are emptied. If there are inflight pages, the retry process will periodically check the recycle ring for recently returned pages, but not the alloc cache (alloc cache is only emptied once). As a result, any pages returned to the alloc cache after the page pool destruction will be stuck there and cause the retry process to continuously look for inflight pages and report warnings.
Quoted 5 lines
Quoted 2 lines
To safeguard against this situation, any pages returning to the alloc cache after pool destruction should be prevented.
Let's hear from the page pool maintainers but I think the driver is supposed to prevent allocations while pool is getting destroyed.
Yes, this is a driver API violation. Direct returns (
allow_direct) can only happen from drivers RX path, e.g while driver is active processing packets (in NAPI). When driver is shutting down apage_pool, it MUST have stopped RX path and NAPI (napi_disable()) before callingpage_pool_destroy()Thus, this situation cannot happen and if it does it is a driver bug.Quoted 3 lines
Perhaps we can add
DEBUG_NET_WARN_ON_ONCE()for this condition to prevent wasting cycles in production builds?For this
page_poolcode path ("allow_direct") it is extremely important we avoid wasting cycles in production. As this is used forXDP_DROPuse-cases for 100Gbit/s NICs.At 100Gbit/s with 64 bytes Ethernet frames (84 on wire), the wirespeed is 148.8Mpps which gives CPU 6.72 nanosec to process each packet. The microbench[1] shows (below signature) that
page_pool_alloc_pages()+page_pool_recycle_direct()cost 4.041 ns (or 14 cycles(tsc)). Thus, for this code fast-path every cycle counts.In practice PCIe transactions/sec seems limit total system to 108Mpps (with multiple RX-queues + descriptor compression) thus 9.26 nanosec to process each packet. Individual hardware RX queues seems be limited to around 36Mpps thus 27.77 nanosec to process each packet.
Quoted 9 lines
Adding a
DEBUG_NET_WARN_ON_ONCEwill be annoying as I like to run my testlab kernels withCONFIG_DEBUG_NET, which will change this extreme fash-path slightly (adding some unlikely's affecting code layout to the mix).Question to Liang Chen: Did you hit this bug in practice?
--Jesper
Yeah, we hit this problem while implementing page pool support for virtio_net driver, where we only enable page pool for xdp path, i.e. turning on/off page pool when xdp is enabled/disabled. The problem turns up when the xdp program is uninstalled, and there are still inflight page pool page buffers. Then napi is enabled again, the driver starts to process those inflight page pool buffers. So we will need to be aware of the state of the page pool (if it is being destructed) while returning the pages back. That's what motivated us to add this check to __page_pool_put_page.
Thanks,
Liang
Quoted 8 lines
CPU E5-1650 v4 @ 3.60GHz
tasklet_page_pool01_fast_pathPer elem: 14 cycles(tsc) 4.041 nstasklet_page_pool02_ptr_ringPer elem: 49 cycles(tsc) 13.622 ns
tasklet_page_pool03_slowPer elem: 162 cycles(tsc) 45.198 ns[1]
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/lib/bench_page_pool_simple.c