bnxt: Detach page from page pool before sending up the stack
PATCH 1/1 bnxt: Detach page from page pool before sending up the stack
When running in XDP mode, pages come from the page pool, and should be freed back to the same pool or specifically detached. Currently, when the driver re-initializes, the page pool destruction is delayed forever since it thinks there are oustanding pages.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -0
@@ -944,6 +944,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp, dma_addr -= bp->rx_dma_offset;dma_unmap_page_attrs (&bp->pdev->dev, dma_addr,PAGE_SIZE , bp->rx_dir,DMA_ATTR_WEAK_ORDERING ); +page_pool_release_page (rxr->page_pool, page);if (unlikely (!payload)) payload =eth_get_headlen (bp->dev, data_ptr, len);
Signature · 1 line
2.17.1
Quoted 4 lines · Jonathan Lemon
When running in XDP mode, pages come from the page pool, and should be freed back to the same pool or specifically detached. Currently, when the driver re-initializes, the page pool destruction is delayed forever since it thinks there are oustanding pages.
If you can please share a reproduction script/steps that would be helpful for me.
Since this is an XDP_PASS case I can easily create a program that does that, so no need to share that program -- just the sequence to remove the program, shutdown the driver, whatever is done and the error you see.
Thanks!
Quoted 22 lines
Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-0@@ -944,6 +944,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp, dma_addr -= bp->rx_dma_offset;dma_unmap_page_attrs (&bp->pdev->dev, dma_addr,PAGE_SIZE , bp->rx_dir,DMA_ATTR_WEAK_ORDERING ); +page_pool_release_page (rxr->page_pool, page);if (unlikely (!payload)) payload =eth_get_headlen (bp->dev, data_ptr, len);Signature · 1 line
2.17.1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 28 lines · Jonathan Lemon
Sure, on our FB systems, the reproduction is simple:
- Start an XDP program (which uses the page pool). In our case, it's droplet.
- Send normal traffic, so pages are released to the system.
- Restart droplet (or do something which causes the driver to re-initialize) This frees and reallocates the rings and page pool.After a bit, observe incessant warnings about being unable to release the page pool due to pages inflight:
Jan 4 03:19:17 twtraffic0235.06.atn5.facebook.com kernel: [1941287.997351] page_pool_release_retry() stalled pool shutdown -1783494730 inflight 1451031 sec Jan 4 03:19:24 twtraffic0235.06.atn5.facebook.com kernel: [1941295.031546] page_pool_release_retry() stalled pool shutdown -1962791719 inflight 1451040 sec Jan 4 03:19:38 twtraffic0235.06.atn5.facebook.com kernel: [1941308.131831] page_pool_release_retry() stalled pool shutdown -1978246510 inflight 1451052 sec Jan 4 03:19:57 twtraffic0235.06.atn5.facebook.com kernel: [1941327.842882] page_pool_release_retry() stalled pool shutdown -1893208809 inflight 1451072 sec Jan 4 03:19:58 twtraffic0235.06.atn5.facebook.com kernel: [1941328.332587] page_pool_release_retry() stalled pool shutdown -1822809109 inflight 1451070 sec Jan 4 03:20:10 twtraffic0235.06.atn5.facebook.com kernel: [1941340.994990]
Quoted 6 lines
page_pool_release_retry()stalled pool shutdown -2064970262 inflight 1451087 secAlso note that if the count is negative, this triggers a kernel stack trace, which has a deleterious affect on the performance of the system.
Thanks. I actually tried something similar this morning before you sent this and was unable to reproduce at first.
I thought about it more and realized that I needed to send packets larger than BNXT_RX_COPY_THRESH. I see it now and will have more feedback in a few mins.
Quoted 45 lines · Andy Gospodarek
Quoted 42 lines · Jonathan Lemon
Quoted 4 lines
When running in XDP mode, pages come from the page pool, and should be freed back to the same pool or specifically detached. Currently, when the driver re-initializes, the page pool destruction is delayed forever since it thinks there are oustanding pages.
If you can please share a reproduction script/steps that would be helpful for me.
Since this is an
XDP_PASScase I can easily create a program that does that, so no need to share that program -- just the sequence to remove the program, shutdown the driver, whatever is done and the error you see.Thanks!
Quoted 25 lines
Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Signed-off-by Jonathan Lemon <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-0@@ -944,6 +944,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp, dma_addr -= bp->rx_dma_offset;dma_unmap_page_attrs (&bp->pdev->dev, dma_addr,PAGE_SIZE , bp->rx_dir,DMA_ATTR_WEAK_ORDERING ); +page_pool_release_page (rxr->page_pool, page);drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-0if (unlikely (!payload)) payload =eth_get_headlen (bp->dev, data_ptr, len);Signature · 1 line
2.17.1
Quoted 4 lines · Jonathan Lemon
When running in XDP mode, pages come from the page pool, and should be freed back to the same pool or specifically detached. Currently, when the driver re-initializes, the page pool destruction is delayed forever since it thinks there are oustanding pages.
Looks good. Thanks for the patch!
Quoted 2 lines
Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Signed-off-by Jonathan Lemon <[email protected]>
Quoted 19 lines
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-0@@ -944,6 +944,7 @@ static struct sk_buff *bnxt_rx_page_skb(struct bnxt *bp, dma_addr -= bp->rx_dma_offset;dma_unmap_page_attrs (&bp->pdev->dev, dma_addr,PAGE_SIZE , bp->rx_dir,DMA_ATTR_WEAK_ORDERING ); +page_pool_release_page (rxr->page_pool, page);if (unlikely (!payload)) payload =eth_get_headlen (bp->dev, data_ptr, len);Signature · 1 line
2.17.1
From: Jonathan Lemon <[email protected]>
Date: Thu, 9 Jan 2020 11:35:42 -0800
Quoted 7 lines
When running in XDP mode, pages come from the page pool, and should be freed back to the same pool or specifically detached. Currently, when the driver re-initializes, the page pool destruction is delayed forever since it thinks there are oustanding pages.
Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Signed-off-by Jonathan Lemon <[email protected]>
Applied, thank you.