bnxt_en: Add page_pool_destroy() during RX ring cleanup.
Michael Chan
maintainerPATCH 1/1 bnxt_en: Add page_pool_destroy() during RX ring cleanup.
Add page_pool_destroy() in bnxt_free_rx_rings() during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41 ("net: core: page_pool: add user refcnt and reintroduce page_pool_destroy")
The special error handling code to call page_pool_free() can now be removed. bnxt_free_rx_rings() will always be called during normal shutdown or any error paths.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -6
@@ -2500,6 +2500,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)if (xdp_rxq_info_is_reg (&rxr->xdp_rxq))xdp_rxq_info_unreg (&rxr->xdp_rxq); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;kfree (rxr->rx_tpa);
@@ -2560,19 +2561,14 @@ static int bnxt_alloc_rx_rings(struct bnxt *bp)return rc; rc =xdp_rxq_info_reg (&rxr->xdp_rxq, bp->dev, i); -if (rc <0 ) { -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ; +if (rc <0 )return rc; - } rc =xdp_rxq_info_reg_mem_model (&rxr->xdp_rxq,MEM_TYPE_PAGE_POOL , rxr->page_pool);if (rc) {xdp_rxq_info_unreg (&rxr->xdp_rxq); -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ;return rc; }
Signature · 1 line
2.5.1
Quoted 38 lines · Michael Chan
Add
page_pool_destroy()inbnxt_free_rx_rings()during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41("net: core:page_pool: add user refcnt and reintroducepage_pool_destroy")The special error handling code to call
page_pool_free()can now be removed.bnxt_free_rx_rings()will always be called during normal shutdown or any error paths.Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Cc Ilias Apalodimas <[email protected]>Cc Andy Gospodarek <[email protected]>Signed-off-by Michael Chan <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +2
-3@@ -2500,6 +2500,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)if (xdp_rxq_info_is_reg (&rxr->xdp_rxq))xdp_rxq_info_unreg (&rxr->xdp_rxq); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;kfree (rxr->rx_tpa);@@ -2560,19 +2561,14 @@ static int bnxt_alloc_rx_rings(struct bnxt *bp)return rc; rc =xdp_rxq_info_reg (&rxr->xdp_rxq, bp->dev, i); -if (rc <0 ) { -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ; +if (rc <0 )
drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_alloc_rx_rings(struct bnxt *bp) · line 2575
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-3return rc; - } rc =xdp_rxq_info_reg_mem_model (&rxr->xdp_rxq,MEM_TYPE_PAGE_POOL , rxr->page_pool);if (rc) {xdp_rxq_info_unreg (&rxr->xdp_rxq); -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ;
Rather than deleting these lines it would also be acceptable to do:
if (rc) {
xdp_rxq_info_unreg(&rxr->xdp_rxq);
- page_pool_free(rxr->page_pool);
+ page_pool_destroy(rxr->page_pool);
rxr->page_pool = NULL;
return rc;
}but anytime there is a failure to bnxt_alloc_rx_rings the driver will immediately follow it up with a call to bnxt_free_rx_rings, so page_pool_destroy will be called.
Thanks for pushing this out so quickly!
Hi,
Quoted 59 lines
Quoted 38 lines
Add
page_pool_destroy()inbnxt_free_rx_rings()during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41("net: core:page_pool: add user refcnt and reintroducepage_pool_destroy")The special error handling code to call
page_pool_free()can now be removed.bnxt_free_rx_rings()will always be called during normal shutdown or any error paths.Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Cc Ilias Apalodimas <[email protected]>Cc Andy Gospodarek <[email protected]>Signed-off-by Michael Chan <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +2
-3@@ -2500,6 +2500,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)if (xdp_rxq_info_is_reg (&rxr->xdp_rxq))xdp_rxq_info_unreg (&rxr->xdp_rxq); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;kfree (rxr->rx_tpa);@@ -2560,19 +2561,14 @@ static int bnxt_alloc_rx_rings(struct bnxt *bp)return rc; rc =xdp_rxq_info_reg (&rxr->xdp_rxq, bp->dev, i); -if (rc <0 ) { -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ; +if (rc <0 )drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_alloc_rx_rings(struct bnxt *bp) · line 2575
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-3return rc; - } rc =xdp_rxq_info_reg_mem_model (&rxr->xdp_rxq,MEM_TYPE_PAGE_POOL , rxr->page_pool);if (rc) {xdp_rxq_info_unreg (&rxr->xdp_rxq); -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ;Rather than deleting these lines it would also be acceptable to do:
if (rc) { xdp_rxq_info_unreg(&rxr->xdp_rxq);drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-1-page_pool_free (rxr->page_pool); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;return rc; }
Quoted 6 lines
but anytime there is a failure to
bnxt_alloc_rx_ringsthe driver will immediately follow it up with a call tobnxt_free_rx_rings, sopage_pool_destroywill be called.Thanks for pushing this out so quickly!
I also can't find page_pool_release_page() or page_pool_put_page() called when destroying the pool. Can you try to insmod -> do some traffic -> rmmod ? If there's stale buffers that haven't been unmapped properly you'll get a WARN_ON for them.
This part was added later on in the API when Jesper fixed in-flight packet handling
Quoted 2 lines
Acked-by Andy Gospodarek <[email protected]>
Thanks
/Ilias
Quoted 65 lines · Ilias Apalodimas
Hi,
Quoted 63 lines
Quoted 38 lines
Add
page_pool_destroy()inbnxt_free_rx_rings()during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41("net: core:page_pool: add user refcnt and reintroducepage_pool_destroy")The special error handling code to call
page_pool_free()can now be removed.bnxt_free_rx_rings()will always be called during normal shutdown or any error paths.Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Cc Ilias Apalodimas <[email protected]>Cc Andy Gospodarek <[email protected]>Signed-off-by Michael Chan <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +2
-3@@ -2500,6 +2500,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)if (xdp_rxq_info_is_reg (&rxr->xdp_rxq))xdp_rxq_info_unreg (&rxr->xdp_rxq); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;kfree (rxr->rx_tpa);@@ -2560,19 +2561,14 @@ static int bnxt_alloc_rx_rings(struct bnxt *bp)return rc; rc =xdp_rxq_info_reg (&rxr->xdp_rxq, bp->dev, i); -if (rc <0 ) { -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ; +if (rc <0 )drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_alloc_rx_rings(struct bnxt *bp) · line 2575
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-3return rc; - } rc =xdp_rxq_info_reg_mem_model (&rxr->xdp_rxq,MEM_TYPE_PAGE_POOL , rxr->page_pool);if (rc) {xdp_rxq_info_unreg (&rxr->xdp_rxq); -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ;Rather than deleting these lines it would also be acceptable to do:
if (rc) { xdp_rxq_info_unreg(&rxr->xdp_rxq);drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-1-page_pool_free (rxr->page_pool); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;return rc; }but anytime there is a failure to
bnxt_alloc_rx_ringsthe driver will immediately follow it up with a call tobnxt_free_rx_rings, sopage_pool_destroywill be called.
Quoted 7 lines
Quoted 2 lines
Thanks for pushing this out so quickly!
I also can't find
page_pool_release_page()orpage_pool_put_page()called when destroying the pool. Can you try to insmod -> do some traffic -> rmmod ? If there's stale buffers that haven't been unmapped properly you'll get aWARN_ONfor them.
I did that test a few times with a few different bpf progs but I do not see any WARN messages. Of course this does not mean that the code we have is 100% correct.
Presumably you are talking about one of these messages, right?
215 /* The distance should not be able to become negative */
216 WARN(inflight < 0, "Negative(%d) inflight packet-pages", inflight);
or
356 /* Drivers should fix this, but only problematic when DMA is used */
357 WARN(1, "Still in-flight pages:%d hold:%u released:%u",
358 distance, hold_cnt, release_cnt);
Quoted 2 lines
This part was added later on in the API when Jesper fixed in-flight packet handling
Quoted 69 lines · Andy Gospodarek
Quoted 68 lines · Ilias Apalodimas
Hi,
Quoted 65 lines
Quoted 38 lines
Add
page_pool_destroy()inbnxt_free_rx_rings()during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41("net: core:page_pool: add user refcnt and reintroducepage_pool_destroy")The special error handling code to call
page_pool_free()can now be removed.bnxt_free_rx_rings()will always be called during normal shutdown or any error paths.Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Cc Ilias Apalodimas <[email protected]>Cc Andy Gospodarek <[email protected]>Signed-off-by Michael Chan <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +2
-3@@ -2500,6 +2500,7 @@ static void bnxt_free_rx_rings(struct bnxt *bp)if (xdp_rxq_info_is_reg (&rxr->xdp_rxq))xdp_rxq_info_unreg (&rxr->xdp_rxq); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;kfree (rxr->rx_tpa);@@ -2560,19 +2561,14 @@ static int bnxt_alloc_rx_rings(struct bnxt *bp)return rc; rc =xdp_rxq_info_reg (&rxr->xdp_rxq, bp->dev, i); -if (rc <0 ) { -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ; +if (rc <0 )drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_alloc_rx_rings(struct bnxt *bp) · line 2575
Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-3return rc; - } rc =xdp_rxq_info_reg_mem_model (&rxr->xdp_rxq,MEM_TYPE_PAGE_POOL , rxr->page_pool);if (rc) {xdp_rxq_info_unreg (&rxr->xdp_rxq); -page_pool_free (rxr->page_pool); - rxr->page_pool =NULL ;Rather than deleting these lines it would also be acceptable to do:
if (rc) { xdp_rxq_info_unreg(&rxr->xdp_rxq);drivers/net/ethernet/broadcom/bnxt/bnxt.c +1
-1-page_pool_free (rxr->page_pool); +page_pool_destroy (rxr->page_pool); rxr->page_pool =NULL ;return rc; }but anytime there is a failure to
bnxt_alloc_rx_ringsthe driver will immediately follow it up with a call tobnxt_free_rx_rings, sopage_pool_destroywill be called.Thanks for pushing this out so quickly!
Quoted 9 lines
Quoted 4 lines
I also can't find
page_pool_release_page()orpage_pool_put_page()called when destroying the pool. Can you try to insmod -> do some traffic -> rmmod ? If there's stale buffers that haven't been unmapped properly you'll get aWARN_ONfor them.I did that test a few times with a few different bpf progs but I do not see any WARN messages. Of course this does not mean that the code we have is 100% correct.
I'll try to have a closer look as well
Quoted 11 lines
Presumably you are talking about one of these messages, right?
215 /* The distance should not be able to become negative */
216WARN(inflight < 0, "Negative(%d) inflight packet-pages", inflight);or
356 /* Drivers should fix this, but only problematic when DMA is used */
357 WARN(1, "Still in-flight pages:%d hold:%u released:%u",
358 distance,hold_cnt,release_cnt);
Yea particularly the second one. There's a counter we increase everytime you alloc a fresh page which needs to be decresed before freeing the whole pool. page_pool_release_page will do that for example
Quoted 3 lines
Quoted 3 lines
This part was added later on in the API when Jesper fixed in-flight packet handling
Thanks
/Ilias
From: Michael Chan <[email protected]>
Date: Tue, 9 Jul 2019 03:50:07 -0400
Quoted 6 lines
Add
page_pool_destroy()inbnxt_free_rx_rings()during normal RX ring cleanup, as Ilias has informed us that the following commit has been merged:
1da4bbeffe41("net: core:page_pool: add user refcnt and reintroducepage_pool_destroy")
Quoted 8 lines
The special error handling code to call
page_pool_free()can now be removed.bnxt_free_rx_rings()will always be called during normal shutdown or any error paths.Fixes 322b87ca55f2 ("bnxt_en: add page_pool support")Cc Ilias Apalodimas <[email protected]>Cc Andy Gospodarek <[email protected]>Signed-off-by Michael Chan <[email protected]>
Applied.