docs: net: clarify the NAPI rules around XDP Tx
Subsystems (3)
Jakub Kicinski
maintainerPATCH 1/1 docs: net: clarify the NAPI rules around XDP Tx
page pool and XDP should not be accessed from IRQ context which may happen if drivers try to clean up XDP TX with NAPI budget of 0.
---
CC: [email protected]
CC: [email protected]
--- Documentation/networking/napi.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
Documentation/networking/napi.rst +7 -6
@@ -65,15 +65,16 @@ argument - drivers can process completions for any number of Tx packets but should only process up to ``budget`` number of Rx packets. Rx processing is usually much more expensive. -In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -for RX processing. +In other words for Rx processing the ``budget`` argument limits how many +packets driver can process in a single poll. Rx specific APIs like page +pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs. .. warning:: - The ``budget`` argument may be 0 if core tries to only process Tx completions - and no Rx packets. + The ``budget`` argument may be 0 if core tries to only process + skb Tx completions and no Rx or XDP packets. The poll method returns the amount of work done. If the driver still has outstanding work to do (e.g. ``budget`` was exhausted)
Signature · 1 line
2.41.0
Quoted 33 lines
-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: 2023年7月21日 0:13
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Jakub Kicinski <[email protected]>; [email protected]; [email protected]
Subject: [PATCH net] docs: net: clarify the NAPI rules around XDP Txpage pool and XDP should not be accessed from IRQ context which may happen if drivers try to clean up XDP TX with NAPI budget of 0.
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
CC: [email protected]--- Documentation/networking/napi.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)Documentation/networking/napi.rst +0
-0@@ -65,15 +65,16 @@ argument - drivers can process completions for anynumber of Tx packets but should only process up to ``budget`` number of Rx packets. Rx processing is usually much more expensive.
Documentation/networking/napi.rst +0
-3-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -forRX processing.
Documentation/networking/napi.rst
Quoted 5 lines
Documentation/networking/napi.rst +5
-0+In other words for Rx processing the ``budget`` argument limits how +many packets driver can process in a single poll. Rx specific APIs like +page pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs.
Can I ask a stupid question why tx processing cannot call any XDP (or page pool) APIs if the "budget" is 0?
Jakub Kicinski
maintainerQuoted 4 lines · On Fri, 21 Jul 2023 02:35:41 +0000 Wei Fang wrote:
Quoted 4 lines
Documentation/networking/napi.rst +0
-3-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -forRX processing.
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +5
-0+In other words for Rx processing the ``budget`` argument limits how +many packets driver can process in a single poll. Rx specific APIs like +page pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs.Can I ask a stupid question why tx processing cannot call any XDP (or page pool) APIs if the "budget" is 0?
Because in that case we may be in an interrupt context, and page pool assumes it's either in process or softirq context. See commit afbed3f74830 ("net/mlx5e: do as little as possible in napi poll when budget is 0") for an example stack trace.
Quoted 22 lines
-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: 2023年7月21日 11:07
To: Wei Fang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net] docs: net: clarify the NAPI rules around XDP TxQuoted 4 lines · On Fri, 21 Jul 2023 02:35:41 +0000 Wei Fang wrote:
Quoted 4 lines
-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -for RX processing.
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +5
-0+In other words for Rx processing the ``budget`` argument limits how +many packets driver can process in a single poll. Rx specific APIs +like page pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but +if the argument is 0 driver cannot call any XDP (or page pool) APIs.Can I ask a stupid question why tx processing cannot call any XDP (or page pool) APIs if the "budget" is 0?
Quoted 4 lines
Because in that case we may be in an interrupt context, and page pool assumes it's either in process or softirq context. See commit
afbed3f74830("net/mlx5e: do as little as possible in napi poll when budget is 0") for an example stack trace.
I got it, thank you!
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 10 lines · On Thu, 20 Jul 2023 09:13:23 -0700 you wrote:
page pool and XDP should not be accessed from IRQ context which may happen if drivers try to clean up XDP TX with NAPI budget of 0.
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
CC: [email protected][...]
Here is the summary with links:
- [net] docs: net: clarify the NAPI rules around XDP Tx
https://git.kernel.org/netdev/net/c/32ad45b76990
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 20 lines · Jakub Kicinski
page pool and XDP should not be accessed from IRQ context which may happen if drivers try to clean up XDP TX with NAPI budget of 0.
Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]
CC: [email protected]--- Documentation/networking/napi.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)Documentation/networking/napi.rst +0
-0@@ -65,15 +65,16 @@ argument - drivers can process completions for any number of Tx packets but should only process up to ``budget`` number of Rx packets. Rx processing is usually much more expensive.
Documentation/networking/napi.rst argument - drivers can process completions for any number of Tx · line 73
Quoted 10 lines
Documentation/networking/napi.rst +5
-4-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -for RX processing. +In other words for Rx processing the ``budget`` argument limits how many +packets driver can process in a single poll. Rx specific APIs like page +pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs.
This isn't accurate, and I would say it is somewhat dangerous advice. The Tx still needs to be processed regardless of if it is processing page_pool pages or XDP pages. I agree the Rx should not be processed, but the Tx must be processed using mechanisms that do NOT make use of NAPI optimizations when budget is 0.
So specifically, xdp_return_frame is safe in non-NAPI Tx cleanup. The xdp_return_frame_rx_napi is not.
Likewise there is napi_consume_skb which will use either a NAPI or non- NAPI version of things depending on if budget is 0 or not.
For the page_pool calls there is the "allow_direct" argument that is meant to decide between recycling in directly into the page_pool cache or not. It should only be used in the Rx handler itself when budget is non-zero.
I realise this was written up in response to a patch on the Mellanox driver. Based on the patch in question it looks like they were calling page_pool_recycle_direct outside of NAPI context. There is an explicit warning above that function about NOT calling it outside of NAPI context.
Documentation/networking/napi.rst argument - drivers can process completions for any number of Tx · line 80
Quoted 9 lines
Documentation/networking/napi.rst +2
-2.. warning:: - The ``budget`` argument may be 0 if core tries to only process Tx completions - and no Rx packets. + The ``budget`` argument may be 0 if core tries to only process + skb Tx completions and no Rx or XDP packets. The poll method returns the amount of work done. If the driver still has outstanding work to do (e.g. ``budget`` was exhausted)
We cannot make this distinction if both XDP and skb are processed in the same Tx queue. Otherwise you will cause the Tx to stall and break netpoll. If the ring is XDP only then yes, it can be skipped like what they did in the Mellanox driver, but if it is mixed then the XDP side of things needs to use the "safe" versions of the calls.
Jakub Kicinski
maintainerQuoted 27 lines · On Tue, 25 Jul 2023 10:30:24 -0700 Alexander H Duyck wrote:
Documentation/networking/napi.rst argument - drivers can process completions for any number of Tx · line 72
Quoted 9 lines
Documentation/networking/napi.rst +5
-4-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -for RX processing. +In other words for Rx processing the ``budget`` argument limits how many +packets driver can process in a single poll. Rx specific APIs like page +pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs.This isn't accurate, and I would say it is somewhat dangerous advice. The Tx still needs to be processed regardless of if it is processing
page_poolpages or XDP pages. I agree the Rx should not be processed, but the Tx must be processed using mechanisms that do NOT make use of NAPI optimizations when budget is 0.So specifically,
xdp_return_frameis safe in non-NAPI Tx cleanup. Thexdp_return_frame_rx_napiis not.Likewise there is
napi_consume_skbwhich will use either a NAPI or non- NAPI version of things depending on if budget is 0 or not.For the
page_poolcalls there is the "allow_direct" argument that is meant to decide between recycling in directly into thepage_poolcache or not. It should only be used in the Rx handler itself when budget is non-zero.
Quoted 5 lines
I realise this was written up in response to a patch on the Mellanox driver. Based on the patch in question it looks like they were calling
page_pool_recycle_directoutside of NAPI context. There is an explicit warning above that function about NOT calling it outside of NAPI context.
Unless I'm missing something budget=0 can be called from hard IRQ context. And page pool takes _bh() locks. So unless we "teach it" not to recycle _anything_ in hard IRQ context, it is not safe to call.
Quoted 7 lines
Quoted 7 lines
.. warning::
Documentation/networking/napi.rst +2
-2- The ``budget`` argument may be 0 if core tries to only process Tx completions - and no Rx packets. + The ``budget`` argument may be 0 if core tries to only process + skb Tx completions and no Rx or XDP packets.
Quoted 8 lines
Quoted 2 lines
The poll method returns the amount of work done. If the driver still has outstanding work to do (e.g. ``budget`` was exhausted)
We cannot make this distinction if both XDP and skb are processed in the same Tx queue. Otherwise you will cause the Tx to stall and break netpoll. If the ring is XDP only then yes, it can be skipped like what they did in the Mellanox driver, but if it is mixed then the XDP side of things needs to use the "safe" versions of the calls.
IDK, a rare delay in sending of a netpoll message is not a major concern.
Quoted 35 lines · Jakub Kicinski
Quoted 27 lines · On Tue, 25 Jul 2023 10:30:24 -0700 Alexander H Duyck wrote:
Documentation/networking/napi.rst argument - drivers can process completions for any number of Tx · line 72
Quoted 9 lines
Documentation/networking/napi.rst +5
-4-In other words, it is recommended to ignore the budget argument when -performing TX buffer reclamation to ensure that the reclamation is not -arbitrarily bounded; however, it is required to honor the budget argument -for RX processing. +In other words for Rx processing the ``budget`` argument limits how many +packets driver can process in a single poll. Rx specific APIs like page +pool or XDP cannot be used at all when ``budget`` is 0. +skb Tx processing should happen regardless of the ``budget``, but if +the argument is 0 driver cannot call any XDP (or page pool) APIs.This isn't accurate, and I would say it is somewhat dangerous advice. The Tx still needs to be processed regardless of if it is processing
page_poolpages or XDP pages. I agree the Rx should not be processed, but the Tx must be processed using mechanisms that do NOT make use of NAPI optimizations when budget is 0.So specifically,
xdp_return_frameis safe in non-NAPI Tx cleanup. Thexdp_return_frame_rx_napiis not.Likewise there is
napi_consume_skbwhich will use either a NAPI or non- NAPI version of things depending on if budget is 0 or not.For the
page_poolcalls there is the "allow_direct" argument that is meant to decide between recycling in directly into thepage_poolcache or not. It should only be used in the Rx handler itself when budget is non-zero.Quoted 5 lines
I realise this was written up in response to a patch on the Mellanox driver. Based on the patch in question it looks like they were calling
page_pool_recycle_directoutside of NAPI context. There is an explicit warning above that function about NOT calling it outside of NAPI context.
Quoted 3 lines
Unless I'm missing something budget=0 can be called from hard IRQ context. And page pool takes
_bh()locks. So unless we "teach it" not to recycle _anything_ in hard IRQ context, it is not safe to call.
That is the thing. We have to be able to free the pages regardless of context. Otherwise we make a huge mess of things. Also there isn't much way to differentiate between page_pool and non-page_pool pages because an skb can be composed of page pool pages just as easy as an XDP frame can be. All you would just have to enable routing or bridging for Rx frames to end up with page pool pages in the Tx path.
As far as netpoll itself we are safe because it has BH disabled and so as a result page_pool doesn't use the _bh locks. There is code in place to account for that in the producer locking code, and if it were an issue we would have likely blown up long before now. The fact is that page_pool has proliferated into skbs, so you are still freeing page_pool pages indirectly anyway.
That said, there are calls that are not supposed to be used outside of NAPI context, such as page_pool_recycle_direct(). Those have mostly been called out in the page_pool.h header itself, so if someone decides to shoot themselves in the foot with one of those, that is on them. What we need to watch out for are people abusing the "direct" calls and such or just passing "true" for allow_direct in the page_pool calls without taking proper steps to guarantee the context.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
.. warning::
Documentation/networking/napi.rst +2
-2- The ``budget`` argument may be 0 if core tries to only process Tx completions - and no Rx packets. + The ``budget`` argument may be 0 if core tries to only process + skb Tx completions and no Rx or XDP packets. The poll method returns the amount of work done. If the driver still has outstanding work to do (e.g. ``budget`` was exhausted)
Quoted 8 lines
Quoted 5 lines
We cannot make this distinction if both XDP and skb are processed in the same Tx queue. Otherwise you will cause the Tx to stall and break netpoll. If the ring is XDP only then yes, it can be skipped like what they did in the Mellanox driver, but if it is mixed then the XDP side of things needs to use the "safe" versions of the calls.
IDK, a rare delay in sending of a netpoll message is not a major concern.
The whole point of netpoll is to get data out after something like a crash. Otherwise we could have just been using regular NAPI. If the Tx ring is hung it might not be a delay but rather a complete stall that prevents data on the Tx queue from being transmitted on since the system will likely not be recovering. Worse yet is if it is a scenario where the Tx queue can recover it might trigger the Tx watchdog since I could see scenarios where the ring fills, but interrupts were dropped because of the netpoll.
Jakub Kicinski
maintainerQuoted 28 lines · On Tue, 25 Jul 2023 13:10:18 -0700 Alexander Duyck wrote:
Quoted 23 lines · Jakub Kicinski
Quoted 17 lines
This isn't accurate, and I would say it is somewhat dangerous advice. The Tx still needs to be processed regardless of if it is processing
page_poolpages or XDP pages. I agree the Rx should not be processed, but the Tx must be processed using mechanisms that do NOT make use of NAPI optimizations when budget is 0.So specifically,
xdp_return_frameis safe in non-NAPI Tx cleanup. Thexdp_return_frame_rx_napiis not.Likewise there is
napi_consume_skbwhich will use either a NAPI or non- NAPI version of things depending on if budget is 0 or not.For the
page_poolcalls there is the "allow_direct" argument that is meant to decide between recycling in directly into thepage_poolcache or not. It should only be used in the Rx handler itself when budget is non-zero.Quoted 5 lines
I realise this was written up in response to a patch on the Mellanox driver. Based on the patch in question it looks like they were calling
page_pool_recycle_directoutside of NAPI context. There is an explicit warning above that function about NOT calling it outside of NAPI context.Quoted 3 lines
Unless I'm missing something budget=0 can be called from hard IRQ context. And page pool takes
_bh()locks. So unless we "teach it" not to recycle _anything_ in hard IRQ context, it is not safe to call.
Quoted 8 lines
That is the thing. We have to be able to free the pages regardless of context. Otherwise we make a huge mess of things. Also there isn't much way to differentiate between
page_pooland non-page_pool pages because an skb can be composed of page pool pages just as easy as an XDP frame can be. All you would just have to enable routing or bridging for Rx frames to end up with page pool pages in the Tx path.As far as netpoll itself we are safe because it has BH disabled and so
We do? Can you point me to where netpoll disables BH?
Quoted 23 lines
as a result
page_pooldoesn't use the_bhlocks. There is code in place to account for that in the producer locking code, and if it were an issue we would have likely blown up long before now. The fact is thatpage_poolhas proliferated into skbs, so you are still freeingpage_poolpages indirectly anyway.That said, there are calls that are not supposed to be used outside of NAPI context, such as
page_pool_recycle_direct(). Those have mostly been called out in thepage_pool.hheader itself, so if someone decides to shoot themselves in the foot with one of those, that is on them. What we need to watch out for are people abusing the "direct" calls and such or just passing "true" forallow_directin thepage_poolcalls without taking proper steps to guarantee the context.Quoted 8 lines
Quoted 5 lines
We cannot make this distinction if both XDP and skb are processed in the same Tx queue. Otherwise you will cause the Tx to stall and break netpoll. If the ring is XDP only then yes, it can be skipped like what they did in the Mellanox driver, but if it is mixed then the XDP side of things needs to use the "safe" versions of the calls.
IDK, a rare delay in sending of a netpoll message is not a major concern.
Quoted 8 lines
The whole point of netpoll is to get data out after something like a crash. Otherwise we could have just been using regular NAPI. If the Tx ring is hung it might not be a delay but rather a complete stall that prevents data on the Tx queue from being transmitted on since the system will likely not be recovering. Worse yet is if it is a scenario where the Tx queue can recover it might trigger the Tx watchdog since I could see scenarios where the ring fills, but interrupts were dropped because of the netpoll.
I'm not disagreeing with you. I just don't have time to take a deeper look and add the IRQ checks myself and I'm 90% sure the current code can't work with netpoll. So I thought I'd at least document that :(
Quoted 37 lines · Jakub Kicinski
Quoted 35 lines · On Tue, 25 Jul 2023 13:10:18 -0700 Alexander Duyck wrote:
Quoted 23 lines · Jakub Kicinski
Quoted 17 lines
This isn't accurate, and I would say it is somewhat dangerous advice. The Tx still needs to be processed regardless of if it is processing
page_poolpages or XDP pages. I agree the Rx should not be processed, but the Tx must be processed using mechanisms that do NOT make use of NAPI optimizations when budget is 0.So specifically,
xdp_return_frameis safe in non-NAPI Tx cleanup. Thexdp_return_frame_rx_napiis not.Likewise there is
napi_consume_skbwhich will use either a NAPI or non- NAPI version of things depending on if budget is 0 or not.For the
page_poolcalls there is the "allow_direct" argument that is meant to decide between recycling in directly into thepage_poolcache or not. It should only be used in the Rx handler itself when budget is non-zero.Quoted 5 lines
I realise this was written up in response to a patch on the Mellanox driver. Based on the patch in question it looks like they were calling
page_pool_recycle_directoutside of NAPI context. There is an explicit warning above that function about NOT calling it outside of NAPI context.Quoted 3 lines
Unless I'm missing something budget=0 can be called from hard IRQ context. And page pool takes
_bh()locks. So unless we "teach it" not to recycle _anything_ in hard IRQ context, it is not safe to call.That is the thing. We have to be able to free the pages regardless of context. Otherwise we make a huge mess of things. Also there isn't much way to differentiate between
page_pooland non-page_pool pages because an skb can be composed of page pool pages just as easy as an XDP frame can be. All you would just have to enable routing or bridging for Rx frames to end up with page pool pages in the Tx path.
Quoted 3 lines
Quoted 1 line
As far as netpoll itself we are safe because it has BH disabled and so
We do? Can you point me to where netpoll disables BH?
I misread the code. Basically it looks like netconsole is explicitly disabling interrupts via spin_lock_irqsave in write_msg is what is going on.
Quoted 32 lines
Quoted 23 lines
as a result
page_pooldoesn't use the_bhlocks. There is code in place to account for that in the producer locking code, and if it were an issue we would have likely blown up long before now. The fact is thatpage_poolhas proliferated into skbs, so you are still freeingpage_poolpages indirectly anyway.That said, there are calls that are not supposed to be used outside of NAPI context, such as
page_pool_recycle_direct(). Those have mostly been called out in thepage_pool.hheader itself, so if someone decides to shoot themselves in the foot with one of those, that is on them. What we need to watch out for are people abusing the "direct" calls and such or just passing "true" forallow_directin thepage_poolcalls without taking proper steps to guarantee the context.Quoted 8 lines
Quoted 5 lines
We cannot make this distinction if both XDP and skb are processed in the same Tx queue. Otherwise you will cause the Tx to stall and break netpoll. If the ring is XDP only then yes, it can be skipped like what they did in the Mellanox driver, but if it is mixed then the XDP side of things needs to use the "safe" versions of the calls.
IDK, a rare delay in sending of a netpoll message is not a major concern.
Quoted 8 lines
The whole point of netpoll is to get data out after something like a crash. Otherwise we could have just been using regular NAPI. If the Tx ring is hung it might not be a delay but rather a complete stall that prevents data on the Tx queue from being transmitted on since the system will likely not be recovering. Worse yet is if it is a scenario where the Tx queue can recover it might trigger the Tx watchdog since I could see scenarios where the ring fills, but interrupts were dropped because of the netpoll.
Quoted 3 lines
I'm not disagreeing with you. I just don't have time to take a deeper look and add the IRQ checks myself and I'm 90% sure the current code can't work with netpoll. So I thought I'd at least document that :(
So looking at it more I realized the way we are getting around the issue is that the skbuffs are ALWAYS freed in softirq context. Basically we hand them off to dev_consume_skb_any, which will hand them off to dev_kfree_skb_irq_reason, and it is queueing them up to be processed in the net_tx_action handler.
As far as the page pool pages themselves I wonder if we couldn't just look at modifying __page_pool_put_page() so that it had something similar to dev_consume_skb_any_reason() so if we are in a hardirq or IRQs are disabled we just force the page to be freed.
Jakub Kicinski
maintainerQuoted 5 lines · On Tue, 25 Jul 2023 17:02:42 -0700 Alexander Duyck wrote:
So looking at it more I realized the way we are getting around the issue is that the skbuffs are ALWAYS freed in softirq context. Basically we hand them off to
dev_consume_skb_any, which will hand them off todev_kfree_skb_irq_reason, and it is queueing them up to be processed in thenet_tx_actionhandler.
SG.
Quoted 4 lines
As far as the page pool pages themselves I wonder if we couldn't just look at modifying
__page_pool_put_page()so that it had something similar todev_consume_skb_any_reason()so if we are in a hardirq or IRQs are disabled we just force the page to be freed.
Yup (same for the bulk API). I think that Olek was trying to implement this somehow nicely, not sure how far he got.