xsk: increase NAPI budget for AF_XDP zero-copy path
Subsystems (5)
cover xsk: increase NAPI budget for AF_XDP zero-copy path
The NAPI budget (NAPI_POLL_WEIGHT), meaning the number of received packets that are allowed to be processed for each NAPI invocation, takes into consideration that each received packet is aimed for the kernel networking stack.
That is not the case for the AF_XDP receive path, where the cost of each packet is significantly less. Therefore, this commit adds a new NAPI budget, which is the NAPI_POLL_WEIGHT scaled by 4. Typically that would be 256 in most configuration. It is encouraged that AF_XDP zero-copy capable drivers use the XSK_NAPI_WEIGHT, when zero-copy is enabled.
Processing 256 packets targeted for AF_XDP is still less work than 64 (NAPI_POLL_WEIGHT) packets going to the kernel networking stack.
Jakub suggested in [1] that a more generic approach was preferred over "driver hacks". It is arguable if adding the budget as a define which is a scaled NAPI_POLL_WEIGHT would classify as "generic", but it is a bit further away from "driver hacks". ;-)
The first patch adds the actual define, and last three make the Intel drivers use it.
The AF_XDP Rx performance for "rxdrop" is up ~8% on my machine.
Routing this series via bpf-next instead of Intel Wired/netdev, since it is a core AF_XDP addition, and hopefully Nvidia will pick this up for the mlx5 driver.
Bj?rn
[1] https://lore.kernel.org/netdev/20200728131512.17c41621 at kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com/
Bj?rn T?pel (4):
xsk: add XSK_NAPI_WEIGHT define
i40e, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget ice, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +- drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- include/net/xdp_sock_drv.h | 3 +++ 4 files changed, 6 insertions(+), 3 deletions(-)
base-commit: bc0b5a03079bd78fb3d5cba1ccabf0a7efb1d99f
Signature · 1 line
2.25.1
PATCH 1/4 xsk: add XSK_NAPI_WEIGHT define
From: Bj?rn T?pel <[email protected]>
The NAPI budget (NAPI_POLL_WEIGHT), meaning the number of received packets that are allowed to be processed for each NAPI invocation, takes into consideration that each received packet is aimed for the kernel networking stack.
That is not the case for the AF_XDP receive path, where the cost of each packet is significantly less. Therefore, this commit adds a new NAPI budget, which is the NAPI_POLL_WEIGHT scaled by 4. Typically that would be 256 in most configuration. It is encouraged that AF_XDP zero-copy capable drivers use the XSK_NAPI_WEIGHT, when zero-copy is enabled.
Processing 256 packets targeted for AF_XDP is still less work than 64 (NAPI_POLL_WEIGHT) packets going to the kernel networking stack.
--- include/net/xdp_sock_drv.h | 3 +++ 1 file changed, 3 insertions(+)
include/net/xdp_sock_drv.h +3 -0
@@ -7,8 +7,11 @@#define _LINUX_XDP_SOCK_DRV_H#include <net/xdp_sock.h> +#include <linux/netdevice.h> #include <net/xsk_buff_pool.h> +#define XSK_NAPI_WEIGHT (NAPI_POLL_WEIGHT << 2) +#ifdef CONFIG_XDP_SOCKETS void xsk_tx_completed (struct xsk_buff_pool *pool,u32 nb_entries);
Signature · 1 line
2.25.1
PATCH 2/4 i40e, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
In reply to Björn TöpelFrom: Bj?rn T?pel <[email protected]>
Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.
--- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +1 -1
@@ -272,7 +272,7 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget)bool failure = false;struct sk_buff *skb; -while (likely (total_rx_packets < (unsigned int )budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {union i40e_rx_desc *rx_desc;struct xdp_buff **bi;unsigned int size;
Signature · 1 line
2.25.1
PATCH 3/4 ice, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
In reply to Björn TöpelFrom: Bj?rn T?pel <[email protected]>
Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.
--- drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ice/ice_xsk.c +1 -1
@@ -561,7 +561,7 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget)unsigned int xdp_xmit =0 ;bool failure = false; -while (likely (total_rx_packets < (unsigned int )budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {union ice_32b_rx_flex_desc *rx_desc;unsigned int size, xdp_res =0 ;struct ice_rx_buf *rx_buf;
Signature · 1 line
2.25.1
PATCH 4/4 ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
In reply to Björn TöpelFrom: Bj?rn T?pel <[email protected]>
Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.
--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,bool failure = false;struct sk_buff *skb; -while (likely (total_rx_packets < budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;unsigned int size;
Signature · 1 line
2.25.1
Jakub Kicinski
maintainerRe: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Quoted 15 lines · On Mon, 7 Sep 2020 17:02:17 +0200 Bj?rn T?pel wrote:
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Signed-off-by Bj?rn T?pel <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 242
Quoted 5 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1
-1@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,bool failure = false;struct sk_buff *skb; -while (likely (total_rx_packets < budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {
I was thinking that we'd multiply 'budget' here, not replace it with a constant. Looks like ixgbe dutifully passes 'per_ring_budget' into the clean_rx functions, not a complete NAPI budget.
Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;unsigned int size;
Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Quoted 19 lines · Jakub Kicinski
Quoted 18 lines · On Mon, 7 Sep 2020 17:02:17 +0200 Bj?rn T?pel wrote:
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Signed-off-by Bj?rn T?pel <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,bool failure = false;struct sk_buff *skb;
Quoted 7 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 242
Quoted 2 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1
-1-while (likely (total_rx_packets < budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {I was thinking that we'd multiply 'budget' here, not replace it with a constant. Looks like ixgbe dutifully passes '
per_ring_budget' into theclean_rxfunctions, not a complete NAPI budget.
Correct, and i40e/ice does the same ("per_ring_budget").
As for budget << XSK_NAPI_MULT vs replacing; Replacing the budget is more in line with what the drivers do for the Tx cleanup (xxx_clean_tx_irq), where the napi budget is discarded completely; Again, with the idea that "this is much cheaper than a "per-packet through the stack".
Do you prefer the multiplier way that you describe?
Cheers,
Bj?rn
Quoted 4 lines
Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;unsigned int size;
Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
In reply to Björn TöpelQuoted 15 lines · Bj?rn T?pel
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Signed-off-by Bj?rn T?pel <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 245
Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1
-1@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,bool failure = false;struct sk_buff *skb; -while (likely (total_rx_packets < budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;unsigned int size
This is a violation of NAPI API. IXGBE is already diverging a bit from best practices.
There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?
I would rather use a generic patch.
include/linux/netdevice.h +6 -3
@@ -2311,11 +2311,14 @@ static inline void *netdev_priv(const struct net_device *dev)*/ #define SET_NETDEV_DEVTYPE (net, devtype) ((net)->dev.type = (devtype)) -/* Default NAPI poll() weight -* Device drivers are strongly advised to not use bigger value -*/ +/* Default NAPI poll() weight. Highly recommended. */ #define NAPI_POLL_WEIGHT 64 +/* Device drivers are strongly advised to not use bigger value, +* as this might cause latencies in stress conditions. +*/ +#define NAPI_POLL_WEIGHT_MAX 256 +/** * netif_napi_add - initialize a NAPI context * @dev: network device
net/core/dev.c +1 -1
@@ -6608,7 +6608,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi,INIT_LIST_HEAD (&napi->rx_list); napi->rx_count =0 ; napi->poll = poll; -if (weight >NAPI_POLL_WEIGHT ) +if (weight >NAPI_POLL_WEIGHT_MAX )netdev_err_once (dev,"%s() called with weight %d\n" , __func__, weight); napi->weight = weight;
Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Quoted 21 lines · Eric Dumazet
Quoted 18 lines · Bj?rn T?pel
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Signed-off-by Bj?rn T?pel <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0
-0@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,bool failure = false;struct sk_buff *skb;
Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 245
Quoted 5 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1
-1-while (likely (total_rx_packets < budget)) { +while (likely (total_rx_packets <XSK_NAPI_WEIGHT )) {union ixgbe_adv_rx_desc *rx_desc;struct ixgbe_rx_buffer *bi;unsigned int sizeThis is a violation of NAPI API. IXGBE is already diverging a bit from best practices.
Thanks for having a look, Eric! By diverging from best practices, do you mean that multiple queues share one NAPI context, and the budget is split over the queues (say, 4 queues, 64/4 per queue), or that Tx simply ignores the budget? Or both?
Quoted 5 lines
There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?
I would rather use a generic patch.
Hmm, so a configurable NAPI budget for, say, the AF_XDP enabled queues/NAPIs? Am I reading that correct? (Note that this is *only* for the AF_XDP enabled queues.)
I'll try to rework this to something more palatable.
Thanks,
Bj?rn
Quoted 36 lines
include/linux/netdevice.h +6
-3@@ -2311,11 +2311,14 @@ static inline void *netdev_priv(const struct net_device *dev)*/ #define SET_NETDEV_DEVTYPE (net, devtype) ((net)->dev.type = (devtype)) -/* Default NAPI poll() weight -* Device drivers are strongly advised to not use bigger value -*/ +/* Default NAPI poll() weight. Highly recommended. */ #define NAPI_POLL_WEIGHT 64 +/* Device drivers are strongly advised to not use bigger value, +* as this might cause latencies in stress conditions. +*/ +#define NAPI_POLL_WEIGHT_MAX 256 +/** * netif_napi_add - initialize a NAPI context * @dev: network device net/core/dev.c +1
-1@@ -6608,7 +6608,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi,INIT_LIST_HEAD (&napi->rx_list); napi->rx_count =0 ; napi->poll = poll; -if (weight >NAPI_POLL_WEIGHT ) +if (weight >NAPI_POLL_WEIGHT_MAX )netdev_err_once (dev,"%s() called with weight %d\n" , __func__, weight); napi->weight = weight;
Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Quoted 30 lines · Bj?rn T?pel
Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Bj?rn T?pel
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Signed-off-by Bj?rn T?pel <[email protected]>---
?drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c| 2 +- ? 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 15 lines
Quoted 11 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1
-0@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, ????? bool failure = false; ????? struct sk_buff *skb; ? -??? while (likely(total_rx_packets < budget)) { +??? while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) { ????????? union ixgbe_adv_rx_desc *rx_desc; ????????? struct ixgbe_rx_buffer *bi;Quoted 1 line
????????? unsigned int size
This is a violation of NAPI API. IXGBE is already diverging a bit from best practices.
Quoted 4 lines
Thanks for having a look, Eric! By diverging from best practices, do you mean that multiple queues share one NAPI context, and the budget is split over the queues (say, 4 queues, 64/4 per queue), or that Tx simply ignores the budget? Or both?
For instance, having ixgbe_poll() doing :
return min(work_done, budget - 1);
is quite interesting. It could hide bugs like :
I got a budget of 4, and processed 9999 packets because my maths have a bug, but make sure to pretend we processed 3 packets...
Quoted 18 lines
Quoted 6 lines
There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?
I would rather use a generic patch.
Hmm, so a configurable NAPI budget for, say, the
AF_XDPenabled queues/NAPIs? Am I reading that correct? (Note that this is *only* for theAF_XDPenabled queues.)I'll try to rework this to something more palatable.
Thanks,
Bj?rn
Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
In reply to Björn TöpelDear Bj?rn,
Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:
Quoted 4 lines
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.
Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.
Quoted 4 lines
Signed-off-by Bj?rn T?pel <[email protected]>--- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[?]
Kind regards,
Paul
Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Quoted 4 lines · Paul Menzel
Dear Bj?rn,
Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:
Quoted 9 lines
Quoted 4 lines
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.
Paul, thanks for the input! The netdev/bpf trees always include the cover letter in the merge commit.
Cheers,
Bj?rn
Quoted 11 lines
Quoted 4 lines
Signed-off-by Bj?rn T?pel <[email protected]>---
?drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c| 2 +- ? 1 file changed, 1 insertion(+), 1 deletion(-)[?]
Kind regards,
Paul
Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
Dear Bj?rn,
Am 08.09.20 um 13:12 schrieb Bj?rn T?pel:
Quoted 8 lines · Paul Menzel
Quoted 7 lines
Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:
Quoted 4 lines
From: Bj?rn T?pel <[email protected]>
Start using
XSK_NAPI_WEIGHTas NAPI poll budget for theAF_XDPRx zero-copy path.
Quoted 6 lines
Quoted 3 lines
Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.
Paul, thanks for the input! The netdev/bpf trees always include the cover letter in the merge commit.
Yes, for pull/merge requests. But you posted them to the list, so I?d assume they will be applied with git am and not merged, or am I missing something. Could you please point me to a merge commit where the patches were posted to the list?
Kind regards,
Paul
Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget
On 2020-09-08 13:20, Paul Menzel wrote:
[...]>> Paul, thanks for the input! The netdev/bpf trees always include the
Quoted 7 lines
Quoted 1 line
cover letter in the merge commit.
Yes, for pull/merge requests. But you posted them to the list, so I?d assume they will be applied with
git amand not merged, or am I missing something. Could you please point me to a merge commit where the patches were posted to the list?
An example: A series is posted to the list [1], and when merged the merge commit look like [2].
Thanks,
Bj?rn
[1]
https://lore.kernel.org/bpf/20200520192103.355233-1-bjorn.topel at gmail.com/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=79917b242c3fe0d89e4752bc25ffef4574c2194b