Skip to contentSkip to conversation

xsk: increase NAPI budget for AF_XDP zero-copy path

PATCH In review Björn Töpel 5 participants 14 messages Sep 7, 2020 – Sep 8, 2020

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover xsk: increase NAPI budget for AF_XDP zero-copy path

The NAPI budget (NAPI_POLL_WEIGHT), meaning the number of received packets that are allowed to be processed for each NAPI invocation, takes into consideration that each received packet is aimed for the kernel networking stack.

That is not the case for the AF_XDP receive path, where the cost of each packet is significantly less. Therefore, this commit adds a new NAPI budget, which is the NAPI_POLL_WEIGHT scaled by 4. Typically that would be 256 in most configuration. It is encouraged that AF_XDP zero-copy capable drivers use the XSK_NAPI_WEIGHT, when zero-copy is enabled.

Processing 256 packets targeted for AF_XDP is still less work than 64 (NAPI_POLL_WEIGHT) packets going to the kernel networking stack.

Jakub suggested in [1] that a more generic approach was preferred over "driver hacks". It is arguable if adding the budget as a define which is a scaled NAPI_POLL_WEIGHT would classify as "generic", but it is a bit further away from "driver hacks". ;-)

The first patch adds the actual define, and last three make the Intel drivers use it.

The AF_XDP Rx performance for "rxdrop" is up ~8% on my machine.

Routing this series via bpf-next instead of Intel Wired/netdev, since it is a core AF_XDP addition, and hopefully Nvidia will pick this up for the mlx5 driver.

Bj?rn

[1] https://lore.kernel.org/netdev/20200728131512.17c41621 at kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com/

Bj?rn T?pel (4):
xsk: add XSK_NAPI_WEIGHT define
i40e, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget ice, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 2 +-
 drivers/net/ethernet/intel/ice/ice_xsk.c     | 2 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
 include/net/xdp_sock_drv.h                   | 3 +++
 4 files changed, 6 insertions(+), 3 deletions(-)

base-commit: bc0b5a03079bd78fb3d5cba1ccabf0a7efb1d99f

Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/4 xsk: add XSK_NAPI_WEIGHT define

From: Bj?rn T?pel <[email protected]>

The NAPI budget (NAPI_POLL_WEIGHT), meaning the number of received packets that are allowed to be processed for each NAPI invocation, takes into consideration that each received packet is aimed for the kernel networking stack.

That is not the case for the AF_XDP receive path, where the cost of each packet is significantly less. Therefore, this commit adds a new NAPI budget, which is the NAPI_POLL_WEIGHT scaled by 4. Typically that would be 256 in most configuration. It is encouraged that AF_XDP zero-copy capable drivers use the XSK_NAPI_WEIGHT, when zero-copy is enabled.

Processing 256 packets targeted for AF_XDP is still less work than 64 (NAPI_POLL_WEIGHT) packets going to the kernel networking stack.

---
 include/net/xdp_sock_drv.h | 3 +++
 1 file changed, 3 insertions(+)
include/net/xdp_sock_drv.h +3 -0
index 5b1ee8a9976d..4fc8e931d56f 100644
@@ -7,8 +7,11 @@
 #define _LINUX_XDP_SOCK_DRV_H
 
 #include <net/xdp_sock.h>
+#include <linux/netdevice.h>
 #include <net/xsk_buff_pool.h>
 
+#define XSK_NAPI_WEIGHT (NAPI_POLL_WEIGHT << 2)
+
 #ifdef CONFIG_XDP_SOCKETS
 
 void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries);
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/4 i40e, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

In reply to Björn Töpel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/i40e/i40e_xsk.c +1 -1
index 2a1153d8957b..a8018736ca32 100644
@@ -272,7 +272,7 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget)
 	bool failure = false;
 	struct sk_buff *skb;
 
-	while (likely(total_rx_packets < (unsigned int)budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
 		union i40e_rx_desc *rx_desc;
 		struct xdp_buff **bi;
 		unsigned int size;
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/4 ice, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

In reply to Björn Töpel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
 drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ice/ice_xsk.c +1 -1
index 797886524054..cb473ccdf613 100644
@@ -561,7 +561,7 @@ int ice_clean_rx_irq_zc(struct ice_ring *rx_ring, int budget)
 	unsigned int xdp_xmit = 0;
 	bool failure = false;
 
-	while (likely(total_rx_packets < (unsigned int)budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
 		union ice_32b_rx_flex_desc *rx_desc;
 		unsigned int size, xdp_res = 0;
 		struct ice_rx_buf *rx_buf;
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/4 ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

In reply to Björn Töpel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
index 3771857cf887..f32c1ba0d237 100644
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
 	bool failure = false;
 	struct sk_buff *skb;
 
-	while (likely(total_rx_packets < budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
 		union ixgbe_adv_rx_desc *rx_desc;
 		struct ixgbe_rx_buffer *bi;
 		unsigned int size;
Signature · 1 line
2.25.1
maintainer
#5 lore raw

Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Quoted 15 lines · On Mon, 7 Sep 2020 17:02:17 +0200 Bj?rn T?pel wrote:

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
index 3771857cf887..f32c1ba0d237 100644

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 242

Quoted 5 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
 	bool failure = false;
 	struct sk_buff *skb;
 
-	while (likely(total_rx_packets < budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {

I was thinking that we'd multiply 'budget' here, not replace it with a constant. Looks like ixgbe dutifully passes 'per_ring_budget' into the clean_rx functions, not a complete NAPI budget.

Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
 		union ixgbe_adv_rx_desc *rx_desc;
 		struct ixgbe_rx_buffer *bi;
 		unsigned int size;
#6 lore raw

Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Quoted 19 lines · Jakub Kicinski
Quoted 18 lines · On Mon, 7 Sep 2020 17:02:17 +0200 Bj?rn T?pel wrote:

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
  drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
index 3771857cf887..f32c1ba0d237 100644
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
  	bool failure = false;
  	struct sk_buff *skb;
 
Quoted 7 lines

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 242

Quoted 2 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
-	while (likely(total_rx_packets < budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {

I was thinking that we'd multiply 'budget' here, not replace it with a constant. Looks like ixgbe dutifully passes 'per_ring_budget' into the clean_rx functions, not a complete NAPI budget.

Correct, and i40e/ice does the same ("per_ring_budget").

As for budget << XSK_NAPI_MULT vs replacing; Replacing the budget is more in line with what the drivers do for the Tx cleanup (xxx_clean_tx_irq), where the napi budget is discarded completely; Again, with the idea that "this is much cheaper than a "per-packet through the stack".

Do you prefer the multiplier way that you describe?

Cheers,
Bj?rn

Quoted 4 lines
Quoted 3 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
  		union ixgbe_adv_rx_desc *rx_desc;
  		struct ixgbe_rx_buffer *bi;
  		unsigned int size;
#7 lore raw

Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

In reply to Björn Töpel
Quoted 15 lines · Bj?rn T?pel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
index 3771857cf887..f32c1ba0d237 100644

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 245

Quoted 8 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
 	bool failure = false;
 	struct sk_buff *skb;
 
-	while (likely(total_rx_packets < budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
 		union ixgbe_adv_rx_desc *rx_desc;
 		struct ixgbe_rx_buffer *bi;
 		unsigned int size

This is a violation of NAPI API. IXGBE is already diverging a bit from best practices.

There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?

I would rather use a generic patch.

include/linux/netdevice.h +6 -3
index 7bd4fcdd0738a718d8b0f7134523cd87e4dcdb7b..33bcbdb6fef488983438c6584e3cbb0a44febb1a 100644
@@ -2311,11 +2311,14 @@ static inline void *netdev_priv(const struct net_device *dev)
  */
 #define SET_NETDEV_DEVTYPE(net, devtype)       ((net)->dev.type = (devtype))
 
-/* Default NAPI poll() weight
- * Device drivers are strongly advised to not use bigger value
- */
+/* Default NAPI poll() weight. Highly recommended. */
 #define NAPI_POLL_WEIGHT 64
 
+/* Device drivers are strongly advised to not use bigger value,
+ * as this might cause latencies in stress conditions.
+ */
+#define NAPI_POLL_WEIGHT_MAX 256
+
 /**
  *     netif_napi_add - initialize a NAPI context
  *     @dev:  network device
net/core/dev.c +1 -1
index 4086d335978c1bf62bd3965bd2ea96a4ac06b13d..496713fb6075bd8e5e22725e7c817172858e1dd7 100644
@@ -6608,7 +6608,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi,
        INIT_LIST_HEAD(&napi->rx_list);
        napi->rx_count = 0;
        napi->poll = poll;
-       if (weight > NAPI_POLL_WEIGHT)
+       if (weight > NAPI_POLL_WEIGHT_MAX)
                netdev_err_once(dev, "%s() called with weight %d\n", __func__,
                                weight);
        napi->weight = weight;
#8 lore raw

Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Quoted 21 lines · Eric Dumazet
Quoted 18 lines · Bj?rn T?pel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
  drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +0 -0
index 3771857cf887..f32c1ba0d237 100644
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
  	bool failure = false;
  	struct sk_buff *skb;
 
Quoted 8 lines

drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, · line 245

Quoted 5 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -1
-	while (likely(total_rx_packets < budget)) {
+	while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
  		union ixgbe_adv_rx_desc *rx_desc;
  		struct ixgbe_rx_buffer *bi;
  		unsigned int size

This is a violation of NAPI API. IXGBE is already diverging a bit from best practices.

Thanks for having a look, Eric! By diverging from best practices, do you mean that multiple queues share one NAPI context, and the budget is split over the queues (say, 4 queues, 64/4 per queue), or that Tx simply ignores the budget? Or both?

Quoted 5 lines

There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?

I would rather use a generic patch.

Hmm, so a configurable NAPI budget for, say, the AF_XDP enabled queues/NAPIs? Am I reading that correct? (Note that this is *only* for the AF_XDP enabled queues.)

I'll try to rework this to something more palatable.

Thanks,
Bj?rn

Quoted 36 lines
include/linux/netdevice.h +6 -3
index 7bd4fcdd0738a718d8b0f7134523cd87e4dcdb7b..33bcbdb6fef488983438c6584e3cbb0a44febb1a 100644
@@ -2311,11 +2311,14 @@ static inline void *netdev_priv(const struct net_device *dev)
   */
  #define SET_NETDEV_DEVTYPE(net, devtype)       ((net)->dev.type = (devtype))
 
-/* Default NAPI poll() weight
- * Device drivers are strongly advised to not use bigger value
- */
+/* Default NAPI poll() weight. Highly recommended. */
  #define NAPI_POLL_WEIGHT 64
 
+/* Device drivers are strongly advised to not use bigger value,
+ * as this might cause latencies in stress conditions.
+ */
+#define NAPI_POLL_WEIGHT_MAX 256
+
  /**
   *     netif_napi_add - initialize a NAPI context
   *     @dev:  network device
net/core/dev.c +1 -1
index 4086d335978c1bf62bd3965bd2ea96a4ac06b13d..496713fb6075bd8e5e22725e7c817172858e1dd7 100644
@@ -6608,7 +6608,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi,
         INIT_LIST_HEAD(&napi->rx_list);
         napi->rx_count = 0;
         napi->poll = poll;
-       if (weight > NAPI_POLL_WEIGHT)
+       if (weight > NAPI_POLL_WEIGHT_MAX)
                 netdev_err_once(dev, "%s() called with weight %d\n", __func__,
                                 weight);
         napi->weight = weight;
#9 lore raw

Re: [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Quoted 30 lines · Bj?rn T?pel
Quoted 13 lines · Eric Dumazet
Quoted 10 lines · Bj?rn T?pel

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

---
? drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- ? 1 file changed, 1 insertion(+), 1 deletion(-)

Quoted 15 lines
Quoted 11 lines
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +1 -0
index 3771857cf887..f32c1ba0d237 100644
@@ -239,7 +239,7 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector,
 ????? bool failure = false;
 ????? struct sk_buff *skb;
 ? -??? while (likely(total_rx_packets < budget)) {
+??? while (likely(total_rx_packets < XSK_NAPI_WEIGHT)) {
 ????????? union ixgbe_adv_rx_desc *rx_desc;
 ????????? struct ixgbe_rx_buffer *bi;
Quoted 1 line

????????? unsigned int size

This is a violation of NAPI API. IXGBE is already diverging a bit from best practices.

Quoted 4 lines

Thanks for having a look, Eric! By diverging from best practices, do you mean that multiple queues share one NAPI context, and the budget is split over the queues (say, 4 queues, 64/4 per queue), or that Tx simply ignores the budget? Or both?

For instance, having ixgbe_poll() doing :

return min(work_done, budget - 1);

is quite interesting. It could hide bugs like :

I got a budget of 4, and processed 9999 packets because my maths have a bug, but make sure to pretend we processed 3 packets...

Quoted 18 lines
Quoted 6 lines

There are reasons we want to control the budget from callers, if you want bigger budget just increase it instead of using your own ?

I would rather use a generic patch.

Hmm, so a configurable NAPI budget for, say, the AF_XDP enabled queues/NAPIs? Am I reading that correct? (Note that this is *only* for the AF_XDP enabled queues.)

I'll try to rework this to something more palatable.

Thanks,
Bj?rn

#10 lore raw

Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

In reply to Björn Töpel

Dear Bj?rn,

Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:

Quoted 4 lines

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.

Quoted 4 lines
---
  drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

[?]

Kind regards,

Paul

#11 lore raw

Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Quoted 4 lines · Paul Menzel

Dear Bj?rn,

Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:

Quoted 9 lines
Quoted 4 lines

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.

Paul, thanks for the input! The netdev/bpf trees always include the cover letter in the merge commit.

Cheers,
Bj?rn

Quoted 11 lines
Quoted 4 lines

---
? drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +- ? 1 file changed, 1 insertion(+), 1 deletion(-)

[?]

Kind regards,

Paul

#12 lore raw

Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

Dear Bj?rn,

Am 08.09.20 um 13:12 schrieb Bj?rn T?pel:

Quoted 8 lines · Paul Menzel
Quoted 7 lines

Am 07.09.20 um 17:02 schrieb Bj?rn T?pel:

Quoted 4 lines

From: Bj?rn T?pel <[email protected]>

Start using XSK_NAPI_WEIGHT as NAPI poll budget for the AF_XDP Rx zero-copy path.

Quoted 6 lines
Quoted 3 lines

Could you please add the description from the patch series cover letter to this commit too? To my knowledge, the message in the cover letter won?t be stored in the git repository.

Paul, thanks for the input! The netdev/bpf trees always include the cover letter in the merge commit.

Yes, for pull/merge requests. But you posted them to the list, so I?d assume they will be applied with git am and not merged, or am I missing something. Could you please point me to a merge commit where the patches were posted to the list?

Kind regards,

Paul

#13 lore raw

Re: [Intel-wired-lan] [PATCH bpf-next 4/4] ixgbe, xsk: use XSK_NAPI_WEIGHT as NAPI poll budget

On 2020-09-08 13:20, Paul Menzel wrote:
[...]>> Paul, thanks for the input! The netdev/bpf trees always include the

Quoted 7 lines
Quoted 1 line

cover letter in the merge commit.

Yes, for pull/merge requests. But you posted them to the list, so I?d assume they will be applied with git am and not merged, or am I missing something. Could you please point me to a merge commit where the patches were posted to the list?

An example: A series is posted to the list [1], and when merged the merge commit look like [2].

Thanks,
Bj?rn

[1]
https://lore.kernel.org/bpf/20200520192103.355233-1-bjorn.topel at gmail.com/
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=79917b242c3fe0d89e4752bc25ffef4574c2194b