net: xsk: introduce XDP_MAX_TX_BUDGET setsockopt
Subsystems (5)
PATCH v5 1/1 net: xsk: introduce XDP_MAX_TX_BUDGET setsockopt
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control.
The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.
Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572
[1]: https://lore.kernel.org/all/[email protected]/
---
v5
1. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.
V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message
V2
1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.
--- Documentation/networking/af_xdp.rst | 8 ++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 20 ++++++++++++++++---- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 27 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +8 -0
@@ -442,6 +442,14 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the non-zero +copy mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. Default is 32. + XDP_STATISTICS getsockopt -------------------------
include/net/xdp_sock.h +1 -0
@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */
net/xdp/xsk.c +16 -4
@@ -33,8 +33,7 @@#include "xdp_umem.h" #include "xsk.h" -#define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {
@@ -779,7 +778,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_budget =READ_ONCE (xs->max_tx_budget);bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;
@@ -797,7 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk) goto out;while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) { -if (max_batch-- ==0 ) { +if (max_budget-- ==0 ) { err = -EAGAIN ; goto out; }
@@ -1437,6 +1436,18 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_MAX_TX_BUDGET : + { +unsigned int budget; + +if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; + +WRITE_ONCE (xs->max_tx_budget,max (budget,1 )); +return 0 ; + }default :break ; }
@@ -1734,6 +1745,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =32 ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);
tools/include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */
Signature · 1 line
2.41.3
Quoted 155 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_BUDGETsocket option.
- ConvertTX_BATCH_SIZEtx_budget_spent.
- Settx_budget_spentto 32 by default in the initialization phase as a per-socket granular control.The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v51. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 8 ++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 20 ++++++++++++++++---- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 27 insertions(+), 4 deletions(-)Documentation/networking/af_xdp.rst +8
-0@@ -442,6 +442,14 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the non-zero +copy mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. Default is 32. + XDP_STATISTICS getsockopt -------------------------include/net/xdp_sock.h +1
-0@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */ include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ net/xdp/xsk.c +3
-4@@ -33,8 +33,7 @@#include "xdp_umem.h" #include "xsk.h" -#define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {@@ -779,7 +778,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_budget =READ_ONCE (xs->max_tx_budget);bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;@@ -797,7 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk) goto out;while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) { -if (max_batch-- ==0 ) { +if (max_budget-- ==0 ) { err = -EAGAIN ; goto out; }@@ -1437,6 +1436,18 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; }
net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1448
Quoted 10 lines
net/xdp/xsk.c +10
-0+case XDP_MAX_TX_BUDGET : + { +unsigned int budget; + +if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; + +WRITE_ONCE (xs->max_tx_budget,max (budget,1 ));
I still think that this needs a more sane upper bound than U32_MAX.
One limiting factor is the XSK TxQ length. At least it should be possible to fail if trying to set beyond that.
Quoted 28 lines
net/xdp/xsk.c +3
-0+return 0 ; + }default :break ; }@@ -1734,6 +1745,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =32 ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);tools/include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ Signature · 1 line
2.41.3
Maciej Fijalkowski
maintainerQuoted 169 lines · Willem de Bruijn
Quoted 155 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_BUDGETsocket option.
- ConvertTX_BATCH_SIZEtx_budget_spent.
- Settx_budget_spentto 32 by default in the initialization phase as a per-socket granular control.The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v51. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 8 ++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 20 ++++++++++++++++---- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 27 insertions(+), 4 deletions(-)Documentation/networking/af_xdp.rst +8
-0@@ -442,6 +442,14 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the non-zero +copy mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. Default is 32. + XDP_STATISTICS getsockopt -------------------------include/net/xdp_sock.h +1
-0@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */ include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ net/xdp/xsk.c +3
-4@@ -33,8 +33,7 @@#include "xdp_umem.h" #include "xsk.h" -#define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {@@ -779,7 +778,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_budget =READ_ONCE (xs->max_tx_budget);bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;@@ -797,7 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk) goto out;while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) { -if (max_batch-- ==0 ) { +if (max_budget-- ==0 ) { err = -EAGAIN ; goto out; }@@ -1437,6 +1436,18 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; }net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1448
Quoted 10 lines
net/xdp/xsk.c +10
-0+case XDP_MAX_TX_BUDGET : + { +unsigned int budget; + +if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; + +WRITE_ONCE (xs->max_tx_budget,max (budget,1 ));I still think that this needs a more sane upper bound than
U32_MAX.
Quoted 2 lines
One limiting factor is the XSK TxQ length. At least it should be possible to fail if trying to set beyond that.
+1 and I don't really see a reason for something below 32. So how about [32, xs->tx->nentries] range?
Also if there's no xsk tx ring present we could bail out.
Quoted 31 lines
Quoted 29 lines
net/xdp/xsk.c +3
-0+return 0 ; + }default :break ; }@@ -1734,6 +1745,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =32 ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);tools/include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ Signature · 1 line
2.41.3
Quoted 162 lines · Maciej Fijalkowski
Quoted 160 lines · Willem de Bruijn
Quoted 159 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_BUDGETsocket option.
- ConvertTX_BATCH_SIZEtx_budget_spent.
- Settx_budget_spentto 32 by default in the initialization phase as a per-socket granular control.The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v51. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 8 ++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 20 ++++++++++++++++---- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 27 insertions(+), 4 deletions(-)Documentation/networking/af_xdp.rst +8
-0@@ -442,6 +442,14 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the non-zero +copy mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. Default is 32. + XDP_STATISTICS getsockopt -------------------------include/net/xdp_sock.h +1
-0@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */ include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ net/xdp/xsk.c +7
-4@@ -33,8 +33,7 @@#include "xdp_umem.h" #include "xsk.h" -#define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {@@ -779,7 +778,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_budget =READ_ONCE (xs->max_tx_budget);bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;@@ -797,7 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk) goto out;while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) { -if (max_batch-- ==0 ) { +if (max_budget-- ==0 ) { err = -EAGAIN ; goto out; }@@ -1437,6 +1436,18 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_MAX_TX_BUDGET : + { +unsigned int budget; +
Quoted 8 lines
Quoted 8 lines
net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1448
Quoted 6 lines
net/xdp/xsk.c +6
-0+if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; + +WRITE_ONCE (xs->max_tx_budget,max (budget,1 ));I still think that this needs a more sane upper bound than
U32_MAX.
The reason why I didn't touch is that I refer to some existing mechanisms, like net.ipv4.tcp_limit_output_bytes that also doesn't constrain its max value for a single flow.
I really don't think it matters because 1) normally no one sets that large value to hope one send call can deal with that many desc, 2) the so-called proper value that may do harm to some unexpected cases which I'm unware of for now. I wonder why not give the user a choice. After all, it's his own decision.
Quoted 5 lines
Quoted 3 lines
One limiting factor is the XSK TxQ length. At least it should be possible to fail if trying to set beyond that.
+1 and I don't really see a reason for something below 32.
Really? U32_MAX is a very very large value.
Quoted 4 lines
So how about
[32,xs->tx->nentries] range?Also if there's no xsk tx ring present we could bail out.
Sorry, I'm not sure about this. It seems to add more dependencies. But my opinion on the lower bound is one just like dev_weight :)
Thanks,
Jason
Quoted 32 lines
Quoted 32 lines
Quoted 30 lines
net/xdp/xsk.c +3
-0+return 0 ; + }default :break ; }@@ -1734,6 +1745,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =32 ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);tools/include/uapi/linux/if_xdp.h +1
-1@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ --2.41.3
Quoted 185 lines · Jason Xing
Quoted 162 lines · Maciej Fijalkowski
Quoted 160 lines · Willem de Bruijn
Quoted 159 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_BUDGETsocket option.
- ConvertTX_BATCH_SIZEtx_budget_spent.
- Settx_budget_spentto 32 by default in the initialization phase as a per-socket granular control.The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v51. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 8 ++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 20 ++++++++++++++++---- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 27 insertions(+), 4 deletions(-)Documentation/networking/af_xdp.rst +8
-0@@ -442,6 +442,14 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the non-zero +copy mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. Default is 32. + XDP_STATISTICS getsockopt -------------------------include/net/xdp_sock.h +1
-0@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */ include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ net/xdp/xsk.c +7
-4@@ -33,8 +33,7 @@#include "xdp_umem.h" #include "xsk.h" -#define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {@@ -779,7 +778,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_budget =READ_ONCE (xs->max_tx_budget);bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;@@ -797,7 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk) goto out;while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) { -if (max_batch-- ==0 ) { +if (max_budget-- ==0 ) { err = -EAGAIN ; goto out; }@@ -1437,6 +1436,18 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_MAX_TX_BUDGET : + { +unsigned int budget; +Quoted 8 lines
Quoted 8 lines
net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1448
Quoted 6 lines
net/xdp/xsk.c +6
-0+if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; + +WRITE_ONCE (xs->max_tx_budget,max (budget,1 ));I still think that this needs a more sane upper bound than
U32_MAX.The reason why I didn't touch is that I refer to some existing mechanisms, like net.ipv4.tcp_limit_output_bytes that also doesn't constrain its max value for a single flow.
I really don't think it matters because 1) normally no one sets that large value to hope one send call can deal with that many desc, 2) the so-called proper value that may do harm to some unexpected cases which I'm unware of for now. I wonder why not give the user a choice. After all, it's his own decision.
Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
One limiting factor is the XSK TxQ length. At least it should be possible to fail if trying to set beyond that.
+1 and I don't really see a reason for something below 32.
Really?
U32_MAXis a very very large value.Quoted 2 lines
So how about
[32,xs->tx->nentries] range?
Sorry, I still think one is an appropriate lower value because setting a lower value than 32 is not that a big bad thing for users or even not a bad thing at all. The first version of the value is 16 which had been used for a while. I guess Willem is worried that a too large value may have an adverse impact on the system/xsk flow?
Quoted 5 lines
Quoted 2 lines
Also if there's no xsk tx ring present we could bail out.
Sorry, I'm not sure about this. It seems to add more dependencies. But my opinion on the lower bound is one just like
dev_weight:)
After rethinking setting xs->tx->nentries as its upper bound, it makes sense to me overall. But I'm not sure if such a case can happen in the real world:
1) CPU 0: keep calling sendto() or poll() syscall to driver the xsk to send the packets (in __xsk_generic_xmit())
2) CPU 1: keep writing the data into the tx ring in the application (like by using xsk_ring_prod__submit(xsk->tx, index))
?
If the asynchrony of writing and sending desc might happen, the 'xs->tx->nentries' doesn't have a strong underlying theory to support any more. At least, AFAIK, with my limited knowledge, we don't prohibit users from trying the above scenario, right?
Since the patch was marked as change-requested, please allow me to ask one more time: if we really need a valid max number for it?
Thanks,
Jason
Quoted 36 lines
Thanks,
JasonQuoted 32 lines
Quoted 32 lines
Quoted 30 lines
net/xdp/xsk.c +3
-0+return 0 ; + }default :break ; }@@ -1734,6 +1745,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =32 ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);tools/include/uapi/linux/if_xdp.h +1
-1@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ --2.41.3