net: xsk: introduce XDP_MAX_TX_SKB_BUDGET setsockopt
Subsystems (5)
PATCH v7 1/1 net: xsk: introduce XDP_MAX_TX_SKB_BUDGET setsockopt
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- Add XDP_MAX_TX_SKB_BUDGET socket option.
- Set max_tx_budget to 32 by default in the initialization phase as a per-socket granular control.
- Set the range of max_tx_budget as [32, xs->tx->nentries].
The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_SKB_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.
Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_SKB_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572
[1]: https://lore.kernel.org/all/[email protected]/
---
v7
1. use 'copy mode' in Doc
2. move init of max_tx_budget to a proper position
3. use the max value in the if condition in setsockopt
4. change sockopt name to XDP_MAX_TX_SKB_BUDGET
5. set MAX_PER_SOCKET_BUDGET to 32 instead of TX_BATCH_SIZE because they have no correlation at all.
v6
1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.
v5
1. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.
V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message
V2
1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.
--- Documentation/networking/af_xdp.rst | 9 +++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 21 +++++++++++++++++++-- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 31 insertions(+), 2 deletions(-)
Documentation/networking/af_xdp.rst +9 -0
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_SKB_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the copy +mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. +Allowed range is [32, xs->tx->nentries]. + XDP_STATISTICS getsockopt -------------------------
include/net/xdp_sock.h +1 -0
@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_SKB_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */
net/xdp/xsk.c +19 -2
@@ -34,7 +34,7 @@#include "xsk.h" #define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_batch;bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;
@@ -796,6 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk)if (xs->queue_id >= xs->dev->real_num_tx_queues) goto out; + max_batch =READ_ONCE (xs->max_tx_budget);while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) {if (max_batch-- ==0 ) { err = -EAGAIN ;
@@ -1437,6 +1438,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_MAX_TX_SKB_BUDGET : + { +unsigned int budget; + +if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; +if (!xs->tx || + budget <TX_BATCH_SIZE || budget > xs->tx->nentries) +return -EACCES ; + +WRITE_ONCE (xs->max_tx_budget, budget); +return 0 ; + }default :break ; }
@@ -1734,6 +1750,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =TX_BATCH_SIZE ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);
tools/include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_SKB_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */
Signature · 1 line
2.41.3
Quoted 149 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_SKB_BUDGETsocket option.
- Setmax_tx_budgetto 32 by default in the initialization phase as a per-socket granular control.
- Set the range ofmax_tx_budgetas [32,xs->tx->nentries].The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_SKB_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_SKB_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v71. use 'copy mode' in Doc
2. move init ofmax_tx_budgetto a proper position
3. use the max value in the if condition in setsockopt
4. change sockopt name toXDP_MAX_TX_SKB_BUDGET
5. setMAX_PER_SOCKET_BUDGETto 32 instead ofTX_BATCH_SIZEbecause they have no correlation at all.v6
1. use [32,
xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.v5
1. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 9 +++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 21 +++++++++++++++++++-- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 31 insertions(+), 2 deletions(-)Documentation/networking/af_xdp.rst +9
-0@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_SKB_BUDGET setsockopt +---------------------------- + +This setsockopt sets the maximum number of descriptors that can be handled +and passed to the driver at one send syscall. It is applied in the copy +mode to allow application to tune the per-socket maximum iteration for +better throughput and less frequency of send syscall. +Allowed range is [32, xs->tx->nentries]. + XDP_STATISTICS getsockopt -------------------------include/net/xdp_sock.h +1
-0@@ -84,6 +84,7 @@ struct xdp_sock {struct list_head map_list;/* Protects map_list */ spinlock_t map_list_lock; +u32 max_tx_budget;/* Protects multiple processes in the control path */ struct mutex mutex;struct xsk_queue *fq_tmp;/* Only as tmp storage before bind */ include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_SKB_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ net/xdp/xsk.c +1
-1@@ -34,7 +34,7 @@#include "xsk.h" #define TX_BATCH_SIZE 32 -#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) +#define MAX_PER_SOCKET_BUDGET 32void xsk_set_rx_need_wakeup (struct xsk_buff_pool *pool) {
net/xdp/xsk.c static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, · line 782
Quoted 5 lines
net/xdp/xsk.c +1
-1@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,static int __xsk_generic_xmit (struct sock *sk) {struct xdp_sock *xs =xdp_sk (sk); -u32 max_batch =TX_BATCH_SIZE ; +u32 max_batch;
still rct is off - this is "reverse christmas tree" which refers to a non-written rule for networking that we organize the declarations of variables from longest line to shortest (so it looks like RCT...).
Maybe maintainers could fix this while applying? because rest of the changes looks sane.
Quoted 56 lines
net/xdp/xsk.c +17
-0bool sent_frame = false;struct xdp_desc desc;struct sk_buff *skb;@@ -796,6 +796,7 @@ static int __xsk_generic_xmit(struct sock *sk)if (xs->queue_id >= xs->dev->real_num_tx_queues) goto out; + max_batch =READ_ONCE (xs->max_tx_budget);while (xskq_cons_peek_desc (xs->tx, &desc, xs->pool)) {if (max_batch-- ==0 ) { err = -EAGAIN ;@@ -1437,6 +1438,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,mutex_unlock (&xs->mutex);return err; } +case XDP_MAX_TX_SKB_BUDGET : + { +unsigned int budget; + +if (optlen !=sizeof (budget)) +return -EINVAL ; +if (copy_from_sockptr (&budget, optval,sizeof (budget))) +return -EFAULT ; +if (!xs->tx || + budget <TX_BATCH_SIZE || budget > xs->tx->nentries) +return -EACCES ; + +WRITE_ONCE (xs->max_tx_budget, budget); +return 0 ; + }default :break ; }@@ -1734,6 +1750,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol, xs =xdp_sk (sk); xs->state =XSK_READY ; + xs->max_tx_budget =TX_BATCH_SIZE ;mutex_init (&xs->mutex);INIT_LIST_HEAD (&xs->map_list);tools/include/uapi/linux/if_xdp.h +1
-0@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {#define XDP_UMEM_COMPLETION_RING 6#define XDP_STATISTICS 7#define XDP_OPTIONS 8 +#define XDP_MAX_TX_SKB_BUDGET 9struct xdp_umem_reg {__u64 addr;/* Start of packet data area */ Signature · 1 line
2.41.3
Paolo Abeni
maintainerQuoted 96 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_SKB_BUDGETsocket option.
- Setmax_tx_budgetto 32 by default in the initialization phase as a per-socket granular control.
- Set the range ofmax_tx_budgetas [32,xs->tx->nentries].The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_SKB_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_SKB_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v71. use 'copy mode' in Doc
2. move init ofmax_tx_budgetto a proper position
3. use the max value in the if condition in setsockopt
4. change sockopt name toXDP_MAX_TX_SKB_BUDGET
5. setMAX_PER_SOCKET_BUDGETto 32 instead ofTX_BATCH_SIZEbecause they have no correlation at all.v6
1. use [32,
xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.v5
1. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 9 +++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 21 +++++++++++++++++++-- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 31 insertions(+), 2 deletions(-)Documentation/networking/af_xdp.rst +0
-0
Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 446
Quoted 5 lines
Documentation/networking/af_xdp.rst +2
-0@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW. +XDP_MAX_TX_SKB_BUDGET setsockopt +----------------------------
I'm sorry, kdoc is not happy about the above:
/home/doc-build/testing/Documentation/networking/af_xdp.rst:442:
WARNING: Title underline too short.XDP_MAX_TX_SKB_BUDGET setsockopt
----------------------------
/P
Quoted 101 lines · Paolo Abeni
Quoted 99 lines · Jason Xing
From: Jason Xing <[email protected]>
This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.
Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.
The patch does the following things:
- AddXDP_MAX_TX_SKB_BUDGETsocket option.
- Setmax_tx_budgetto 32 by default in the initialization phase as a per-socket granular control.
- Set the range ofmax_tx_budgetas [32,xs->tx->nentries].The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the
-EAGAINerror number passed fromsendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging theXDP_MAX_TX_SKB_BUDGETvalue contributes to less frequency ofsendto()and higher throughput/PPS.Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 asmax_tx_budgetbecause I saw two limitations without changing any default configuration:
1)XDP_MAX_TX_SKB_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As toXDP_MAX_TX_SKB_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time ofsendto()based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572[1]: https://lore.kernel.org/all/[email protected]/
Signed-off-by Jason Xing <[email protected]>---
v71. use 'copy mode' in Doc
2. move init ofmax_tx_budgetto a proper position
3. use the max value in the if condition in setsockopt
4. change sockopt name toXDP_MAX_TX_SKB_BUDGET
5. setMAX_PER_SOCKET_BUDGETto 32 instead ofTX_BATCH_SIZEbecause they have no correlation at all.v6
1. use [32,
xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.v5
1. remove changes around zc mode
v4
1. remove getsockopt as it seems no real use case.
2. adjust the position ofmax_tx_budgetto make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound ofmax_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.V3
1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit messageV2
1. use a per-netns sysctl knob
2. usesysctl_xsk_max_tx_budgetto unify both definitions.--- Documentation/networking/af_xdp.rst | 9 +++++++++ include/net/xdp_sock.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 21 +++++++++++++++++++-- tools/include/uapi/linux/if_xdp.h | 1 + 5 files changed, 31 insertions(+), 2 deletions(-)Documentation/networking/af_xdp.rst +0
-0@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one. Once the option is set, kernel will refuse attempts to bind that socket to a different interface. Updating the value requires CAP_NET_RAW.
Quoted 8 lines
Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 446
Quoted 2 lines
Documentation/networking/af_xdp.rst +2
-0+XDP_MAX_TX_SKB_BUDGET setsockopt +----------------------------I'm sorry, kdoc is not happy about the above:
/home/doc-build/testing/Documentation/networking/af_xdp.rst:442:
WARNING: Title underline too short.XDP_MAX_TX_SKB_BUDGET setsockopt
----------------------------
Ah, I see... Sorry, I should have checked this.
I will post a v8 patch with this one and RCT issues solved.
Thanks,
Jason
Quoted 3 lines
/P