Skip to contentSkip to conversation

net: xsk: introduce XDP_MAX_TX_BUDGET set/getsockopt

PATCH In review Jason Xing 3 participants 17 messages Jun 27, 2025 – Jul 3, 2025

Subsystems (5)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH v6 1/1 net: xsk: introduce XDP_MAX_TX_BUDGET set/getsockopt

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

---
v6

1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.

v5

1. remove changes around zc mode

v4

1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.

V3

1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message

V2

1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.

---
 Documentation/networking/af_xdp.rst |  9 +++++++
 include/net/xdp_sock.h              |  1 +
 include/uapi/linux/if_xdp.h         |  1 +
 net/xdp/xsk.c                       | 39 ++++++++++++++++++++++++++---
 tools/include/uapi/linux/if_xdp.h   |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +9 -0
index dceeb0d763aa..253afee00162 100644
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one.
 Once the option is set, kernel will refuse attempts to bind that socket
 to a different interface.  Updating the value requires CAP_NET_RAW.
 
+XDP_MAX_TX_BUDGET setsockopt
+----------------------------
+
+This setsockopt sets the maximum number of descriptors that can be handled
+and passed to the driver at one send syscall. It is applied in the non-zero
+copy mode to allow application to tune the per-socket maximum iteration for
+better throughput and less frequency of send syscall.
+Allowed range is [32, xs->tx->nentries].
+
 XDP_STATISTICS getsockopt
 -------------------------
 
include/net/xdp_sock.h +1 -0
index e8bd6ddb7b12..ce587a225661 100644
@@ -84,6 +84,7 @@ struct xdp_sock {
 	struct list_head map_list;
 	/* Protects map_list */
 	spinlock_t map_list_lock;
+	u32 max_tx_budget;
 	/* Protects multiple processes in the control path */
 	struct mutex mutex;
 	struct xsk_queue *fq_tmp; /* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
 #define XDP_OPTIONS			8
+#define XDP_MAX_TX_BUDGET		9
 
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
net/xdp/xsk.c +35 -4
index 72c000c0ae5f..41efe7b27b0e 100644
@@ -33,8 +33,8 @@
 #include "xdp_umem.h"
 #include "xsk.h"
 
-#define TX_BATCH_SIZE 32
-#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE)
+#define TX_BUDGET_SIZE 32
+#define MAX_PER_SOCKET_BUDGET 32
 
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 static int __xsk_generic_xmit(struct sock *sk)
 {
 	struct xdp_sock *xs = xdp_sk(sk);
-	u32 max_batch = TX_BATCH_SIZE;
+	u32 max_budget = READ_ONCE(xs->max_tx_budget);
 	bool sent_frame = false;
 	struct xdp_desc desc;
 	struct sk_buff *skb;
@@ -797,7 +797,7 @@ static int __xsk_generic_xmit(struct sock *sk)
 		goto out;
 
 	while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
-		if (max_batch-- == 0) {
+		if (max_budget-- == 0) {
 			err = -EAGAIN;
 			goto out;
 		}
@@ -1437,6 +1437,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
 		mutex_unlock(&xs->mutex);
 		return err;
 	}
+	case XDP_MAX_TX_BUDGET:
+	{
+		unsigned int budget;
+
+		if (optlen != sizeof(budget))
+			return -EINVAL;
+		if (copy_from_sockptr(&budget, optval, sizeof(budget)))
+			return -EFAULT;
+		if (!xs->tx || xs->tx->nentries < TX_BUDGET_SIZE)
+			return -EACCES;
+
+		WRITE_ONCE(xs->max_tx_budget,
+			   clamp(budget, TX_BUDGET_SIZE, xs->tx->nentries));
+		return 0;
+	}
 	default:
 		break;
 	}
@@ -1588,6 +1603,21 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
 		return 0;
 	}
+	case XDP_MAX_TX_BUDGET:
+	{
+		unsigned int budget;
+
+		if (len < sizeof(budget))
+			return -EINVAL;
+
+		budget = READ_ONCE(xs->max_tx_budget);
+		if (copy_to_user(optval, &budget, sizeof(budget)))
+			return -EFAULT;
+		if (put_user(sizeof(budget), optlen))
+			return -EFAULT;
+
+		return 0;
+	}
 	default:
 		break;
 	}
@@ -1734,6 +1764,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 
 	xs = xdp_sk(sk);
 	xs->state = XSK_READY;
+	xs->max_tx_budget = TX_BUDGET_SIZE;
 	mutex_init(&xs->mutex);
 
 	INIT_LIST_HEAD(&xs->map_list);
tools/include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
 #define XDP_OPTIONS			8
+#define XDP_MAX_TX_BUDGET		9
 
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
Signature · 1 line
2.41.3
author
#1 lore raw
Quoted 29 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Quoted 14 lines

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

If the next-respin is needed, I will attach those convincing numbers in the commit message.

Thanks,
Jason

author
#2 lore raw
Quoted 63 lines · Jason Xing
Quoted 29 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Quoted 14 lines

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736
Quoted 8 lines

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

Weird thing. I purchased another VM and didn't manage to see such a huge improvement.... Good luck is that I own that good machine which is still reproducible and I'm still digging in it. So please ignore this noise for now :|

Thanks,
Jason

#3 lore raw
Quoted 47 lines · Jason Xing
Quoted 46 lines · Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 4 lines
Quoted 4 lines

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

do you have a patch against xdpsock that does setsockopt you're introducing here?

-B -b 256 was for enabling busy polling and giving it 256 budget, which is not what you wanted to achieve.

Quoted 29 lines
Quoted 13 lines

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736
Quoted 8 lines

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

Weird thing. I purchased another VM and didn't manage to see such a huge improvement.... Good luck is that I own that good machine which is still reproducible and I'm still digging in it. So please ignore this noise for now :|

Thanks,
Jason

author
#4 lore raw
Quoted 49 lines · Maciej Fijalkowski
Quoted 47 lines · Jason Xing
Quoted 46 lines · Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

do you have a patch against xdpsock that does setsockopt you're introducing here?

Sure, I added the following code in the apply_setsockopt():
if (setsockopt(xsk_socket__fd(xsk->xsk), SOL_XDP, 9, &a, sizeof(a)) < 0)
...

Quoted 3 lines

-B -b 256 was for enabling busy polling and giving it 256 budget, which is not what you wanted to achieve.

I checked that I can use getsockopt to get the budget value the same as what I use setsockopt().

Sorry, I don't know what you meant here. Could you say more about it?

Thanks,
Jason

Quoted 30 lines
Quoted 30 lines
Quoted 14 lines

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736
Quoted 8 lines

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

Weird thing. I purchased another VM and didn't manage to see such a huge improvement.... Good luck is that I own that good machine which is still reproducible and I'm still digging in it. So please ignore this noise for now :|

Thanks,
Jason

#5 lore raw
Quoted 64 lines · Jason Xing
Quoted 49 lines · Maciej Fijalkowski
Quoted 47 lines · Jason Xing
Quoted 46 lines · Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

do you have a patch against xdpsock that does setsockopt you're introducing here?

Sure, I added the following code in the apply_setsockopt():
if (setsockopt(xsk_socket__fd(xsk->xsk), SOL_XDP, 9, &a, sizeof(a)) < 0)
...

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

-B -b 256 was for enabling busy polling and giving it 256 budget, which is not what you wanted to achieve.

I checked that I can use getsockopt to get the budget value the same as what I use setsockopt().

Sorry, I don't know what you meant here. Could you say more about it?

I meant that -b is for setting SO_BUSY_POLL_BUDGET. just pick different knob for your use case.

Quoted 34 lines

Thanks,
Jason

Quoted 30 lines
Quoted 30 lines
Quoted 14 lines

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736
Quoted 8 lines

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

Weird thing. I purchased another VM and didn't manage to see such a huge improvement.... Good luck is that I own that good machine which is still reproducible and I'm still digging in it. So please ignore this noise for now :|

Thanks,
Jason

author
#6 lore raw
Quoted 69 lines · Maciej Fijalkowski
Quoted 67 lines · Jason Xing
Quoted 49 lines · Maciej Fijalkowski
Quoted 47 lines · Jason Xing
Quoted 46 lines · Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

do you have a patch against xdpsock that does setsockopt you're introducing here?

Sure, I added the following code in the apply_setsockopt():
if (setsockopt(xsk_socket__fd(xsk->xsk), SOL_XDP, 9, &a, sizeof(a)) < 0)
...

Quoted 3 lines

-B -b 256 was for enabling busy polling and giving it 256 budget, which is not what you wanted to achieve.

Quoted 7 lines
Quoted 4 lines

I checked that I can use getsockopt to get the budget value the same as what I use setsockopt().

Sorry, I don't know what you meant here. Could you say more about it?

I meant that -b is for setting SO_BUSY_POLL_BUDGET. just pick different knob for your use case.

Sorry again. I still don't follow you. Do you mean this patch doesn't work as expected in busy polling mode? I tried to use '-B -b 256' and I managed to see the correct budget value returned from getsockopt().

How is this max tx budget related to the busy polling mode, I wonder?

Or are you talking about the performance test I mentioned? I probably know what happened there (due to some limit on the host side), but I still need more time.

Thanks,
Jason

Quoted 35 lines
Quoted 35 lines

Thanks,
Jason

Quoted 30 lines
Quoted 30 lines
Quoted 14 lines

Using the default configure 32 as the max budget iteration:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.01
rx                 0              0
tx                 48,574         49,152

Enlarging the value to 256:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 148,277        148,736
Quoted 8 lines

Enlarging the value to 512:
sock0@eth0:1 txonly xdp-drv

                   pps            pkts           1.00
rx                 0              0
tx                 226,306        227,072

The performance of pps goes up by 365% (with max budget set as 512) which is an incredible number :)

Weird thing. I purchased another VM and didn't manage to see such a huge improvement.... Good luck is that I own that good machine which is still reproducible and I'm still digging in it. So please ignore this noise for now :|

Thanks,
Jason

author
#7 lore raw
In reply to Maciej Fijalkowski
Quoted 69 lines · Maciej Fijalkowski
Quoted 67 lines · Jason Xing
Quoted 49 lines · Maciej Fijalkowski
Quoted 47 lines · Jason Xing
Quoted 46 lines · Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

More interesting numbers are arriving here as I run some benchmarks from xdp-project/bpf-examples/AF_XDP-example/ in my VM.

Running "sudo taskset -c 2 ./xdpsock -i eth0 -q 1 -l -N -t -b 256"

do you have a patch against xdpsock that does setsockopt you're introducing here?

Sure, I added the following code in the apply_setsockopt():
if (setsockopt(xsk_socket__fd(xsk->xsk), SOL_XDP, 9, &a, sizeof(a)) < 0)
...

Quoted 3 lines

-B -b 256 was for enabling busy polling and giving it 256 budget, which is not what you wanted to achieve.

Quoted 7 lines
Quoted 4 lines

I checked that I can use getsockopt to get the budget value the same as what I use setsockopt().

Sorry, I don't know what you meant here. Could you say more about it?

I meant that -b is for setting SO_BUSY_POLL_BUDGET. just pick different knob for your use case.

After taking a deep sleep, I clearly know what that is... I will try to test with other parameters. But for me, the primary factor is the security interception on the host side because of a larger number of descs containing useless information. Well, I will find a good way to avoid this in the future...

And what is your opinion of the current patch? I used [32, nentries] as the min/max range as you advised :)

Thanks,
Jason

author
#8 lore raw
In reply to Jason Xing
Quoted 44 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 3 lines

[1]: https://lore.kernel.org/all/[email protected]/

Hi Maciej, Stan, Willem, and other maintainers,

I wonder if you have further suggestions on the current patch?

Thanks in advance!

maintainer
#9 lore raw
In reply to Jason Xing
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 3 lines

[1]: https://lore.kernel.org/all/[email protected]/

LGTM, waiting a little more for an explicit an ack from XDP maintainers.

Side note: it could be useful to extend the xdp selftest to trigger the new code path.

/P

author
#10 lore raw
Quoted 45 lines · Paolo Abeni
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 5 lines
Quoted 3 lines

[1]: https://lore.kernel.org/all/[email protected]/

LGTM, waiting a little more for an explicit an ack from XDP maintainers.

Thanks. No problem.

Quoted 3 lines

Side note: it could be useful to extend the xdp selftest to trigger the new code path.

Roger that, sir. I will do it after this gets merged, maybe later this month, still studying for various tests in recent days :)

Thanks,
Jason

#13 lore raw
In reply to Jason Xing
Quoted 89 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

---
v6

1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.

v5

1. remove changes around zc mode

v4

1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.

V3

1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message

V2

1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.

---
 Documentation/networking/af_xdp.rst |  9 +++++++
 include/net/xdp_sock.h              |  1 +
 include/uapi/linux/if_xdp.h         |  1 +
 net/xdp/xsk.c                       | 39 ++++++++++++++++++++++++++---
 tools/include/uapi/linux/if_xdp.h   |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +0 -0
index dceeb0d763aa..253afee00162 100644

Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 449

Quoted 8 lines
Documentation/networking/af_xdp.rst +5 -0
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one.
 Once the option is set, kernel will refuse attempts to bind that socket
 to a different interface.  Updating the value requires CAP_NET_RAW.
 
+XDP_MAX_TX_BUDGET setsockopt
+----------------------------
+
+This setsockopt sets the maximum number of descriptors that can be handled
+and passed to the driver at one send syscall. It is applied in the non-zero

just 'copy mode' would be enough IMHO.

Quoted 36 lines
Documentation/networking/af_xdp.rst +4 -0
+copy mode to allow application to tune the per-socket maximum iteration for
+better throughput and less frequency of send syscall.
+Allowed range is [32, xs->tx->nentries].
+
 XDP_STATISTICS getsockopt
 -------------------------
 
include/net/xdp_sock.h +1 -0
index e8bd6ddb7b12..ce587a225661 100644
@@ -84,6 +84,7 @@ struct xdp_sock {
 	struct list_head map_list;
 	/* Protects map_list */
 	spinlock_t map_list_lock;
+	u32 max_tx_budget;
 	/* Protects multiple processes in the control path */
 	struct mutex mutex;
 	struct xsk_queue *fq_tmp; /* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
 #define XDP_OPTIONS			8
+#define XDP_MAX_TX_BUDGET		9
 
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
net/xdp/xsk.c +0 -0
index 72c000c0ae5f..41efe7b27b0e 100644

net/xdp/xsk.c line 37

Quoted 7 lines
net/xdp/xsk.c +2 -2
@@ -33,8 +33,8 @@
 #include "xdp_umem.h"
 #include "xsk.h"
 
-#define TX_BATCH_SIZE 32
-#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE)
+#define TX_BUDGET_SIZE 32
+#define MAX_PER_SOCKET_BUDGET 32

that could have stayed as it was before?

net/xdp/xsk.c static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, · line 782

Quoted 9 lines
net/xdp/xsk.c +1 -1
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 static int __xsk_generic_xmit(struct sock *sk)
 {
 	struct xdp_sock *xs = xdp_sk(sk);
-	u32 max_batch = TX_BATCH_SIZE;
+	u32 max_budget = READ_ONCE(xs->max_tx_budget);

you're breaking RCT here...maybe you could READ_ONCE() the budget right before the while() loop and keep the declaration only at the top of func?

also nothing wrong with keeping @max_batch as a name for this budget.

Quoted 19 lines
net/xdp/xsk.c +4 -1
 	bool sent_frame = false;
 	struct xdp_desc desc;
 	struct sk_buff *skb;
@@ -797,7 +797,7 @@ static int __xsk_generic_xmit(struct sock *sk)
 		goto out;
 
 	while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
-		if (max_batch-- == 0) {
+		if (max_budget-- == 0) {
 			err = -EAGAIN;
 			goto out;
 		}
@@ -1437,6 +1437,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
 		mutex_unlock(&xs->mutex);
 		return err;
 	}
+	case XDP_MAX_TX_BUDGET:
+	{
+		unsigned int budget;

net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1452

Quoted 10 lines
net/xdp/xsk.c +10 -0
+
+		if (optlen != sizeof(budget))
+			return -EINVAL;
+		if (copy_from_sockptr(&budget, optval, sizeof(budget)))
+			return -EFAULT;
+		if (!xs->tx || xs->tx->nentries < TX_BUDGET_SIZE)
+			return -EACCES;
+
+		WRITE_ONCE(xs->max_tx_budget,
+			   clamp(budget, TX_BUDGET_SIZE, xs->tx->nentries));

I think it would be better to throw errno when budget set by user is bigger than nentries. you do it for min case, let's do it for max case as well and skip the clamp() altogether?

this is rather speculative that someone would ever set such a big budget but silently setting a lower value than provided might be confusing to me. i'd rather get an error thrown at my face and find out the valid range.

Quoted 40 lines
net/xdp/xsk.c +18 -0
+		return 0;
+	}
 	default:
 		break;
 	}
@@ -1588,6 +1603,21 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
 		return 0;
 	}
+	case XDP_MAX_TX_BUDGET:
+	{
+		unsigned int budget;
+
+		if (len < sizeof(budget))
+			return -EINVAL;
+
+		budget = READ_ONCE(xs->max_tx_budget);
+		if (copy_to_user(optval, &budget, sizeof(budget)))
+			return -EFAULT;
+		if (put_user(sizeof(budget), optlen))
+			return -EFAULT;
+
+		return 0;
+	}
 	default:
 		break;
 	}
@@ -1734,6 +1764,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 
 	xs = xdp_sk(sk);
 	xs->state = XSK_READY;
+	xs->max_tx_budget = TX_BUDGET_SIZE;
 	mutex_init(&xs->mutex);
 
 	INIT_LIST_HEAD(&xs->map_list);
tools/include/uapi/linux/if_xdp.h +0 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644

tools/include/uapi/linux/if_xdp.h struct xdp_mmap_offsets { · line 82

Quoted 4 lines
tools/include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
 #define XDP_OPTIONS			8
+#define XDP_MAX_TX_BUDGET		9

could we have 'SKB' in name?

Quoted 6 lines
tools/include/uapi/linux/if_xdp.h +0 -0
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
Signature · 1 line
2.41.3
#11 lore raw
In reply to Jason Xing
Quoted 46 lines · Jason Xing
Quoted 45 lines · Paolo Abeni
Quoted 43 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

[1]: https://lore.kernel.org/all/[email protected]/

LGTM, waiting a little more for an explicit an ack from XDP maintainers.

Thanks. No problem.

Hey! i did review. Jason sorry but I got confused that you need to sort out the performance results on your side, hence the silence.

Quoted 7 lines
Quoted 4 lines

Side note: it could be useful to extend the xdp selftest to trigger the new code path.

Roger that, sir. I will do it after this gets merged, maybe later this month, still studying for various tests in recent days :)

IMHO nothing worth testing with this patch per-se, it's rather the matter of performance.

I would like however to ask you for follow-up with patch against xdpsock that adds support for using this new setsockopt (once we accept this onto kernel).

Quoted 3 lines

Thanks,
Jason

author
#14 lore raw
In reply to Maciej Fijalkowski

Hi Maciej,

Thanks for the review.

Quoted 94 lines · Maciej Fijalkowski
Quoted 92 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

---
v6

1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.

v5

1. remove changes around zc mode

v4

1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.

V3

1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message

V2

1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.

---
 Documentation/networking/af_xdp.rst |  9 +++++++
 include/net/xdp_sock.h              |  1 +
 include/uapi/linux/if_xdp.h         |  1 +
 net/xdp/xsk.c                       | 39 ++++++++++++++++++++++++++---
 tools/include/uapi/linux/if_xdp.h   |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +0 -0
index dceeb0d763aa..253afee00162 100644
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one.
 Once the option is set, kernel will refuse attempts to bind that socket
 to a different interface.  Updating the value requires CAP_NET_RAW.
 
Quoted 7 lines

Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 449

Quoted 5 lines
Documentation/networking/af_xdp.rst +5 -0
+XDP_MAX_TX_BUDGET setsockopt
+----------------------------
+
+This setsockopt sets the maximum number of descriptors that can be handled
+and passed to the driver at one send syscall. It is applied in the non-zero

just 'copy mode' would be enough IMHO.

Okay.

Quoted 40 lines
Quoted 40 lines
Documentation/networking/af_xdp.rst +4 -0
+copy mode to allow application to tune the per-socket maximum iteration for
+better throughput and less frequency of send syscall.
+Allowed range is [32, xs->tx->nentries].
+
 XDP_STATISTICS getsockopt
 -------------------------
 
include/net/xdp_sock.h +1 -0
index e8bd6ddb7b12..ce587a225661 100644
@@ -84,6 +84,7 @@ struct xdp_sock {
      struct list_head map_list;
      /* Protects map_list */
      spinlock_t map_list_lock;
+     u32 max_tx_budget;
      /* Protects multiple processes in the control path */
      struct mutex mutex;
      struct xsk_queue *fq_tmp; /* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9
 
 struct xdp_umem_reg {
      __u64 addr; /* Start of packet data area */
net/xdp/xsk.c +0 -0
index 72c000c0ae5f..41efe7b27b0e 100644
@@ -33,8 +33,8 @@
 #include "xdp_umem.h"
 #include "xsk.h"
 
Quoted 6 lines

net/xdp/xsk.c line 37

Quoted 4 lines
net/xdp/xsk.c +2 -2
-#define TX_BATCH_SIZE 32
-#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE)
+#define TX_BUDGET_SIZE 32
+#define MAX_PER_SOCKET_BUDGET 32

that could have stayed as it was before?

Sorry, I don't think so because one definition has nothing to do with the other one. Why do we need to bind them together?

Quoted 12 lines

net/xdp/xsk.c static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, · line 782

Quoted 10 lines
net/xdp/xsk.c +1 -1
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 static int __xsk_generic_xmit(struct sock *sk)
 {
      struct xdp_sock *xs = xdp_sk(sk);
-     u32 max_batch = TX_BATCH_SIZE;
+     u32 max_budget = READ_ONCE(xs->max_tx_budget);

you're breaking RCT here...

What's the RCT? Never heard of this abbreviation.

Quoted 30 lines

maybe you could READ_ONCE() the budget right
before the while() loop and keep the declaration only at the top of func?

also nothing wrong with keeping @max_batch as a name for this budget.

Quoted 25 lines
net/xdp/xsk.c +10 -1
      bool sent_frame = false;
      struct xdp_desc desc;
      struct sk_buff *skb;
@@ -797,7 +797,7 @@ static int __xsk_generic_xmit(struct sock *sk)
              goto out;
 
      while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
-             if (max_batch-- == 0) {
+             if (max_budget-- == 0) {
                      err = -EAGAIN;
                      goto out;
              }
@@ -1437,6 +1437,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
              mutex_unlock(&xs->mutex);
              return err;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (optlen != sizeof(budget))
+                     return -EINVAL;
+             if (copy_from_sockptr(&budget, optval, sizeof(budget)))
+                     return -EFAULT;
+             if (!xs->tx || xs->tx->nentries < TX_BUDGET_SIZE)
Quoted 8 lines

net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1452

Quoted 4 lines
net/xdp/xsk.c +4 -0
+                     return -EACCES;
+
+             WRITE_ONCE(xs->max_tx_budget,
+                        clamp(budget, TX_BUDGET_SIZE, xs->tx->nentries));

I think it would be better to throw errno when budget set by user is bigger than nentries. you do it for min case, let's do it for max case as well and skip the clamp() altogether?

Okay.

After this, do you think it's necessary to keep the getsockopt()?

Quoted 4 lines

this is rather speculative that someone would ever set such a big budget but silently setting a lower value than provided might be confusing to me. i'd rather get an error thrown at my face and find out the valid range.

On point.

Quoted 35 lines
Quoted 35 lines
net/xdp/xsk.c +18 -0
+             return 0;
+     }
      default:
              break;
      }
@@ -1588,6 +1603,21 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
              return 0;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (len < sizeof(budget))
+                     return -EINVAL;
+
+             budget = READ_ONCE(xs->max_tx_budget);
+             if (copy_to_user(optval, &budget, sizeof(budget)))
+                     return -EFAULT;
+             if (put_user(sizeof(budget), optlen))
+                     return -EFAULT;
+
+             return 0;
+     }
      default:
              break;
      }
@@ -1734,6 +1764,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 
      xs = xdp_sk(sk);
      xs->state = XSK_READY;
+     xs->max_tx_budget = TX_BUDGET_SIZE;
      mutex_init(&xs->mutex);
 
Quoted 12 lines

tools/include/uapi/linux/if_xdp.h struct xdp_mmap_offsets { · line 82

Quoted 10 lines
net/xdp/xsk.c +0 -0
      INIT_LIST_HEAD(&xs->map_list);
tools/include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9

could we have 'SKB' in name?

Like XDP_MAX_TX_BUDGET_SKB ?

Thanks,
Jason

Quoted 7 lines
Quoted 7 lines
tools/include/uapi/linux/if_xdp.h +0 -1
 struct xdp_umem_reg {
      __u64 addr; /* Start of packet data area */
--

2.41.3

author
#12 lore raw
In reply to Maciej Fijalkowski
Quoted 50 lines · Maciej Fijalkowski
Quoted 48 lines · Jason Xing
Quoted 47 lines · Paolo Abeni
Quoted 45 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

LGTM, waiting a little more for an explicit an ack from XDP maintainers.

Thanks. No problem.

Hey! i did review. Jason sorry but I got confused that you need to sort out the performance results on your side, hence the silence.

Thanks for the review. My environment doesn't allow me to continue the xdpsock experiment because of many limitations from the host side.

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Side note: it could be useful to extend the xdp selftest to trigger the new code path.

Roger that, sir. I will do it after this gets merged, maybe later this month, still studying for various tests in recent days :)

Quoted 6 lines

IMHO nothing worth testing with this patch per-se, it's rather the matter of performance.

I would like however to ask you for follow-up with patch against xdpsock that adds support for using this new setsockopt (once we accept this onto kernel).

That xdp-project in the github. I will finish it after it's done.

Thanks,
Jason

Quoted 4 lines
Quoted 4 lines

Thanks,
Jason

author
#15 lore raw
In reply to Jason Xing
Quoted 274 lines · Jason Xing

Hi Maciej,

Thanks for the review.

Quoted 94 lines · Maciej Fijalkowski
Quoted 92 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

---
v6

1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.

v5

1. remove changes around zc mode

v4

1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.

V3

1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message

V2

1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.

---
 Documentation/networking/af_xdp.rst |  9 +++++++
 include/net/xdp_sock.h              |  1 +
 include/uapi/linux/if_xdp.h         |  1 +
 net/xdp/xsk.c                       | 39 ++++++++++++++++++++++++++---
 tools/include/uapi/linux/if_xdp.h   |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +0 -0
index dceeb0d763aa..253afee00162 100644
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one.
 Once the option is set, kernel will refuse attempts to bind that socket
 to a different interface.  Updating the value requires CAP_NET_RAW.
 
Quoted 7 lines

Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 449

Quoted 5 lines
Documentation/networking/af_xdp.rst +5 -0
+XDP_MAX_TX_BUDGET setsockopt
+----------------------------
+
+This setsockopt sets the maximum number of descriptors that can be handled
+and passed to the driver at one send syscall. It is applied in the non-zero

just 'copy mode' would be enough IMHO.

Okay.

Quoted 40 lines
Quoted 40 lines
Documentation/networking/af_xdp.rst +4 -0
+copy mode to allow application to tune the per-socket maximum iteration for
+better throughput and less frequency of send syscall.
+Allowed range is [32, xs->tx->nentries].
+
 XDP_STATISTICS getsockopt
 -------------------------
 
include/net/xdp_sock.h +1 -0
index e8bd6ddb7b12..ce587a225661 100644
@@ -84,6 +84,7 @@ struct xdp_sock {
      struct list_head map_list;
      /* Protects map_list */
      spinlock_t map_list_lock;
+     u32 max_tx_budget;
      /* Protects multiple processes in the control path */
      struct mutex mutex;
      struct xsk_queue *fq_tmp; /* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9
 
 struct xdp_umem_reg {
      __u64 addr; /* Start of packet data area */
net/xdp/xsk.c +0 -0
index 72c000c0ae5f..41efe7b27b0e 100644
@@ -33,8 +33,8 @@
 #include "xdp_umem.h"
 #include "xsk.h"
 
Quoted 6 lines

net/xdp/xsk.c line 37

Quoted 4 lines
net/xdp/xsk.c +2 -2
-#define TX_BATCH_SIZE 32
-#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE)
+#define TX_BUDGET_SIZE 32
+#define MAX_PER_SOCKET_BUDGET 32

that could have stayed as it was before?

Sorry, I don't think so because one definition has nothing to do with the other one. Why do we need to bind them together?

Quoted 12 lines

net/xdp/xsk.c static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, · line 782

Quoted 10 lines
net/xdp/xsk.c +1 -1
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 static int __xsk_generic_xmit(struct sock *sk)
 {
      struct xdp_sock *xs = xdp_sk(sk);
-     u32 max_batch = TX_BATCH_SIZE;
+     u32 max_budget = READ_ONCE(xs->max_tx_budget);

you're breaking RCT here...

What's the RCT? Never heard of this abbreviation.

Quoted 30 lines

maybe you could READ_ONCE() the budget right
before the while() loop and keep the declaration only at the top of func?

also nothing wrong with keeping @max_batch as a name for this budget.

Quoted 25 lines
net/xdp/xsk.c +10 -1
      bool sent_frame = false;
      struct xdp_desc desc;
      struct sk_buff *skb;
@@ -797,7 +797,7 @@ static int __xsk_generic_xmit(struct sock *sk)
              goto out;
 
      while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
-             if (max_batch-- == 0) {
+             if (max_budget-- == 0) {
                      err = -EAGAIN;
                      goto out;
              }
@@ -1437,6 +1437,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
              mutex_unlock(&xs->mutex);
              return err;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (optlen != sizeof(budget))
+                     return -EINVAL;
+             if (copy_from_sockptr(&budget, optval, sizeof(budget)))
+                     return -EFAULT;
+             if (!xs->tx || xs->tx->nentries < TX_BUDGET_SIZE)
Quoted 8 lines

net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1452

Quoted 4 lines
net/xdp/xsk.c +4 -0
+                     return -EACCES;
+
+             WRITE_ONCE(xs->max_tx_budget,
+                        clamp(budget, TX_BUDGET_SIZE, xs->tx->nentries));

I think it would be better to throw errno when budget set by user is bigger than nentries. you do it for min case, let's do it for max case as well and skip the clamp() altogether?

Okay.

After this, do you think it's necessary to keep the getsockopt()?

Quoted 4 lines

this is rather speculative that someone would ever set such a big budget but silently setting a lower value than provided might be confusing to me. i'd rather get an error thrown at my face and find out the valid range.

On point.

Quoted 35 lines
Quoted 35 lines
net/xdp/xsk.c +18 -0
+             return 0;
+     }
      default:
              break;
      }
@@ -1588,6 +1603,21 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
              return 0;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (len < sizeof(budget))
+                     return -EINVAL;
+
+             budget = READ_ONCE(xs->max_tx_budget);
+             if (copy_to_user(optval, &budget, sizeof(budget)))
+                     return -EFAULT;
+             if (put_user(sizeof(budget), optlen))
+                     return -EFAULT;
+
+             return 0;
+     }
      default:
              break;
      }
@@ -1734,6 +1764,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 
      xs = xdp_sk(sk);
      xs->state = XSK_READY;
+     xs->max_tx_budget = TX_BUDGET_SIZE;
      mutex_init(&xs->mutex);
 
Quoted 12 lines

tools/include/uapi/linux/if_xdp.h struct xdp_mmap_offsets { · line 82

Quoted 10 lines
net/xdp/xsk.c +0 -0
      INIT_LIST_HEAD(&xs->map_list);
tools/include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9

could we have 'SKB' in name?

Quoted 1 line

Like XDP_MAX_TX_BUDGET_SKB ?

XDP_MAX_TX_SKB_BUDGET sounds better, I assume.

Thanks,
Jason

author
#16 lore raw
Quoted 276 lines · Jason Xing
Quoted 274 lines · Jason Xing

Hi Maciej,

Thanks for the review.

Quoted 94 lines · Maciej Fijalkowski
Quoted 92 lines · Jason Xing

From: Jason Xing <[email protected]>

This patch provides a setsockopt method to let applications leverage to adjust how many descs to be handled at most in one send syscall. It mitigates the situation where the default value (32) that is too small leads to higher frequency of triggering send syscall.

Considering the prosperity/complexity the applications have, there is no absolutely ideal suggestion fitting all cases. So keep 32 as its default value like before.

The patch does the following things:
- Add XDP_MAX_TX_BUDGET socket option.
- Convert TX_BATCH_SIZE to tx_budget_spent.
- Set tx_budget_spent to 32 by default in the initialization phase as a per-socket granular control. 32 is also the min value for tx_budget_spent.
- Set the range of tx_budget_spent as [32, xs->tx->nentries].

The idea behind this comes out of real workloads in production. We use a user-level stack with xsk support to accelerate sending packets and minimize triggering syscalls. When the packets are aggregated, it's not hard to hit the upper bound (namely, 32). The moment user-space stack fetches the -EAGAIN error number passed from sendto(), it will loop to try again until all the expected descs from tx ring are sent out to the driver. Enlarging the XDP_MAX_TX_BUDGET value contributes to less frequency of sendto() and higher throughput/PPS.

Here is what I did in production, along with some numbers as follows:
For one application I saw lately, I suggested using 128 as max_tx_budget because I saw two limitations without changing any default configuration:
1) XDP_MAX_TX_BUDGET, 2) socket sndbuf which is 212992 decided by net.core.wmem_default. As to XDP_MAX_TX_BUDGET, the scenario behind this was I counted how many descs are transmitted to the driver at one time of sendto() based on [1] patch and then I calculated the possibility of hitting the upper bound. Finally I chose 128 as a suitable value because 1) it covers most of the cases, 2) a higher number would not bring evident results. After twisting the parameters, a stable improvement of around 4% for both PPS and throughput and less resources consumption were found to be observed by strace -c -p xxx:
1) %time was decreased by 7.8%
2) error counter was decreased from 18367 to 572

[1]: https://lore.kernel.org/all/[email protected]/

---
v6

1. use [32, xs->tx->nentries] range
2. Since setsockopt may generate a different value, add getsockopt to help application know what value takes effect finally.

v5

1. remove changes around zc mode

v4

1. remove getsockopt as it seems no real use case.
2. adjust the position of max_tx_budget to make sure it stays with other read-most fields in one cacheline.
3. set one as the lower bound of max_tx_budget
4. add more descriptions/performance data in Doucmentation and commit message.

V3

1. use a per-socket control (suggested by Stanislav)
2. unify both definitions into one
3. support setsockopt and getsockopt
4. add more description in commit message

V2

1. use a per-netns sysctl knob
2. use sysctl_xsk_max_tx_budget to unify both definitions.

---
 Documentation/networking/af_xdp.rst |  9 +++++++
 include/net/xdp_sock.h              |  1 +
 include/uapi/linux/if_xdp.h         |  1 +
 net/xdp/xsk.c                       | 39 ++++++++++++++++++++++++++---
 tools/include/uapi/linux/if_xdp.h   |  1 +
 5 files changed, 47 insertions(+), 4 deletions(-)
Documentation/networking/af_xdp.rst +0 -0
index dceeb0d763aa..253afee00162 100644
@@ -442,6 +442,15 @@ is created by a privileged process and passed to a non-privileged one.
 Once the option is set, kernel will refuse attempts to bind that socket
 to a different interface.  Updating the value requires CAP_NET_RAW.
 
Quoted 7 lines

Documentation/networking/af_xdp.rst is created by a privileged process and passed to a non-privileged one. · line 449

Quoted 5 lines
Documentation/networking/af_xdp.rst +5 -0
+XDP_MAX_TX_BUDGET setsockopt
+----------------------------
+
+This setsockopt sets the maximum number of descriptors that can be handled
+and passed to the driver at one send syscall. It is applied in the non-zero

just 'copy mode' would be enough IMHO.

Okay.

Quoted 40 lines
Quoted 40 lines
Documentation/networking/af_xdp.rst +4 -0
+copy mode to allow application to tune the per-socket maximum iteration for
+better throughput and less frequency of send syscall.
+Allowed range is [32, xs->tx->nentries].
+
 XDP_STATISTICS getsockopt
 -------------------------
 
include/net/xdp_sock.h +1 -0
index e8bd6ddb7b12..ce587a225661 100644
@@ -84,6 +84,7 @@ struct xdp_sock {
      struct list_head map_list;
      /* Protects map_list */
      spinlock_t map_list_lock;
+     u32 max_tx_budget;
      /* Protects multiple processes in the control path */
      struct mutex mutex;
      struct xsk_queue *fq_tmp; /* Only as tmp storage before bind */
include/uapi/linux/if_xdp.h +1 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9
 
 struct xdp_umem_reg {
      __u64 addr; /* Start of packet data area */
net/xdp/xsk.c +0 -0
index 72c000c0ae5f..41efe7b27b0e 100644
@@ -33,8 +33,8 @@
 #include "xdp_umem.h"
 #include "xsk.h"
 
Quoted 6 lines

net/xdp/xsk.c line 37

Quoted 4 lines
net/xdp/xsk.c +2 -2
-#define TX_BATCH_SIZE 32
-#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE)
+#define TX_BUDGET_SIZE 32
+#define MAX_PER_SOCKET_BUDGET 32

that could have stayed as it was before?

Sorry, I don't think so because one definition has nothing to do with the other one. Why do we need to bind them together?

Quoted 12 lines

net/xdp/xsk.c static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, · line 782

Quoted 10 lines
net/xdp/xsk.c +1 -1
 void xsk_set_rx_need_wakeup(struct xsk_buff_pool *pool)
 {
@@ -779,7 +779,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
 static int __xsk_generic_xmit(struct sock *sk)
 {
      struct xdp_sock *xs = xdp_sk(sk);
-     u32 max_batch = TX_BATCH_SIZE;
+     u32 max_budget = READ_ONCE(xs->max_tx_budget);

you're breaking RCT here...

What's the RCT? Never heard of this abbreviation.

Quoted 30 lines

maybe you could READ_ONCE() the budget right
before the while() loop and keep the declaration only at the top of func?

also nothing wrong with keeping @max_batch as a name for this budget.

Quoted 25 lines
net/xdp/xsk.c +10 -1
      bool sent_frame = false;
      struct xdp_desc desc;
      struct sk_buff *skb;
@@ -797,7 +797,7 @@ static int __xsk_generic_xmit(struct sock *sk)
              goto out;
 
      while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) {
-             if (max_batch-- == 0) {
+             if (max_budget-- == 0) {
                      err = -EAGAIN;
                      goto out;
              }
@@ -1437,6 +1437,21 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
              mutex_unlock(&xs->mutex);
              return err;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (optlen != sizeof(budget))
+                     return -EINVAL;
+             if (copy_from_sockptr(&budget, optval, sizeof(budget)))
+                     return -EFAULT;
+             if (!xs->tx || xs->tx->nentries < TX_BUDGET_SIZE)
Quoted 8 lines

net/xdp/xsk.c static int xsk_setsockopt(struct socket *sock, int level, int optname, · line 1452

Quoted 4 lines
net/xdp/xsk.c +4 -0
+                     return -EACCES;
+
+             WRITE_ONCE(xs->max_tx_budget,
+                        clamp(budget, TX_BUDGET_SIZE, xs->tx->nentries));

I think it would be better to throw errno when budget set by user is bigger than nentries. you do it for min case, let's do it for max case as well and skip the clamp() altogether?

Okay.

After this, do you think it's necessary to keep the getsockopt()?

Quoted 4 lines

this is rather speculative that someone would ever set such a big budget but silently setting a lower value than provided might be confusing to me. i'd rather get an error thrown at my face and find out the valid range.

On point.

Quoted 47 lines
Quoted 41 lines
net/xdp/xsk.c +18 -0
+             return 0;
+     }
      default:
              break;
      }
@@ -1588,6 +1603,21 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
              return 0;
      }
+     case XDP_MAX_TX_BUDGET:
+     {
+             unsigned int budget;
+
+             if (len < sizeof(budget))
+                     return -EINVAL;
+
+             budget = READ_ONCE(xs->max_tx_budget);
+             if (copy_to_user(optval, &budget, sizeof(budget)))
+                     return -EFAULT;
+             if (put_user(sizeof(budget), optlen))
+                     return -EFAULT;
+
+             return 0;
+     }
      default:
              break;
      }
@@ -1734,6 +1764,7 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 
      xs = xdp_sk(sk);
      xs->state = XSK_READY;
+     xs->max_tx_budget = TX_BUDGET_SIZE;
      mutex_init(&xs->mutex);
 
      INIT_LIST_HEAD(&xs->map_list);
tools/include/uapi/linux/if_xdp.h +0 -0
index 44f2bb93e7e6..07c6d21c2f1c 100644

tools/include/uapi/linux/if_xdp.h struct xdp_mmap_offsets { · line 82

Quoted 4 lines
tools/include/uapi/linux/if_xdp.h +1 -0
@@ -79,6 +79,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_COMPLETION_RING     6
 #define XDP_STATISTICS                       7
 #define XDP_OPTIONS                  8
+#define XDP_MAX_TX_BUDGET            9

could we have 'SKB' in name?

Quoted 3 lines
Quoted 1 line

Like XDP_MAX_TX_BUDGET_SKB ?

XDP_MAX_TX_SKB_BUDGET sounds better, I assume.

Updated v7 at https://lore.kernel.org/all/[email protected]/
.

Cross finger :S

Thanks,
Jason