net: add sysctl to toggle napi_consume_skb() alien skb defer
Subsystems (3)
PATCH v2 1/1 net: add sysctl to toggle napi_consume_skb() alien skb defer
Commit e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs") defers freeing of alien SKBs (alloc_cpu != current cpu) via skb_attempt_defer_free() on the TX completion path to reduce cross-NUMA SLUB spinlock contention to improve multi-queue UDP workloads.
However, this unconditionally impacts the napi_skb_cache fast recycle path for single-flow / few-flow workloads (e.g. AF_XDP benchmarks[1]):
when the TX completion NAPI CPU differs from the SKB allocation CPU, SKBs are deferred instead of being returned to the local napi_skb_cache, forcing RX allocations back to the slow slab path.
The existing net.core.skb_defer_max=0 could disable this, but it is a global switch that also disables the defer mechanism in TCP/UDP/MPTCP recvmsg paths, losing its positive SLUB locality benefits there. AF_XDP can co-exist with other protocols. That's the reason why I gave up reusing skb_defer_disable_key. Besides, if the defer path is disabled, that means TCP/UDP/MPTCP in process path will trigger directly freeing skb with enabling/disabling bottom half(in kfree_skb_napi_cache()) which could affect others. So my thinking is not to touch this path.
Add a dedicated sysctl net.core.napi_consume_skb_defer backed by a static key to selectively control the alien skb defer feature. Let users decide which is the best fit for their own requirements.
This patch also avoids touching local_bh* pair(in kfree_skb_napi_cache()) to minimize the overhead.
[1]: taskset -c 0 ./xdpsock -i enp2s0f1 -q 1 -t -S -s 64
1) sysctl -w net.core.napi_consume_skb_defer=1 (as default) sock0@enp2s0f1:1 txonly xdp-skb
pps pkts 1.00 rx 0 0 tx 1,851,950 20,397,952
2)sysctl -w net.core.napi_consume_skb_defer=0
sock0@enp2s0f1:1 txonly xdp-skb
pps pkts 1.00 rx 0 0 tx 1,985,067 25,530,432
For AF_XDP scenario, it turns out to be around 6.6% improvement.
---
V2
1. reuse proc_do_static_key() (Eric)
2. add doc (Stan)
--- Documentation/admin-guide/sysctl/net.rst | 13 +++++++++++++ net/core/net-sysfs.h | 1 + net/core/skbuff.c | 5 ++++- net/core/sysctl_net_core.c | 7 +++++++ 4 files changed, 25 insertions(+), 1 deletion(-)
Documentation/admin-guide/sysctl/net.rst +13 -0
@@ -368,6 +368,19 @@ by the cpu which allocated them. Default: 128 +napi_consume_skb_defer +---------------------- +When set to 1 (default), napi_consume_skb() defers freeing SKBs whose +allocation CPU differs from the current CPU via skb_attempt_defer_free(). +This reduces cross-NUMA SLUB spinlock contention for multi-queue workloads. + +Setting this to 0 disables the defer path in napi_consume_skb() only, +allowing SKBs to be returned to the local napi_skb_cache immediately. +This can benefit single-flow or few-flow workloads (e.g. AF_XDP TX) +where the defer detour hurts the fast recycle path. + +Default: 1 + optmem_max ----------
net/core/net-sysfs.h +1 -0
@@ -14,4 +14,5 @@ int netdev_change_owner(struct net_device *, const struct net *net_old,extern struct mutex rps_default_mask_mutex;DECLARE_STATIC_KEY_FALSE (skb_defer_disable_key); +DECLARE_STATIC_KEY_TRUE (napi_consume_skb_defer_key);#endif
net/core/skbuff.c +4 -1
@@ -94,6 +94,7 @@#include "dev.h" #include "devmem.h" +#include "net-sysfs.h" #include "netmem_priv.h" #include "sock_destructor.h"
@@ -1519,7 +1520,8 @@ void napi_consume_skb(struct sk_buff *skb, int budget)DEBUG_NET_WARN_ON_ONCE (!in_softirq ()); -if (skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) { +if (static_branch_likely (&napi_consume_skb_defer_key) && + skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) {skb_release_head_state (skb);return skb_attempt_defer_free (skb); }
@@ -7257,6 +7259,7 @@ static void kfree_skb_napi_cache(struct sk_buff *skb) }DEFINE_STATIC_KEY_FALSE (skb_defer_disable_key); +DEFINE_STATIC_KEY_TRUE (napi_consume_skb_defer_key);/** * skb_attempt_defer_free - queue skb for remote freeing
net/core/sysctl_net_core.c +7 -0
@@ -676,6 +676,13 @@ static struct ctl_table net_core_table[] = { .proc_handler = proc_do_skb_defer_max, .extra1 = SYSCTL_ZERO, }, + { + .procname = "napi_consume_skb_defer", + .data = &napi_consume_skb_defer_key.key, + .maxlen = sizeof(napi_consume_skb_defer_key), + .mode = 0644, + .proc_handler = proc_do_static_key, + }, };static struct ctl_table netns_core_table[] = {
Signature · 1 line
2.41.3
Paolo Abeni
maintainerQuoted 34 lines · Jason Xing
Commit
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") defers freeing of alien SKBs (alloc_cpu!= current cpu) viaskb_attempt_defer_free()on the TX completion path to reduce cross-NUMA SLUB spinlock contention to improve multi-queue UDP workloads.However, this unconditionally impacts the
napi_skb_cachefast recycle path for single-flow / few-flow workloads (e.g.AF_XDPbenchmarks[1]):
when the TX completion NAPI CPU differs from the SKB allocation CPU, SKBs are deferred instead of being returned to the localnapi_skb_cache, forcing RX allocations back to the slow slab path.The existing net.core.skb_defer_max=0 could disable this, but it is a global switch that also disables the defer mechanism in TCP/UDP/MPTCP recvmsg paths, losing its positive SLUB locality benefits there.
AF_XDPcan co-exist with other protocols. That's the reason why I gave up reusingskb_defer_disable_key. Besides, if the defer path is disabled, that means TCP/UDP/MPTCP in process path will trigger directly freeing skb with enabling/disabling bottomhalf(in kfree_skb_napi_cache())which could affect others. So my thinking is not to touch this path.Add a dedicated sysctl net.core.napi_consume_skb_defer backed by a static key to selectively control the alien skb defer feature. Let users decide which is the best fit for their own requirements.
This patch also avoids touching local_bh*
pair(in kfree_skb_napi_cache())to minimize the overhead.[1]: taskset -c 0 ./xdpsock -i enp2s0f1 -q 1 -t -S -s 64
1) sysctl -w net.core.napi_consume_skb_defer=1 (as default) sock0@enp2s0f1:1 txonly xdp-skbpps pkts 1.00 rx 0 0 tx 1,851,950 20,397,952
Quoted 7 lines
2)sysctl -w net.core.napi_consume_skb_defer=0
sock0@enp2s0f1:1 txonly xdp-skbpps pkts 1.00 rx 0 0 tx 1,985,067 25,530,432For
AF_XDPscenario, it turns out to be around 6.6% improvement.
I'm not a big fan of multiple tunables around the same feature, but possibly here the use-case extends beyond AF_XDP right? Do you observe some measurable positive delta even with UDP? Possibly even with TCP, when the bottleneck is the sender?
More data would be helpful.
Quoted 21 lines
Signed-off-by Jason Xing <[email protected]>---
V21. reuse
proc_do_static_key()(Eric)
2. add doc (Stan)--- Documentation/admin-guide/sysctl/net.rst | 13 +++++++++++++ net/core/net-sysfs.h | 1 + net/core/skbuff.c | 5 ++++- net/core/sysctl_net_core.c | 7 +++++++ 4 files changed, 25 insertions(+), 1 deletion(-)Documentation/admin-guide/sysctl/net.rst +0
-0@@ -368,6 +368,19 @@ by the cpu which allocated them. Default: 128
Documentation/admin-guide/sysctl/net.rst by the cpu which allocated them. · line 380
Quoted 10 lines
Documentation/admin-guide/sysctl/net.rst +10
-0+napi_consume_skb_defer +---------------------- +When set to 1 (default), napi_consume_skb() defers freeing SKBs whose +allocation CPU differs from the current CPU via skb_attempt_defer_free(). +This reduces cross-NUMA SLUB spinlock contention for multi-queue workloads. + +Setting this to 0 disables the defer path in napi_consume_skb() only, +allowing SKBs to be returned to the local napi_skb_cache immediately. +This can benefit single-flow or few-flow workloads (e.g. AF_XDP TX) +where the defer detour hurts the fast recycle path.
I think it should be clarified that skb_defer_max takes priority, i.e. no defer with skb_defer_max == 0. Also it would be great if you could additional extend skb_defer_max documentation noting that 0 disable such feature.
Thanks,
Paolo
Quoted 42 lines · Paolo Abeni
Quoted 40 lines · Jason Xing
Commit
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") defers freeing of alien SKBs (alloc_cpu!= current cpu) viaskb_attempt_defer_free()on the TX completion path to reduce cross-NUMA SLUB spinlock contention to improve multi-queue UDP workloads.However, this unconditionally impacts the
napi_skb_cachefast recycle path for single-flow / few-flow workloads (e.g.AF_XDPbenchmarks[1]):
when the TX completion NAPI CPU differs from the SKB allocation CPU, SKBs are deferred instead of being returned to the localnapi_skb_cache, forcing RX allocations back to the slow slab path.The existing net.core.skb_defer_max=0 could disable this, but it is a global switch that also disables the defer mechanism in TCP/UDP/MPTCP recvmsg paths, losing its positive SLUB locality benefits there.
AF_XDPcan co-exist with other protocols. That's the reason why I gave up reusingskb_defer_disable_key. Besides, if the defer path is disabled, that means TCP/UDP/MPTCP in process path will trigger directly freeing skb with enabling/disabling bottomhalf(in kfree_skb_napi_cache())which could affect others. So my thinking is not to touch this path.Add a dedicated sysctl net.core.napi_consume_skb_defer backed by a static key to selectively control the alien skb defer feature. Let users decide which is the best fit for their own requirements.
This patch also avoids touching local_bh*
pair(in kfree_skb_napi_cache())to minimize the overhead.[1]: taskset -c 0 ./xdpsock -i enp2s0f1 -q 1 -t -S -s 64
1) sysctl -w net.core.napi_consume_skb_defer=1 (as default) sock0@enp2s0f1:1 txonly xdp-skbpps pkts 1.00 rx 0 0 tx 1,851,950 20,397,9522)sysctl -w net.core.napi_consume_skb_defer=0
sock0@enp2s0f1:1 txonly xdp-skbpps pkts 1.00 rx 0 0 tx 1,985,067 25,530,432
Quoted 3 lines
Quoted 1 line
For
AF_XDPscenario, it turns out to be around 6.6% improvement.I'm not a big fan of multiple tunables around the same feature, but
Another interesting thing about this is for AI/auto tuning systems more sysctls can be leveraged and help users find the best combination of sysctls for their own scenario. I realized that especially a sysctl can decide the different/opposite fate of one particular case, which means we can give it a try.
Quoted 5 lines
possibly here the use-case extends beyond
AF_XDPright? Do you observe some measurable positive delta even with UDP? Possibly even with TCP, when the bottleneck is the sender?More data would be helpful.
Unfortunately, I didn't see any improvement with a simple udp flood test that I wrote. Maybe I need to do more experiments around it with more adjustments.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Jason Xing <[email protected]>---
V21. reuse
proc_do_static_key()(Eric)
2. add doc (Stan)--- Documentation/admin-guide/sysctl/net.rst | 13 +++++++++++++ net/core/net-sysfs.h | 1 + net/core/skbuff.c | 5 ++++- net/core/sysctl_net_core.c | 7 +++++++ 4 files changed, 25 insertions(+), 1 deletion(-)Documentation/admin-guide/sysctl/net.rst +0
-0@@ -368,6 +368,19 @@ by the cpu which allocated them. Default: 128
Quoted 15 lines
Documentation/admin-guide/sysctl/net.rst by the cpu which allocated them. · line 380
Quoted 10 lines
Documentation/admin-guide/sysctl/net.rst +10
-0+napi_consume_skb_defer +---------------------- +When set to 1 (default), napi_consume_skb() defers freeing SKBs whose +allocation CPU differs from the current CPU via skb_attempt_defer_free(). +This reduces cross-NUMA SLUB spinlock contention for multi-queue workloads. + +Setting this to 0 disables the defer path in napi_consume_skb() only, +allowing SKBs to be returned to the local napi_skb_cache immediately. +This can benefit single-flow or few-flow workloads (e.g. AF_XDP TX) +where the defer detour hurts the fast recycle path.I think it should be clarified that
skb_defer_maxtakes priority, i.e. no defer withskb_defer_max== 0. Also it would be great if you could additional extendskb_defer_maxdocumentation noting that 0 disable such feature.
The skb_defer_max and the newly added sysctl control different paths respectively. In napi_consume_skb, napi_consume_skb_defer takes higher priority.
These days, I'm wondering if you don't think it's worth a standalone knob, how about a simple patch to save more cycles when skb_defer_max is zero, like this:
net/core/skbuff.c +3 -1
@@ -94,6 +94,7 @@#include "dev.h" #include "devmem.h" +#include "net-sysfs.h" #include "netmem_priv.h" #include "sock_destructor.h"
@@ -1519,7 +1520,8 @@ void napi_consume_skb(struct sk_buff *skb, int budget)DEBUG_NET_WARN_ON_ONCE (!in_softirq ()); -if (skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) { +if (!static_branch_unlikely (&skb_defer_disable_key) && + skb->alloc_cpu !=smp_processor_id () && !skb_shared (skb)) {skb_release_head_state (skb);return skb_attempt_defer_free (skb); }
Signature · 4 lines
2.41.3 Thanks, Jason