net: use sync wakeups for socket error reports
Subsystems (8)
PATCH 1/1 net: use sync wakeups for socket error reports
sock_def_readable() and sock_def_write_space() pass WF_SYNC to their waitqueue wakeups, switch sock_def_error_report() to wake_up_interruptible_sync_poll() so EPOLLERR waiters get the same hint. ep_poll_callback() forwards it through totry_to_wake_up() / select_task_rq_fair(), where wake_affine() can prefer the waker CPU and skip a cross-CPU wakelist IPI when its heuristics agree.
WF_SYNC matches the choice already made for readable/write_space. Some error reports have the same producer/consumer shape: the waker has queued an skb on sk->sk_error_queue, and the wakee is about to dequeue and copy it out. Other reports publish sk_err/socket state directly, but still wake a task likely to consume that socket state immediately.
Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the sock_def_error_report -> ep_poll_callback -> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue chain to 16,326 IPIs/min. Switching to wake_up_interruptible_sync_poll() will help reduce those IPIs.
--- net/core/sock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/sock.c +1 -1
@@ -3639,7 +3639,7 @@ static void sock_def_error_report(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq);if (skwq_has_sleeper (wq)) -wake_up_interruptible_poll (&wq->wait,EPOLLERR ); +wake_up_interruptible_sync_poll (&wq->wait,EPOLLERR );sk_wake_async_rcu (sk,SOCK_WAKE_IO ,POLL_ERR );rcu_read_unlock (); }
Signature · 1 line
2.53.0-Meta
Quoted 5 lines · Usama Arif
Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the
sock_def_error_report->ep_poll_callback->try_to_wake_up->ttwu_queue_wakelist->__smp_call_single_queuechain to 16,326 IPIs/min.
I am interested in why so many sock_def_error_report().
That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?
Can you share the full stack above sock_def_error_report()?
Quoted 7 lines · Breno Leitao
Quoted 5 lines · Usama Arif
Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the
sock_def_error_report->ep_poll_callback->try_to_wake_up->ttwu_queue_wakelist->__smp_call_single_queuechain to 16,326 IPIs/min.
Quoted 6 lines
I am interested in why so many
sock_def_error_report().That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?
Can you share the full stack above
sock_def_error_report()?
I ran this bpftrace script the host now (results added at the end):
sudo bpftrace -e '
kprobe:sock_def_error_report
{
@wake_src[kstack()] = count();
}
interval:s:60
{
print(@wake_src, 5);
exit();}'
The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE) to get the SND/ACK timestamp.
So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is why sock_def_error_report fires.
Attached 2 probes
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1630
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 1812
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_dequeue_err_skb+194
ipv6_recv_error+74
bpf_trampoline_6442598004+73
____sys_recvmsg.llvm.18251018526254450710+168
___sys_recvmsg+312
__x64_sys_recvmsg+95
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+125
asm_common_interrupt+34
cpuidle_enter_state+202
cpuidle_enter+40
cpu_startup_entry+497
ap_starting+0
common_startup_64+318
]: 12603
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
tcp_ack+3399
tcp_rcv_established+1258
tcp_v6_do_rcv+372
tcp_v6_rcv+4748
ip6_protocol_deliver_rcu+653
ip6_input_finish+79
ip6_input+43
ipv6_list_rcv+4339
__netif_receive_skb_list_core+244
netif_receive_skb_list_internal+433
napi_complete_done+149
bnxt_poll_p5+499
net_rx_action+513
irq_exit_rcu+312
common_interrupt+62
asm_common_interrupt+34
]: 19314
@wake_src[
sock_def_error_report+1
sk_error_report+17
sock_queue_err_skb+285
__skb_tstamp_tx+903
bnxt_start_xmit+1769
dev_hard_start_xmit+160
sch_direct_xmit+165
__qdisc_run+714
__dev_queue_xmit+2052
skb_do_redirect+2531
netkit_xmit+715
dev_hard_start_xmit+160
__dev_queue_xmit+1049
ip6_finish_output2+848
ip6_finish_output+213
ip6_output+86
ip6_xmit+933
inet6_csk_xmit+163
__tcp_transmit_skb+2733
tcp_write_xmit+2948
__tcp_push_pending_frames+46
tcp_sendmsg_locked+4187
tcp_sendmsg+40
__x64_sys_sendmsg+567
do_syscall_64+316
entry_SYSCALL_64_after_hwframe+75]: 33514
Quoted 40 lines · Usama Arif
Quoted 7 lines · Breno Leitao
Quoted 5 lines · Usama Arif
Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the
sock_def_error_report->ep_poll_callback->try_to_wake_up->ttwu_queue_wakelist->__smp_call_single_queuechain to 16,326 IPIs/min.Quoted 6 lines
I am interested in why so many
sock_def_error_report().That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?
Can you share the full stack above
sock_def_error_report()?I ran this bpftrace script the host now (results added at the end):
sudo bpftrace -e '
kprobe:sock_def_error_report{ @wake_src[kstack()] = count(); } interval:s:60 { print(@wake_src, 5); exit();}'
The biggest source is
tcp_sendmsg->__skb_tstamp_tx, which as you said is not an actual error.__skb_tstamp_txclones the outgoing skb, tags it withee_origin=SO_EE_ORIGIN_TIMESTAMPINGandee_errno= ENOMSG, enqueues it onsk->sk_error_queueviasock_queue_err_skb, and callssk_error_reportso epoll raises EPOLLERR. Userspace then reads it withrecvmsg(MSG_ERRQUEUE)to get the SND/ACK timestamp.
Quoted 4 lines
So the workload has
SO_TIMESTAMPINGenabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is whysock_def_error_reportfires.
It seems we can not please everyone.
https://lore.kernel.org/netdev/[email protected]/
Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.
Quoted 2 lines · Eric Dumazet
Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.
Something like this?
net: add SO_ERR_WAKE_SYNC for sync error-report wakeups
sock_def_error_report() wakes EPOLLERR waiters with wake_up_interruptible_poll(), while sock_def_readable() and sock_def_write_space() already pass the sync hint. A socket with SO_TIMESTAMPING enabled delivers every TX and ACK timestamp through sk_error_queue and raises EPOLLERR, so the error path wakes a sleeping consumer very often.
Without the sync hint the scheduler often places the woken consumer on a remote CPU, which costs a rescheduling IPI. Usama Arif measured 16,326 such IPIs/min on a 176-core host running a production workload with SO_TIMESTAMPING enabled. [1]
Switching the error path to a sync wakeup unconditionally is not the right fix, as there are different requirements for it to be async, see [2].
Eric suggested that an options is to add SO_ERR_WAKE_SYNC so applications choose per socket, so a consumer draining a high-rate error queue can opt in to keep the wakeup local and drop the IPI, using socket flag SO_ERR_WAKE_SYNC.
arch/alpha/include/uapi/asm/socket.h +2 -0
@@ -155,6 +155,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64
arch/mips/include/uapi/asm/socket.h +2 -0
@@ -166,6 +166,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64
arch/parisc/include/uapi/asm/socket.h +2 -0
@@ -147,6 +147,8 @@#define SO_INQ 0x4052#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 0x4053 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64
arch/sparc/include/uapi/asm/socket.h +2 -0
@@ -148,6 +148,8 @@#define SO_INQ 0x005d#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 0x005e +#if !defined(__KERNEL__)
include/net/sock.h +1 -0
@@ -1022,6 +1022,7 @@ enum sock_flags {SOCK_RCVMARK ,/* Receive SO_MARK ancillary data with packet */ SOCK_RCVPRIORITY ,/* Receive SO_PRIORITY ancillary data with packet */ SOCK_TIMESTAMPING_ANY ,/* Copy of sk_tsflags & TSFLAGS_ANY */ + SOCK_ERR_WAKE_SYNC, /* Sync wakeup on error report, %SO_ERR_WAKE_SYNC */ };#define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))
include/uapi/asm-generic/socket.h +2 -0
@@ -150,6 +150,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 || (defined(__x86_64__) && defined(__ILP32__))
net/core/sock.c +14 -2
@@ -1557,6 +1557,10 @@ int sk_setsockopt(struct sock *sk, int level, int optname,sock_valbool_flag (sk,SOCK_SELECT_ERR_QUEUE , valbool);break ; +case SO_ERR_WAKE_SYNC : +sock_valbool_flag (sk,SOCK_ERR_WAKE_SYNC , valbool); +break ; +case SO_PASSCRED :if (sk_may_scm_recv (sk)) sk->sk_scm_credentials = valbool;
@@ -2064,6 +2068,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname, v.val =sock_flag (sk,SOCK_SELECT_ERR_QUEUE );break ; +case SO_ERR_WAKE_SYNC : + v.val =sock_flag (sk,SOCK_ERR_WAKE_SYNC ); +break ; +#ifdef CONFIG_NET_RX_BUSY_POLL case SO_BUSY_POLL : v.val =READ_ONCE (sk->sk_ll_usec);
@@ -3641,8 +3649,12 @@ static void sock_def_error_report(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_poll (&wq->wait,EPOLLERR ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_ERR_WAKE_SYNC )) +wake_up_interruptible_sync_poll (&wq->wait,EPOLLERR ); +else +wake_up_interruptible_poll (&wq->wait,EPOLLERR ); + }sk_wake_async_rcu (sk,SOCK_WAKE_IO ,POLL_ERR );rcu_read_unlock (); }
Quoted 140 lines · Breno Leitao
Quoted 2 lines · Eric Dumazet
Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.
Something like this?
net: add
SO_ERR_WAKE_SYNCfor sync error-report wakeups
sock_def_error_report()wakes EPOLLERR waiters withwake_up_interruptible_poll(), whilesock_def_readable()andsock_def_write_space()already pass the sync hint. A socket withSO_TIMESTAMPINGenabled delivers every TX and ACK timestamp throughsk_error_queueand raises EPOLLERR, so the error path wakes a sleeping consumer very often.Without the sync hint the scheduler often places the woken consumer on a remote CPU, which costs a rescheduling IPI. Usama Arif measured 16,326 such IPIs/min on a 176-core host running a production workload with
SO_TIMESTAMPINGenabled. [1]Switching the error path to a sync wakeup unconditionally is not the right fix, as there are different requirements for it to be async, see [2].
Eric suggested that an options is to add
SO_ERR_WAKE_SYNCso applications choose per socket, so a consumer draining a high-rate error queue can opt in to keep the wakeup local and drop the IPI, using socket flagSO_ERR_WAKE_SYNC.Link https://lore.kernel.org/all/CANn89iLc1Bv_wmKvr_9mtGRM3gL7kgoy2Prr2SgtHY4C=ZgfBg@mail.gmail.com/ [1] open hereSuggested-by Eric Dumazet <[email protected]>Signed-off-by Breno Leitao <[email protected]>arch/alpha/include/uapi/asm/socket.h +2
-0@@ -155,6 +155,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 arch/mips/include/uapi/asm/socket.h +2
-0@@ -166,6 +166,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 arch/parisc/include/uapi/asm/socket.h +2
-0@@ -147,6 +147,8 @@#define SO_INQ 0x4052#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 0x4053 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 arch/sparc/include/uapi/asm/socket.h +2
-0@@ -148,6 +148,8 @@#define SO_INQ 0x005d#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 0x005e +#if !defined(__KERNEL__)include/net/sock.h +1
-0@@ -1022,6 +1022,7 @@ enum sock_flags {SOCK_RCVMARK ,/* Receive SO_MARK ancillary data with packet */ SOCK_RCVPRIORITY ,/* Receive SO_PRIORITY ancillary data with packet */ SOCK_TIMESTAMPING_ANY ,/* Copy of sk_tsflags & TSFLAGS_ANY */ + SOCK_ERR_WAKE_SYNC, /* Sync wakeup on error report, %SO_ERR_WAKE_SYNC */ };#define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))include/uapi/asm-generic/socket.h +2
-0@@ -150,6 +150,8 @@#define SO_INQ 84#define SCM_INQ SO_INQ +#define SO_ERR_WAKE_SYNC 85 +#if !defined(__KERNEL__)#if __BITS_PER_LONG ==64 || (defined(__x86_64__) && defined(__ILP32__))net/core/sock.c +8
-0@@ -1557,6 +1557,10 @@ int sk_setsockopt(struct sock *sk, int level, int optname,sock_valbool_flag (sk,SOCK_SELECT_ERR_QUEUE , valbool);break ; +case SO_ERR_WAKE_SYNC : +sock_valbool_flag (sk,SOCK_ERR_WAKE_SYNC , valbool); +break ; +case SO_PASSCRED :if (sk_may_scm_recv (sk)) sk->sk_scm_credentials = valbool;@@ -2064,6 +2068,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname, v.val =sock_flag (sk,SOCK_SELECT_ERR_QUEUE );break ; +case SO_ERR_WAKE_SYNC : + v.val =sock_flag (sk,SOCK_ERR_WAKE_SYNC ); +break ; +#ifdef CONFIG_NET_RX_BUSY_POLL case SO_BUSY_POLL : v.val =READ_ONCE (sk->sk_ll_usec);@@ -3641,8 +3649,12 @@ static void sock_def_error_report(struct sock *sk)
net/core/sock.c static void sock_def_error_report(struct sock *sk) · line 3657
Quoted 10 lines
net/core/sock.c +6
-2rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_poll (&wq->wait,EPOLLERR ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_ERR_WAKE_SYNC )) +wake_up_interruptible_sync_poll (&wq->wait,EPOLLERR ); +else +wake_up_interruptible_poll (&wq->wait,EPOLLERR ); + }
Not just in report, but sock_def_write_space() and sock_def_readable() as well?
Quoted 3 lines
net/core/sock.c +0
-0sk_wake_async_rcu (sk,SOCK_WAKE_IO ,POLL_ERR );rcu_read_unlock (); }
Quoted 11 lines · Eric Dumazet
Quoted 3 lines · Usama Arif
So the workload has
SO_TIMESTAMPINGenabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is whysock_def_error_reportfires.It seems we can not please everyone.
https://lore.kernel.org/netdev/[email protected]/
Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.
Instead of adding a new socket option, perhaps use SO_TIMESTAMPING to decide whether to use sync or not? There could be other socket options which also report a high rate of "errors", but until those show up I think just testing on this one socket option should be fine.