Skip to contentSkip to conversation

net: use sync wakeups for socket error reports

PATCH In review Usama Arif 4 participants 7 messages Jul 8 – Jul 17

Subsystems (8)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH 1/1 net: use sync wakeups for socket error reports

sock_def_readable() and sock_def_write_space() pass WF_SYNC to their waitqueue wakeups, switch sock_def_error_report() to wake_up_interruptible_sync_poll() so EPOLLERR waiters get the same hint. ep_poll_callback() forwards it through to
try_to_wake_up() / select_task_rq_fair(), where wake_affine() can prefer the waker CPU and skip a cross-CPU wakelist IPI when its heuristics agree.

WF_SYNC matches the choice already made for readable/write_space. Some error reports have the same producer/consumer shape: the waker has queued an skb on sk->sk_error_queue, and the wakee is about to dequeue and copy it out. Other reports publish sk_err/socket state directly, but still wake a task likely to consume that socket state immediately.

Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the sock_def_error_report -> ep_poll_callback -> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue chain to 16,326 IPIs/min. Switching to wake_up_interruptible_sync_poll() will help reduce those IPIs.

---
 net/core/sock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/sock.c +1 -1
index 8a59bfaa8096..c724f1442987 100644
@@ -3639,7 +3639,7 @@ static void sock_def_error_report(struct sock *sk)
 	rcu_read_lock();
 	wq = rcu_dereference(sk->sk_wq);
 	if (skwq_has_sleeper(wq))
-		wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+		wake_up_interruptible_sync_poll(&wq->wait, EPOLLERR);
 	sk_wake_async_rcu(sk, SOCK_WAKE_IO, POLL_ERR);
 	rcu_read_unlock();
 }
Signature · 1 line
2.53.0-Meta
#1 lore raw
Quoted 5 lines · Usama Arif

Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the sock_def_error_report -> ep_poll_callback -> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue chain to 16,326 IPIs/min.

I am interested in why so many sock_def_error_report().

That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?

Can you share the full stack above sock_def_error_report()?

author
#2 lore raw
Quoted 7 lines · Breno Leitao
Quoted 5 lines · Usama Arif

Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the sock_def_error_report -> ep_poll_callback -> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue chain to 16,326 IPIs/min.

Quoted 6 lines

I am interested in why so many sock_def_error_report().

That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?

Can you share the full stack above sock_def_error_report()?

I ran this bpftrace script the host now (results added at the end):

sudo bpftrace -e '
kprobe:sock_def_error_report

  {
    @wake_src[kstack()] = count();
  }
  interval:s:60
  {
    print(@wake_src, 5);
    exit();

}'

The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE) to get the SND/ACK timestamp.

So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is why sock_def_error_report fires.

Attached 2 probes
@wake_src[

        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1630
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+62
        asm_common_interrupt+34
]: 1812
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_dequeue_err_skb+194
        ipv6_recv_error+74
        bpf_trampoline_6442598004+73
        ____sys_recvmsg.llvm.18251018526254450710+168
        ___sys_recvmsg+312
        __x64_sys_recvmsg+95
        do_syscall_64+316
        entry_SYSCALL_64_after_hwframe+75
]: 8045
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1258
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+125
        asm_common_interrupt+34
        cpuidle_enter_state+202
        cpuidle_enter+40
        cpu_startup_entry+497
        ap_starting+0
        common_startup_64+318
]: 12603
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        tcp_ack+3399
        tcp_rcv_established+1258
        tcp_v6_do_rcv+372
        tcp_v6_rcv+4748
        ip6_protocol_deliver_rcu+653
        ip6_input_finish+79
        ip6_input+43
        ipv6_list_rcv+4339
        __netif_receive_skb_list_core+244
        netif_receive_skb_list_internal+433
        napi_complete_done+149
        bnxt_poll_p5+499
        net_rx_action+513
        irq_exit_rcu+312
        common_interrupt+62
        asm_common_interrupt+34
]: 19314
@wake_src[
        sock_def_error_report+1
        sk_error_report+17
        sock_queue_err_skb+285
        __skb_tstamp_tx+903
        bnxt_start_xmit+1769
        dev_hard_start_xmit+160
        sch_direct_xmit+165
        __qdisc_run+714
        __dev_queue_xmit+2052
        skb_do_redirect+2531
        netkit_xmit+715
        dev_hard_start_xmit+160
        __dev_queue_xmit+1049
        ip6_finish_output2+848
        ip6_finish_output+213
        ip6_output+86
        ip6_xmit+933
        inet6_csk_xmit+163
        __tcp_transmit_skb+2733
        tcp_write_xmit+2948
        __tcp_push_pending_frames+46
        tcp_sendmsg_locked+4187
        tcp_sendmsg+40
        __x64_sys_sendmsg+567
        do_syscall_64+316
        entry_SYSCALL_64_after_hwframe+75

]: 33514

#3 lore raw
Quoted 40 lines · Usama Arif
Quoted 7 lines · Breno Leitao
Quoted 5 lines · Usama Arif

Measured on a 176-core EPYC 9D64 host running a Meta production workload, bpftrace on tracepoint:ipi:ipi_send_cpu with a kstack filter attributed the sock_def_error_report -> ep_poll_callback -> try_to_wake_up -> ttwu_queue_wakelist -> __smp_call_single_queue chain to 16,326 IPIs/min.

Quoted 6 lines

I am interested in why so many sock_def_error_report().

That's seems a lot for genuine socket errors (RST/ICMP) on a healthy host, so I suspect these aren't errors at all?

Can you share the full stack above sock_def_error_report()?

I ran this bpftrace script the host now (results added at the end):

sudo bpftrace -e '
kprobe:sock_def_error_report

  {
    @wake_src[kstack()] = count();
  }
  interval:s:60
  {
    print(@wake_src, 5);
    exit();

}'

The biggest source is tcp_sendmsg -> __skb_tstamp_tx, which as you said is not an actual error. __skb_tstamp_tx clones the outgoing skb, tags it with ee_origin = SO_EE_ORIGIN_TIMESTAMPING and ee_errno = ENOMSG, enqueues it on sk->sk_error_queue via sock_queue_err_skb, and calls sk_error_report so epoll raises EPOLLERR. Userspace then reads it with recvmsg(MSG_ERRQUEUE) to get the SND/ACK timestamp.

Quoted 4 lines

So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is why sock_def_error_report fires.

It seems we can not please everyone.

https://lore.kernel.org/netdev/[email protected]/

Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.

#4 lore raw
Quoted 2 lines · Eric Dumazet

Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.

Something like this?

net: add SO_ERR_WAKE_SYNC for sync error-report wakeups

sock_def_error_report() wakes EPOLLERR waiters with wake_up_interruptible_poll(), while sock_def_readable() and sock_def_write_space() already pass the sync hint. A socket with SO_TIMESTAMPING enabled delivers every TX and ACK timestamp through sk_error_queue and raises EPOLLERR, so the error path wakes a sleeping consumer very often.

Without the sync hint the scheduler often places the woken consumer on a remote CPU, which costs a rescheduling IPI. Usama Arif measured 16,326 such IPIs/min on a 176-core host running a production workload with SO_TIMESTAMPING enabled. [1]

Switching the error path to a sync wakeup unconditionally is not the right fix, as there are different requirements for it to be async, see [2].

Eric suggested that an options is to add SO_ERR_WAKE_SYNC so applications choose per socket, so a consumer draining a high-rate error queue can opt in to keep the wakeup local and drop the IPI, using socket flag SO_ERR_WAKE_SYNC.

arch/alpha/include/uapi/asm/socket.h +2 -0
index 5ef57f88df6b3..2a3c27aaf4e95 100644
@@ -155,6 +155,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/mips/include/uapi/asm/socket.h +2 -0
index 72fb1b006da93..00f31c74a63df 100644
@@ -166,6 +166,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/parisc/include/uapi/asm/socket.h +2 -0
index c16ec36dfee6b..db5b6cad17d49 100644
@@ -147,6 +147,8 @@
 #define SO_INQ			0x4052
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	0x4053
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/sparc/include/uapi/asm/socket.h +2 -0
index 71befa109e1cf..5e9ff3634265c 100644
@@ -148,6 +148,8 @@
 #define SO_INQ                   0x005d
 #define SCM_INQ                  SO_INQ
 
+#define SO_ERR_WAKE_SYNC         0x005e
+
 #if !defined(__KERNEL__)
 
 
include/net/sock.h +1 -0
index 51185222aac29..d8ee1dae8ecaf 100644
@@ -1022,6 +1022,7 @@ enum sock_flags {
 	SOCK_RCVMARK, /* Receive SO_MARK  ancillary data with packet */
 	SOCK_RCVPRIORITY, /* Receive SO_PRIORITY ancillary data with packet */
 	SOCK_TIMESTAMPING_ANY, /* Copy of sk_tsflags & TSFLAGS_ANY */
+	SOCK_ERR_WAKE_SYNC, /* Sync wakeup on error report, %SO_ERR_WAKE_SYNC */
 };
 
 #define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))
include/uapi/asm-generic/socket.h +2 -0
index 53b5a8c002b1e..5527c9318b40e 100644
@@ -150,6 +150,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
net/core/sock.c +14 -2
index 498a57f34f5b5..59caab6a7223a 100644
@@ -1557,6 +1557,10 @@ int sk_setsockopt(struct sock *sk, int level, int optname,
 		sock_valbool_flag(sk, SOCK_SELECT_ERR_QUEUE, valbool);
 		break;
 
+	case SO_ERR_WAKE_SYNC:
+		sock_valbool_flag(sk, SOCK_ERR_WAKE_SYNC, valbool);
+		break;
+
 	case SO_PASSCRED:
 		if (sk_may_scm_recv(sk))
 			sk->sk_scm_credentials = valbool;
@@ -2064,6 +2068,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname,
 		v.val = sock_flag(sk, SOCK_SELECT_ERR_QUEUE);
 		break;
 
+	case SO_ERR_WAKE_SYNC:
+		v.val = sock_flag(sk, SOCK_ERR_WAKE_SYNC);
+		break;
+
 #ifdef CONFIG_NET_RX_BUSY_POLL
 	case SO_BUSY_POLL:
 		v.val = READ_ONCE(sk->sk_ll_usec);
@@ -3641,8 +3649,12 @@ static void sock_def_error_report(struct sock *sk)
 
 	rcu_read_lock();
 	wq = rcu_dereference(sk->sk_wq);
-	if (skwq_has_sleeper(wq))
-		wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+	if (skwq_has_sleeper(wq)) {
+		if (sock_flag(sk, SOCK_ERR_WAKE_SYNC))
+			wake_up_interruptible_sync_poll(&wq->wait, EPOLLERR);
+		else
+			wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+	}
 	sk_wake_async_rcu(sk, SOCK_WAKE_IO, POLL_ERR);
 	rcu_read_unlock();
 }
author
#5 lore raw
Quoted 140 lines · Breno Leitao
Quoted 2 lines · Eric Dumazet

Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.

Something like this?

net: add SO_ERR_WAKE_SYNC for sync error-report wakeups

sock_def_error_report() wakes EPOLLERR waiters with wake_up_interruptible_poll(), while sock_def_readable() and sock_def_write_space() already pass the sync hint. A socket with SO_TIMESTAMPING enabled delivers every TX and ACK timestamp through sk_error_queue and raises EPOLLERR, so the error path wakes a sleeping consumer very often.

Without the sync hint the scheduler often places the woken consumer on a remote CPU, which costs a rescheduling IPI. Usama Arif measured 16,326 such IPIs/min on a 176-core host running a production workload with SO_TIMESTAMPING enabled. [1]

Switching the error path to a sync wakeup unconditionally is not the right fix, as there are different requirements for it to be async, see [2].

Eric suggested that an options is to add SO_ERR_WAKE_SYNC so applications choose per socket, so a consumer draining a high-rate error queue can opt in to keep the wakeup local and drop the IPI, using socket flag SO_ERR_WAKE_SYNC.

arch/alpha/include/uapi/asm/socket.h +2 -0
index 5ef57f88df6b3..2a3c27aaf4e95 100644
@@ -155,6 +155,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/mips/include/uapi/asm/socket.h +2 -0
index 72fb1b006da93..00f31c74a63df 100644
@@ -166,6 +166,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/parisc/include/uapi/asm/socket.h +2 -0
index c16ec36dfee6b..db5b6cad17d49 100644
@@ -147,6 +147,8 @@
 #define SO_INQ			0x4052
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	0x4053
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64
arch/sparc/include/uapi/asm/socket.h +2 -0
index 71befa109e1cf..5e9ff3634265c 100644
@@ -148,6 +148,8 @@
 #define SO_INQ                   0x005d
 #define SCM_INQ                  SO_INQ
 
+#define SO_ERR_WAKE_SYNC         0x005e
+
 #if !defined(__KERNEL__)
 
 
include/net/sock.h +1 -0
index 51185222aac29..d8ee1dae8ecaf 100644
@@ -1022,6 +1022,7 @@ enum sock_flags {
 	SOCK_RCVMARK, /* Receive SO_MARK  ancillary data with packet */
 	SOCK_RCVPRIORITY, /* Receive SO_PRIORITY ancillary data with packet */
 	SOCK_TIMESTAMPING_ANY, /* Copy of sk_tsflags & TSFLAGS_ANY */
+	SOCK_ERR_WAKE_SYNC, /* Sync wakeup on error report, %SO_ERR_WAKE_SYNC */
 };
 
 #define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))
include/uapi/asm-generic/socket.h +2 -0
index 53b5a8c002b1e..5527c9318b40e 100644
@@ -150,6 +150,8 @@
 #define SO_INQ			84
 #define SCM_INQ			SO_INQ
 
+#define SO_ERR_WAKE_SYNC	85
+
 #if !defined(__KERNEL__)
 
 #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__))
net/core/sock.c +8 -0
index 498a57f34f5b5..59caab6a7223a 100644
@@ -1557,6 +1557,10 @@ int sk_setsockopt(struct sock *sk, int level, int optname,
 		sock_valbool_flag(sk, SOCK_SELECT_ERR_QUEUE, valbool);
 		break;
 
+	case SO_ERR_WAKE_SYNC:
+		sock_valbool_flag(sk, SOCK_ERR_WAKE_SYNC, valbool);
+		break;
+
 	case SO_PASSCRED:
 		if (sk_may_scm_recv(sk))
 			sk->sk_scm_credentials = valbool;
@@ -2064,6 +2068,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname,
 		v.val = sock_flag(sk, SOCK_SELECT_ERR_QUEUE);
 		break;
 
+	case SO_ERR_WAKE_SYNC:
+		v.val = sock_flag(sk, SOCK_ERR_WAKE_SYNC);
+		break;
+
 #ifdef CONFIG_NET_RX_BUSY_POLL
 	case SO_BUSY_POLL:
 		v.val = READ_ONCE(sk->sk_ll_usec);
@@ -3641,8 +3649,12 @@ static void sock_def_error_report(struct sock *sk)
 

net/core/sock.c static void sock_def_error_report(struct sock *sk) · line 3657

Quoted 10 lines
net/core/sock.c +6 -2
 	rcu_read_lock();
 	wq = rcu_dereference(sk->sk_wq);
-	if (skwq_has_sleeper(wq))
-		wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+	if (skwq_has_sleeper(wq)) {
+		if (sock_flag(sk, SOCK_ERR_WAKE_SYNC))
+			wake_up_interruptible_sync_poll(&wq->wait, EPOLLERR);
+		else
+			wake_up_interruptible_poll(&wq->wait, EPOLLERR);
+	}

Not just in report, but sock_def_write_space() and sock_def_readable() as well?

Quoted 3 lines
net/core/sock.c +0 -0
 	sk_wake_async_rcu(sk, SOCK_WAKE_IO, POLL_ERR);
 	rcu_read_unlock();
 }
#6 lore raw
In reply to Eric Dumazet
Quoted 11 lines · Eric Dumazet
Quoted 3 lines · Usama Arif

So the workload has SO_TIMESTAMPING enabled on its TCP sockets, and every packet completion and every ACK triggers a timestamp delivery through the error-queue path, which is why sock_def_error_report fires.

It seems we can not please everyone.

https://lore.kernel.org/netdev/[email protected]/

Perhaps this SYNC heuristic should be a per-socket choice so that applications can decide what is best for them.

Instead of adding a new socket option, perhaps use SO_TIMESTAMPING to decide whether to use sync or not? There could be other socket options which also report a high rate of "errors", but until those show up I think just testing on this one socket option should be fine.