net: Use synchronous wakeups selectively
Subsystems (7)
cover net: Use synchronous wakeups selectively
The scheduler assumes in several wakeup paths that a task using WF_SYNC is likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.
During wakeup, with WF_SYNC flag set, because of the assumption that current thread is ready to give up, the wakee thread will be migrated from any core within the chip to the current LLC. If these operations are frequent, and wakers are actually not going away, then it will lead to load imbalance and hurt performance. This is especially true in architectures where LLCs are small and number of LLCs per chip are more.
Running vllm workload was run on Power10 system
No patch with patch %diff Inference Time (sec) 17.84 16.35 -8.35% llm query bandwidth (tokens/sec) 14.78 16.21 +9.68%
Lower inference time and higher tokens/sec is better.
Srikar Dronamraju (2):
net/socket: Record preference for synchronous wakeups net/sock: Propagate WF_SYNC only when requested
include/net/sock.h | 1 + net/core/sock.c | 31 +++++++++++++++++++------ net/sctp/socket.c | 10 +++++++-- net/smc/af_smc.c | 4 ++-- net/smc/smc_rx.c | 10 +++++++-- net/socket.c | 56 +++++++++++++++++++++++++++++++++++++++------- net/tipc/socket.c | 22 +++++++++++++----- net/unix/af_unix.c | 26 ++++++++++++++------- 8 files changed, 126 insertions(+), 34 deletions(-)
Signature · 1 line
2.51.0
PATCH 1/2 net/socket: Record preference for synchronous wakeups
Scheduler differentiates between affine and non-affine wakeups by the way of sync flags. Scheduler prefers to pull the tasks towards the waker if the sync flag is set.
In some cases, socket APIs are blindly requesting sync wakeups. This may cause load-balance issues and non-optimal performance.
Record whether the most recent blocking socket operation could benefit from synchronous wakeups. Subsequent readiness notifications use this hint to determine whether WF_SYNC should be propagated.
The flag is advisory and affects only wakeup placement decisions.
--- include/net/sock.h | 1 + net/socket.c | 56 +++++++++++++++++++++++++++++++++++++++------- 2 files changed, 49 insertions(+), 8 deletions(-)
include/net/sock.h +1 -0
@@ -1022,6 +1022,7 @@ enum sock_flags {SOCK_RCVMARK ,/* Receive SO_MARK ancillary data with packet */ SOCK_RCVPRIORITY ,/* Receive SO_PRIORITY ancillary data with packet */ SOCK_TIMESTAMPING_ANY ,/* Copy of sk_tsflags & TSFLAGS_ANY */ + SOCK_SYNC_WAKEUP, /* Prefer synchronous socket wakeups */ };#define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))
net/socket.c +48 -8
145 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH 1/2] net/socket: Record preference for synchronous wakeups
Hi Srikar.
Quoted 11 lines · Srikar Dronamraju
Scheduler differentiates between affine and non-affine wakeups by the way of sync flags. Scheduler prefers to pull the tasks towards the waker if the sync flag is set.
In some cases, socket APIs are blindly requesting sync wakeups. This may cause load-balance issues and non-optimal performance.
Record whether the most recent blocking socket operation could benefit from synchronous wakeups. Subsequent readiness notifications use this hint to determine whether
WF_SYNCshould be propagated.
What you mean by recent? Was it info on past set of packets? Could you please explain the flow a bit?
Shouldn't it be
- if this socket has been defined as nonblock it shouldn't use sync always?
Quoted 31 lines
The flag is advisory and affects only wakeup placement decisions.
Signed-off-by Srikar Dronamraju <[email protected]>--- include/net/sock.h | 1 + net/socket.c | 56 +++++++++++++++++++++++++++++++++++++++------- 2 files changed, 49 insertions(+), 8 deletions(-)include/net/sock.h +1
-0@@ -1022,6 +1022,7 @@ enum sock_flags {SOCK_RCVMARK ,/* Receive SO_MARK ancillary data with packet */ SOCK_RCVPRIORITY ,/* Receive SO_PRIORITY ancillary data with packet */ SOCK_TIMESTAMPING_ANY ,/* Copy of sk_tsflags & TSFLAGS_ANY */ + SOCK_SYNC_WAKEUP, /* Prefer synchronous socket wakeups */ };#define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))net/socket.c +3
-0@@ -1198,15 +1198,27 @@ static void sock_splice_eof(struct file *file) ops->splice_eof (sock); } +static inline void sock_update_sync_wakeup (struct sock *sk,bool nonblock) +{ +if (unlikely (!sk))
net/socket.c static void sock_splice_eof(struct file *file) · line 1213
Quoted 10 lines
net/socket.c +10
-0+return ; + +if (nonblock) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) +sock_reset_flag (sk,SOCK_SYNC_WAKEUP ); + }else { +if (!sock_flag (sk,SOCK_SYNC_WAKEUP )) +sock_set_flag (sk,SOCK_SYNC_WAKEUP ); + } +}
nit: You can combine two if statements. A bit easier to read.
static inline void sock_update_sync_wakeup(struct sock *sk, bool nonblock)
{
if (unlikely(!sk))
return;
if (nonblock && sock_flag(sk, SOCK_SYNC_WAKEUP))
sock_reset_flag(sk, SOCK_SYNC_WAKEUP);
else if (!nonblock && !sock_flag(sk, SOCK_SYNC_WAKEUP))
sock_set_flag(sk, SOCK_SYNC_WAKEUP);
}Quoted 140 lines
net/socket.c +35
-8+static ssize_t sock_read_iter (struct kiocb *iocb,struct iov_iter *to) {struct file *file = iocb->ki_filp;struct socket *sock = file->private_data;struct msghdr msg = {.msg_iter = *to};ssize_t res; - -if (file->f_flags &O_NONBLOCK || (iocb->ki_flags &IOCB_NOWAIT )) - msg.msg_flags =MSG_DONTWAIT ; +bool nonblock;if (iocb->ki_pos !=0 )return -ESPIPE ;@@ -1214,6 +1226,11 @@ static ssize_t sock_read_iter(struct kiocb *iocb, struct iov_iter *to)if (!iov_iter_count (to))/* Match SYS5 behaviour */ return 0 ; + nonblock = (file->f_flags &O_NONBLOCK ) || (iocb->ki_flags &IOCB_NOWAIT ); +if (nonblock) + msg.msg_flags =MSG_DONTWAIT ; + +sock_update_sync_wakeup (sock->sk, nonblock); res =sock_recvmsg (sock, &msg, msg.msg_flags); *to = msg.msg_iter;return res;@@ -1225,13 +1242,17 @@ static ssize_t sock_write_iter(struct kiocb *iocb, struct iov_iter *from)struct socket *sock = file->private_data;struct msghdr msg = {.msg_iter = *from};ssize_t res; +bool nonblock;if (iocb->ki_pos !=0 )return -ESPIPE ; -if (file->f_flags &O_NONBLOCK || (iocb->ki_flags &IOCB_NOWAIT )) + nonblock = (file->f_flags &O_NONBLOCK ) || (iocb->ki_flags &IOCB_NOWAIT ); +if (nonblock) msg.msg_flags =MSG_DONTWAIT ; +sock_update_sync_wakeup (sock->sk, nonblock); +if (sock->type ==SOCK_SEQPACKET ) msg.msg_flags |=MSG_EOR ;@@ -2221,6 +2242,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags,struct sockaddr_storage address;int err;struct msghdr msg; +bool nonblock; err =import_ubuf (ITER_SOURCE , buff, len, &msg.msg_iter);if (unlikely (err))@@ -2246,8 +2268,11 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_namelen = addr_len; } flags &= ~MSG_INTERNAL_SENDMSG_FLAGS ; -if (sock->file->f_flags &O_NONBLOCK ) + nonblock = (sock->file->f_flags &O_NONBLOCK ); +if (nonblock) flags |=MSG_DONTWAIT ; + +sock_update_sync_wakeup (sock->sk, nonblock); msg.msg_flags = flags;return __sock_sendmsg (sock, &msg); }@@ -2284,6 +2309,7 @@ int __sys_recvfrom(int fd, void __user *ubuf, size_t size, unsigned int flags, };struct socket *sock;int err, err2; +bool nonblock; err =import_ubuf (ITER_DEST , ubuf, size, &msg.msg_iter);if (unlikely (err))@@ -2297,8 +2323,11 @@ int __sys_recvfrom(int fd, void __user *ubuf, size_t size, unsigned int flags,if (unlikely (!sock))return -ENOTSOCK ; -if (sock->file->f_flags &O_NONBLOCK ) + nonblock = (sock->file->f_flags &O_NONBLOCK ); +if (nonblock) flags |=MSG_DONTWAIT ; + +sock_update_sync_wakeup (sock->sk, nonblock); err =sock_recvmsg (sock, &msg, flags);if (err >=0 && addr !=NULL ) {@@ -2634,6 +2663,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,unsigned char *ctl_buf = ctl;int ctl_len;ssize_t err; +bool nonblock; err = -ENOBUFS ;@@ -2666,8 +2696,12 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, flags &= ~MSG_INTERNAL_SENDMSG_FLAGS ; msg_sys->msg_flags = flags; -if (sock->file->f_flags &O_NONBLOCK ) + nonblock = (sock->file->f_flags &O_NONBLOCK ); +if (nonblock) msg_sys->msg_flags |=MSG_DONTWAIT ; + +sock_update_sync_wakeup (sock->sk, nonblock); +/* * If this is sendmmsg() and current destination address is same as * previously succeeded address, omit asking LSM's decision. @@ -2887,6 +2921,7 @@ static int ____sys_recvmsg(struct socket *sock, struct msghdr *msg_sys,unsigned long cmsg_ptr;int len;ssize_t err; +bool nonblock; msg_sys->msg_name = &addr; cmsg_ptr = (unsigned long )msg_sys->msg_control;@@ -2895,9 +2930,12 @@ static int ____sys_recvmsg(struct socket *sock, struct msghdr *msg_sys,/* We assume all kernel code knows the size of sockaddr_storage */ msg_sys->msg_namelen =0 ; -if (sock->file->f_flags &O_NONBLOCK ) + nonblock = (sock->file->f_flags &O_NONBLOCK ); +if (nonblock) flags |=MSG_DONTWAIT ; +sock_update_sync_wakeup (sock->sk, nonblock); +if (unlikely (nosec)) err =sock_recvmsg_nosec (sock, msg_sys, flags);else @@ -3056,6 +3094,8 @@ static int do_recvmmsg(int fd, struct mmsghdr __user *mmsg,if (flags &MSG_WAITFORONE ) flags |=MSG_DONTWAIT ; +sock_update_sync_wakeup (sock->sk, flags &MSG_WAITFORONE ); +if (timeout) {ktime_get_ts64 (&timeout64); *timeout =timespec64_sub (end_time, timeout64);
Re: [PATCH 1/2] net/socket: Record preference for synchronous wakeups
In reply to Srikar DronamrajuQuoted 8 lines · Srikar Dronamraju
Scheduler differentiates between affine and non-affine wakeups by the way of sync flags. Scheduler prefers to pull the tasks towards the waker if the sync flag is set.
In some cases, socket APIs are blindly requesting sync wakeups. This may cause load-balance issues and non-optimal performance.
Record whether the most recent blocking socket operation could benefit
What is the heuristic that determines this?
If respinning, please state that explicitly in the commit message.
Quoted 6 lines
from synchronous wakeups. Subsequent readiness notifications use this hint to determine whether
WF_SYNCshould be propagated.The flag is advisory and affects only wakeup placement decisions.
Signed-off-by Srikar Dronamraju <[email protected]>
PATCH 2/2 net/sock: Propagate WF_SYNC only when requested
In reply to Srikar DronamrajuUse SOCK_SYNC_WAKEUP to select between synchronous and asynchronous wakeup wakeup APIs. This avoids propagating WF_SYNC when no blocking waiter is expected. All wakeup locations in networking code that currently issue synchronous poll-style wakeups unconditionally are updated.
--- net/core/sock.c | 31 ++++++++++++++++++++++++------- net/sctp/socket.c | 10 ++++++++-- net/smc/af_smc.c | 4 ++-- net/smc/smc_rx.c | 10 ++++++++-- net/tipc/socket.c | 22 +++++++++++++++++----- net/unix/af_unix.c | 26 ++++++++++++++++++-------- 6 files changed, 77 insertions(+), 26 deletions(-)
net/core/sock.c +24 -7
@@ -3652,9 +3652,15 @@ void sock_def_readable(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLPRI |EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );rcu_read_unlock (); }
@@ -3670,9 +3676,15 @@ static void sock_def_write_space(struct sock *sk)*/ if (sock_writeable (sk)) { wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }/* Should agree with poll, otherwise some programs break */ sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT );
@@ -3695,10 +3707,15 @@ static void sock_def_write_space_wfree(struct sock *sk, int wmem_alloc)/* rely on refcount_sub from sock_wfree() */ smp_mb__after_atomic (); -if (wq &&waitqueue_active (&wq->wait)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (wq &&waitqueue_active (&wq->wait)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); - + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + } + }/* Should agree with poll, otherwise some programs break */ sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT ); }
net/sctp/socket.c +8 -2
@@ -9348,9 +9348,15 @@ void sctp_data_ready(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );rcu_read_unlock (); }
net/smc/af_smc.c +2 -2
@@ -819,10 +819,10 @@ static void smc_fback_wakeup_waitqueue(struct smc_sock *smc, void *key)wake_up_interruptible_all (&wq->wait); }else { flags =key_to_poll (key); -if (flags & (EPOLLIN |EPOLLOUT )) +if (flags & (EPOLLIN |EPOLLOUT ) &&sock_flag (&smc->sk,SOCK_SYNC_WAKEUP ))/* sk_data_ready or sk_write_space */ wake_up_interruptible_sync_poll (&wq->wait, flags); -else if (flags &EPOLLERR ) +else /* sk_error_report */ wake_up_interruptible_poll (&wq->wait, flags); }
net/smc/smc_rx.c +8 -2
@@ -39,9 +39,15 @@ static void smc_rx_wake_up(struct sock *sk)/* called already in smc_listen_work() */ rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI |EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLPRI | +EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );if ((sk->sk_shutdown ==SHUTDOWN_MASK ) || (sk->sk_state ==SMC_CLOSED ))
net/tipc/socket.c +17 -5
@@ -2116,9 +2116,15 @@ static void tipc_write_space(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + } + }rcu_read_unlock (); }
@@ -2134,9 +2140,15 @@ static void tipc_data_ready(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | -EPOLLRDNORM |EPOLLRDBAND ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + } + }rcu_read_unlock (); }
net/unix/af_unix.c +18 -8
@@ -601,9 +601,15 @@ static void unix_write_space(struct sock *sk)rcu_read_lock ();if (unix_writable (sk,READ_ONCE (sk->sk_state))) { wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait, -EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT ); }rcu_read_unlock ();
@@ -2603,11 +2609,15 @@ int __unix_dgram_recvmsg(struct sock *sk, struct msghdr *msg, size_t size, goto out; } -if (wq_has_sleeper (&u->peer_wait)) -wake_up_interruptible_sync_poll (&u->peer_wait, -EPOLLOUT |EPOLLWRNORM | -EPOLLWRBAND ); - +if (wq_has_sleeper (&u->peer_wait)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&u->peer_wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&u->peer_wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }if (msg->msg_name) {unix_copy_addr (msg, skb->sk);
Signature · 1 line
2.52.0
Re: [PATCH 2/2] net/sock: Propagate WF_SYNC only when requested
Hi Srikar,
Quoted 5 lines · Srikar Dronamraju
Use
SOCK_SYNC_WAKEUPto select between synchronous and asynchronous wakeup wakeup APIs. This avoids propagatingWF_SYNCwhen no blocking waiter is expected. All wakeup locations in networking code that currently issue synchronous poll-style wakeups unconditionally are updated.
You can also add the performance data in the cover-letter to this patch.
Quoted 205 lines
Signed-off-by Srikar Dronamraju <[email protected]>--- net/core/sock.c | 31 ++++++++++++++++++++++++------- net/sctp/socket.c | 10 ++++++++-- net/smc/af_smc.c | 4 ++-- net/smc/smc_rx.c | 10 ++++++++-- net/tipc/socket.c | 22 +++++++++++++++++----- net/unix/af_unix.c | 26 ++++++++++++++++++-------- 6 files changed, 77 insertions(+), 26 deletions(-)net/core/sock.c +24
-7@@ -3652,9 +3652,15 @@ void sock_def_readable(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLPRI |EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );rcu_read_unlock (); }@@ -3670,9 +3676,15 @@ static void sock_def_write_space(struct sock *sk)*/ if (sock_writeable (sk)) { wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }/* Should agree with poll, otherwise some programs break */ sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT );@@ -3695,10 +3707,15 @@ static void sock_def_write_space_wfree(struct sock *sk, int wmem_alloc)/* rely on refcount_sub from sock_wfree() */ smp_mb__after_atomic (); -if (wq &&waitqueue_active (&wq->wait)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (wq &&waitqueue_active (&wq->wait)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); - + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + } + }/* Should agree with poll, otherwise some programs break */ sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT ); }net/sctp/socket.c +8
-2@@ -9348,9 +9348,15 @@ void sctp_data_ready(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );rcu_read_unlock (); }net/smc/af_smc.c +2
-2@@ -819,10 +819,10 @@ static void smc_fback_wakeup_waitqueue(struct smc_sock *smc, void *key)wake_up_interruptible_all (&wq->wait); }else { flags =key_to_poll (key); -if (flags & (EPOLLIN |EPOLLOUT )) +if (flags & (EPOLLIN |EPOLLOUT ) &&sock_flag (&smc->sk,SOCK_SYNC_WAKEUP ))/* sk_data_ready or sk_write_space */ wake_up_interruptible_sync_poll (&wq->wait, flags); -else if (flags &EPOLLERR ) +else /* sk_error_report */ wake_up_interruptible_poll (&wq->wait, flags); }net/smc/smc_rx.c +8
-2@@ -39,9 +39,15 @@ static void smc_rx_wake_up(struct sock *sk)/* called already in smc_listen_work() */ rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN |EPOLLPRI |EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN |EPOLLPRI | +EPOLLRDNORM |EPOLLRDBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_WAITD ,POLL_IN );if ((sk->sk_shutdown ==SHUTDOWN_MASK ) || (sk->sk_state ==SMC_CLOSED ))net/tipc/socket.c +17
-5@@ -2116,9 +2116,15 @@ static void tipc_write_space(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT | +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLOUT | +EPOLLWRNORM |EPOLLWRBAND ); + } + }rcu_read_unlock (); }@@ -2134,9 +2140,15 @@ static void tipc_data_ready(struct sock *sk)rcu_read_lock (); wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | -EPOLLRDNORM |EPOLLRDBAND ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + }else { +wake_up_interruptible_poll (&wq->wait,EPOLLIN | +EPOLLRDNORM |EPOLLRDBAND ); + } + }rcu_read_unlock (); }net/unix/af_unix.c +11
-8@@ -601,9 +601,15 @@ static void unix_write_space(struct sock *sk)rcu_read_lock ();if (unix_writable (sk,READ_ONCE (sk->sk_state))) { wq =rcu_dereference (sk->sk_wq); -if (skwq_has_sleeper (wq)) -wake_up_interruptible_sync_poll (&wq->wait, -EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); +if (skwq_has_sleeper (wq)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) { +wake_up_interruptible_sync_poll (&wq->wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&wq->wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }sk_wake_async_rcu (sk,SOCK_WAKE_SPACE ,POLL_OUT ); }rcu_read_unlock ();@@ -2603,11 +2609,15 @@ int __unix_dgram_recvmsg(struct sock *sk, struct msghdr *msg, size_t size, goto out; } -if (wq_has_sleeper (&u->peer_wait)) -wake_up_interruptible_sync_poll (&u->peer_wait, -EPOLLOUT |EPOLLWRNORM | -EPOLLWRBAND ); - +if (wq_has_sleeper (&u->peer_wait)) { +if (sock_flag (sk,SOCK_SYNC_WAKEUP )) {
net/unix/af_unix.c int __unix_dgram_recvmsg(struct sock *sk, struct msghdr *msg, size_t size, · line 2623
Quoted 10 lines
net/unix/af_unix.c +7
-0+wake_up_interruptible_sync_poll (&u->peer_wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + }else { +wake_up_interruptible_poll (&u->peer_wait, +EPOLLOUT |EPOLLWRNORM |EPOLLWRBAND ); + } + }if (msg->msg_name) {unix_copy_addr (msg, skb->sk);
Would it make sense to write a macro or a wrapper function do the same instead of sprinkling the same at all the places?
similar comment for patch 1.
IMHO, it would make it easier to read.
Jakub Kicinski
maintainerQuoted 12 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.During wakeup, with
WF_SYNCflag set, because of the assumption that current thread is ready to give up, the wakee thread will be migrated from any core within the chip to the current LLC. If these operations are frequent, and wakers are actually not going away, then it will lead to load imbalance and hurt performance. This is especially true in architectures where LLCs are small and number of LLCs per chip are more.
Quoted 6 lines
Running vllm workload was run on Power10 system
No patch with patch %diff Inference Time (sec) 17.84 16.35 -8.35% llm query bandwidth (tokens/sec) 14.78 16.21 +9.68%Lower inference time and higher tokens/sec is better.
Somewhat related recent patch:
https://lore.kernel.org/all/[email protected]/
Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific. The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking.
Signature · 1 line
pw-bot: cr
Quoted 25 lines · Jakub Kicinski
Quoted 12 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.During wakeup, with
WF_SYNCflag set, because of the assumption that current thread is ready to give up, the wakee thread will be migrated from any core within the chip to the current LLC. If these operations are frequent, and wakers are actually not going away, then it will lead to load imbalance and hurt performance. This is especially true in architectures where LLCs are small and number of LLCs per chip are more.Quoted 6 lines
Running vllm workload was run on Power10 system
No patch with patch %diff Inference Time (sec) 17.84 16.35 -8.35% llm query bandwidth (tokens/sec) 14.78 16.21 +9.68%Lower inference time and higher tokens/sec is better.
Somewhat related recent patch:
Quoted 7 lines
Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific. The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking.
--
pw-bot: cr
In any case, adding code to the fast path in every network system call is undesirable.
Quoted 20 lines · Eric Dumazet
Quoted 11 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.Somewhat related recent patch:
Quoted 7 lines
Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific. The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking.
--
pw-bot: cr
Quoted 2 lines
In any case, adding code to the fast path in every network system call is undesirable.
Yes, I understand adding code in fast path is undesirable. However as mentioned previous reply, we need to differentiate between blocking and non-blocking API. Doing unconditional nonblocking, is probably even worse.
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
Quoted 23 lines · Srikar Dronamraju
Quoted 20 lines · Eric Dumazet
Quoted 18 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.Somewhat related recent patch:
https://lore.kernel.org/all/[email protected]/
Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific. The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking.
--
pw-bot: cr
Quoted 6 lines
Quoted 2 lines
In any case, adding code to the fast path in every network system call is undesirable.
Yes, I understand adding code in fast path is undesirable. However as mentioned previous reply, we need to differentiate between blocking and non-blocking API. Doing unconditional nonblocking, is probably even worse.
I suggested that applications choose what they prefer for each socket, once and for all.
(This can also be a cgroup/BPF setting)
Flipping a socket bit at each recvmsg()/sendmsg() is a bad idea, you might break old application expectations.
Quoted 19 lines · Eric Dumazet
Quoted 11 lines · Srikar Dronamraju
Quoted 8 lines · Eric Dumazet
Quoted 6 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.Quoted 6 lines
Quoted 2 lines
In any case, adding code to the fast path in every network system call is undesirable.
Yes, I understand adding code in fast path is undesirable. However as mentioned previous reply, we need to differentiate between blocking and non-blocking API. Doing unconditional nonblocking, is probably even worse.
Quoted 6 lines
I suggested that applications choose what they prefer for each socket, once and for all.
(This can also be a cgroup/BPF setting)Flipping a socket bit at each recvmsg()/sendmsg() is a bad idea, you might break old application expectations.
Set once per socket is fine. However at the time of wake, we still have to check the socket option right? I hope that overhead is fine.
Also do you have any precedent for once per socket option that I could look at?
Thanks for your inputs.
Signature · 2 lines
Thanks and Regards Srikar Dronamraju
* Jakub Kicinski <[email protected]> [2026-07-22 10:08:42]:
Hi Jakub, Thanks for taking a look.
Quoted 20 lines · On Tue, 14 Jul 2026 07:09:41 +0530 Srikar Dronamraju wrote:
Quoted 12 lines
The scheduler assumes in several wakeup paths that a task using
WF_SYNCis likely to yield the CPU shortly. However several networking wakeup paths unconditionally use synchronous wakeups even when the waking task continues execution.During wakeup, with
WF_SYNCflag set, because of the assumption that current thread is ready to give up, the wakee thread will be migrated from any core within the chip to the current LLC. If these operations are frequent, and wakers are actually not going away, then it will lead to load imbalance and hurt performance. This is especially true in architectures where LLCs are small and number of LLCs per chip are more.Quoted 6 lines
Running vllm workload was run on Power10 system
No patch with patch %diff Inference Time (sec) 17.84 16.35 -8.35% llm query bandwidth (tokens/sec) 14.78 16.21 +9.68%Lower inference time and higher tokens/sec is better.
Quoted 7 lines
Somewhat related recent patch:
https://lore.kernel.org/all/[email protected]/
Maybe other maintainers will chime in, but I'm not convinced by the heuristics you're adding. The behavior will be workload specific.
Currently scheduler provides 2 APIs because it supports 2 subtle behaviours. 1st API being wake_up_interruptible_poll and 2nd API being wake_up_interruptible_sync_poll. The reason scheduler provides 2 APIs is because the scheduler cant decide which behaviour the callee wants.
If the scheduler could have figured out which behaviour then there would have been only 1 API. So the onus of using the right API is on the callee.
Now the behaviour between the 2 APIs depends on fact that the callee is going to give up the CPU now or if the callee will continue to run. If the callee is going to continue to run, its better to use the more generic API of the two which is wake_up_interruptible_poll. However if the callee is giving up the CPU, its better to use wake_up_interruptible_sync_poll.
@peterz @ingom can you please confirm this behaviour is inline with scheduler maintainers view.
Currently the networking code as I see is unconditionally (or blindly) asking for sync behaviour i.e the socket API is saying the callees are going to give up CPU soon. However the problem I am seeing is the callees are actually not giving up CPU.
Quoted 3 lines
The extent of networking involvement should be letting the scheduler know that we're waking from IO, not leaking scheduler heuristics into networking.
In this change, what we are depending on is file's flag that the socket request is blocking or non-blocking. So why do we say we are leaking scheduler hueristics? Would using an API or another API mean leaking scheduler hueristics? If we are blocking, this change will use wake_up_interruptible_sync_poll and if non-blocking, it will use wake_up_interruptible_poll.
Signature · 2 lines
Thanks and Regards Srikar Dronamraju