should use scheduler sync hint in tcp_prequeue()?
Subsystems (5)
PATCH 1/1 should use scheduler sync hint in tcp_prequeue()?
Greetings network land.
The reason for this query is that wake_affine() fails if there is one and only one task on a runqueue to encourage tasks spreading out, which increases cpu utilization. However, for tasks which are communicating at high frequency, the cost of the resulting cache misses, should partners land in non-shared caches, is horrible to behold. My Q6600 has shared caches, which may or may not be hit IFF something perturbs the system, and bounces partner to the right core. That won't happen on a box with no shared caches of course, and even with shared caches available, the pain is highly visible in the TCP numbers below.
The sync hint tells wake_affine() that the waker is likely going to sleep soonish, so it subtracts the waker from the load imbalance calculation, allowing the partner task to be awakened affine. In the shared cache available case, that is also an enabler that the task be placed in an idle shared cache, which can increase throughput quite a bit (see .31 vs .33 AF UNIX), or may cost a bit if there is little to no execution overlap (see pipe).
Now, I _could_ change wake_affine() to globally succeed in the one task case, but am loath to do so because that very well may upset delicate load balancing apple cart. I think it's much safer to target the spot that I know hurts like hell. Thoughts?
include/net/tcp.h +1 -1
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
That was the RFC bit, now moving along to the RFT. Busy folks can fast forward to relevant numbers, or just poke 'D' :)
Below the lmbench numbers is a diff from my experimental fast-path cycle recovery tree against .33. That's me hoping that some folks with some time on their hands to test the high speed low drag things that network folks do, and who wouldn't mind seeing such things go a bit faster, will become curious and give it a go.
Should anyone do any testing, I'd be particularly interested in seeing _negative_ results, and a way to reproduce them if it can be done on localhost + generic Q6600 box. No high speed low drag hardware here unfortunately. Any results highly welcome, food for thought comes in many flavors.
(offline preferred... may be considered spam)
Oh, almost forgot. Second set of three runs for 33-smpx is with SD_SHARE_PKG_RESOURCES disabled on MC domain to disallow use of shared cache and compare cost/benefit. TCP numbers is what's relevant to $subject.
*Local* Communication latencies in microseconds - smaller is better
---------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
marge 2.6.31.12-smp 0.730 2.845 4.85 6.463 11.3 26.2 14.9 31. marge 2.6.31.12-smp 0.750 2.864 4.78 6.460 11.2 22.9 14.6 31. marge 2.6.31.12-smp 0.710 2.835 4.81 6.478 11.5 11.0 14.5 30. marge 2.6.33-smp 1.320 4.552 5.02 9.169 12.5 26.5 15.4 18. marge 2.6.33-smp 1.450 4.621 5.45 9.286 12.5 11.4 15.4 18. marge 2.6.33-smp 1.450 4.589 5.53 9.168 12.6 27.5 15.4 18. marge 2.6.33-smpx 1.160 3.565 5.97 7.513 11.3 9.776 13.9 18. marge 2.6.33-smpx 1.140 3.569 6.02 7.479 11.2 9.849 14.0 18. marge 2.6.33-smpx 1.090 3.563 6.39 7.450 11.2 9.785 14.0 18. marge 2.6.33-smpx 0.730 2.665 4.85 6.565 11.9 10.3 15.2 31. marge 2.6.33-smpx 0.740 2.701 4.03 6.573 11.7 10.3 15.4 31. marge 2.6.33-smpx 0.710 2.753 4.86 6.533 11.7 10.3 15.3 31.
File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page 100fd
Create Delete Create Delete Latency Fault Fault selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
marge 2.6.31.12-smp 798.0 0.503 0.96940 1.434
marge 2.6.31.12-smp 795.0 0.523 0.96630 1.435
marge 2.6.31.12-smp 791.0 0.536 0.97410 1.438
marge 2.6.33-smp 841.0 0.603 1.00320 1.492
marge 2.6.33-smp 818.0 0.582 0.99960 1.496
marge 2.6.33-smp 814.0 0.578 1.00750 1.492
marge 2.6.33-smpx 782.0 0.486 0.97290 1.489
marge 2.6.33-smpx 787.0 0.491 0.97950 1.498
marge 2.6.33-smpx 801.0 0.469 0.97690 1.492
marge 2.6.33-smpx 783.0 0.496 0.97200 1.497
marge 2.6.33-smpx 809.0 0.505 0.98070 1.504
marge 2.6.33-smpx 796.0 0.501 0.98820 1.502*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
marge 2.6.31.12-smp 2821 2971 762. 2829.2 4799.0 1243.0 1230.3 4469 1682. marge 2.6.31.12-smp 2824 2931 760. 2833.3 4736.5 1239.5 1235.8 4462 1678. marge 2.6.31.12-smp 2796 2936 1139 2843.3 4815.7 1242.8 1234.6 4471 1685. marge 2.6.33-smp 2670 5151 739. 2816.6 4768.5 1243.7 1237.2 4389 1684. marge 2.6.33-smp 2627 5126 1135 2806.9 4783.1 1245.1 1236.1 4413 1684. marge 2.6.33-smp 2582 5037 1137 2799.6 4755.4 1242.0 1239.1 4471 1683. marge 2.6.33-smpx 2848 5184 2972 2820.5 4804.8 1242.6 1236.9 4315 1686. marge 2.6.33-smpx 2804 5183 2934 2822.8 4759.3 1245.0 1234.7 4462 1688. marge 2.6.33-smpx 2729 5177 2920 2837.6 4820.0 1246.9 1238.5 4467 1684. marge 2.6.33-smpx 2843 2896 1928 2786.5 4751.2 1242.2 1238.6 4493 1682. marge 2.6.33-smpx 2869 2886 1936 2841.4 4748.9 1244.3 1237.7 4456 1683. marge 2.6.33-smpx 2845 2895 1947 2836.0 4813.6 1242.7 1236.3 4473 1674.
include/linux/sched.h +4 -6
@@ -278,6 +278,10 @@ static inline int select_nohz_load_balancer(int cpu) }#endif +#ifdef CONFIG_NO_HZ +extern int nohz_ratelimit (int cpu); +#endif +/* * Only dump TASK_* tasks. (0 for all tasks) */
@@ -1160,14 +1164,8 @@ struct sched_entity {u64 vruntime;u64 prev_sum_exec_runtime; -u64 last_wakeup; -u64 avg_overlap; -u64 nr_migrations; -u64 start_runtime; -u64 avg_wakeup; -#ifdef CONFIG_SCHEDSTATS u64 wait_start;u64 wait_max;
include/net/tcp.h +1 -1
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
kernel/sched.c +28 -67
229 lines not on this page · raw message
kernel/sched_debug.c +0 -2
@@ -407,8 +407,6 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)PN (se.exec_start);PN (se.vruntime);PN (se.sum_exec_runtime); -PN (se.avg_overlap); -PN (se.avg_wakeup); nr_switches = p->nvcsw + p->nivcsw;
kernel/sched_fair.c +10 -51
103 lines not on this page · raw message
kernel/sched_features.h +0 -22
@@ -31,12 +31,6 @@ SCHED_FEAT(START_DEBIT, 1)SCHED_FEAT (WAKEUP_PREEMPT ,1 )/* -* Compute wakeup_gran based on task behaviour, clipped to -* [0, sched_wakeup_gran_ns] - */ -SCHED_FEAT (ADAPTIVE_GRAN ,1 ) - -/* * When converting the wakeup granularity to virtual time, do it such * that heavier tasks preempting a lighter task have an edge. */
@@ -48,12 +42,6 @@ SCHED_FEAT(ASYM_GRAN, 1)SCHED_FEAT (WAKEUP_SYNC ,0 )/* -* Wakeup preempt based on task behaviour. Tasks that do not overlap -* don't get preempted. - */ -SCHED_FEAT (WAKEUP_OVERLAP ,0 ) - -/* * Use the SYNC wakeup hint, pipes and the likes use this to indicate * the remote end is likely to consume the data we just wrote, and * therefore has cache benefit from being placed on the same cpu, see
@@ -70,16 +58,6 @@ SCHED_FEAT(SYNC_WAKEUPS, 1)SCHED_FEAT (AFFINE_WAKEUPS ,1 )/* -* Weaken SYNC hint based on overlap - */ -SCHED_FEAT (SYNC_LESS ,1 ) - -/* -* Add SYNC hint based on overlap - */ -SCHED_FEAT (SYNC_MORE ,0 ) - -/* * Prefer to schedule the task we woke last (assuming it failed * wakeup-preemption), since its likely going to consume data we * touched, increases cache locality.
kernel/time/tick-sched.c +3 -0
@@ -262,6 +262,9 @@ void tick_nohz_stop_sched_tick(int inidle) goto end; } +if (nohz_ratelimit (cpu)) + goto end; + ts->idle_calls++;/* Read jiffies and the time when jiffies were updated last */ do {
Quoted 31 lines · Mike Galbraith
Greetings network land.
The reason for this query is that
wake_affine()fails if there is one and only one task on a runqueue to encourage tasks spreading out, which increases cpu utilization. However, for tasks which are communicating at high frequency, the cost of the resulting cache misses, should partners land in non-shared caches, is horrible to behold. My Q6600 has shared caches, which may or may not be hit IFF something perturbs the system, and bounces partner to the right core. That won't happen on a box with no shared caches of course, and even with shared caches available, the pain is highly visible in the TCP numbers below.The sync hint tells
wake_affine()that the waker is likely going to sleep soonish, so it subtracts the waker from the load imbalance calculation, allowing the partner task to be awakened affine. In the shared cache available case, that is also an enabler that the task be placed in an idle shared cache, which can increase throughput quite a bit (see .31 vs .33 AF UNIX), or may cost a bit if there is little to no execution overlap (see pipe).Now, I _could_ change
wake_affine()to globally succeed in the one task case, but am loath to do so because that very well may upset delicate load balancing apple cart. I think it's much safer to target the spot that I know hurts like hell. Thoughts?include/net/tcp.h +0
-0
include/net/tcp.h static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) · line 945
Quoted 8 lines
include/net/tcp.h +1
-1@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
I suspect this discussion is more a lkml topic but anyway...
This wake_up_interruptible_sync_poll() change might be good for loopback communications (and pleases tbench), but is it desirable for regular multi flows NIC traffic ?
Ingo probably can answer to this question, since he changed sock_def_readable() (and others) in commit 6f3d09291b498299 I suspect he missed tcp_prequeue() case, maybe not...
sched, net: socket wakeups are sync
'sync' wakeups are a hint towards the scheduler that (certain) networking related wakeups likely create coupling between tasks.
Peter Zijlstra
maintainerQuoted 1 line · Eric Dumazet
My Q6600 has shared caches
Isn't core2quad a dual die construct, having 2 cache domains?
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Eric Dumazet
My Q6600 has shared caches
Isn't core2quad a dual die construct, having 2 cache domains?
(yes.. am I using wrong terminology?)
Quoted 3 lines · Eric Dumazet
This
wake_up_interruptible_sync_poll()change might be good for loopback communications (and pleases tbench), but is it desirable for regular multi flows NIC traffic ?
If you can take high frequency cache misses, likely yes. For localhost, that's pretty much guaranteed currently (ergo the question).
-Mike
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Tue, 02 Mar 2010 12:27:53 +0100
Quoted 15 lines · Mike Galbraith
Quoted 14 lines
include/net/tcp.h +1
-1@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
...
Quoted 3 lines
Ingo probably can answer to this question, since he changed
sock_def_readable()(and others) in commit6f3d09291b498299I suspect he missedtcp_prequeue()case, maybe not...
It definitely looks like an oversight to me and we should add be using wake_up_interruptible_sync_poll() in tcp_prequeue() as above.
Ingo?
Quoted 5 lines · David Miller
It definitely looks like an oversight to me and we should add be using
wake_up_interruptible_sync_poll()intcp_prequeue()as above.Ingo?
Implied ack from network maintainer implies ship it to me.
net: add scheduler sync hint to tcp_prequeue().
Decreases the odds wakee will suffer from frequent cache misses.
LKML-Reference: <new-submission>
include/net/tcp.h +1 -1
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
From: Mike Galbraith <[email protected]>
Date: Wed, 03 Mar 2010 15:46:50 +0100
Quoted 8 lines · David Miller
Quoted 6 lines
It definitely looks like an oversight to me and we should add be using
wake_up_interruptible_sync_poll()intcp_prequeue()as above.Ingo?
Quoted 7 lines
Implied ack from network maintainer implies ship it to me.
net: add scheduler sync hint to
tcp_prequeue().Decreases the odds wakee will suffer from frequent cache misses.
Signed-off-by Mike Galbraith <[email protected]>
Applied.
Quoted 23 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Tue, 02 Mar 2010 12:27:53 +0100Quoted 15 lines
Le mardi 02 mars 2010 ? 10:41 +0100, Mike Galbraith a ?crit :
Quoted 14 lines
include/net/tcp.h +1
-1@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) tp->ucopy.memory =0 ; }else if (skb_queue_len (&tp->ucopy.prequeue) ==1 ) { -wake_up_interruptible_poll (sk->sk_sleep, +wake_up_interruptible_sync_poll (sk->sk_sleep,POLLIN |POLLRDNORM |POLLRDBAND );if (!inet_csk_ack_scheduled (sk)) inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,...
Quoted 3 lines
Ingo probably can answer to this question, since he changed
sock_def_readable()(and others) in commit6f3d09291b498299I suspect he missedtcp_prequeue()case, maybe not...
Quoted 5 lines
It definitely looks like an oversight to me and we should add be using
wake_up_interruptible_sync_poll()intcp_prequeue()as above.Ingo?
Yeah. I remember that patch, it was minimalistic, based on the workload i could trigger back then.
Thanks,
Ingo