Skip to contentSkip to conversation

should use scheduler sync hint in tcp_prequeue()?

PATCH Applied Mike Galbraith 5 participants 9 messages Mar 2, 2010 – Mar 4, 2010

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 should use scheduler sync hint in tcp_prequeue()?

Greetings network land.

The reason for this query is that wake_affine() fails if there is one and only one task on a runqueue to encourage tasks spreading out, which increases cpu utilization. However, for tasks which are communicating at high frequency, the cost of the resulting cache misses, should partners land in non-shared caches, is horrible to behold. My Q6600 has shared caches, which may or may not be hit IFF something perturbs the system, and bounces partner to the right core. That won't happen on a box with no shared caches of course, and even with shared caches available, the pain is highly visible in the TCP numbers below.

The sync hint tells wake_affine() that the waker is likely going to sleep soonish, so it subtracts the waker from the load imbalance calculation, allowing the partner task to be awakened affine. In the shared cache available case, that is also an enabler that the task be placed in an idle shared cache, which can increase throughput quite a bit (see .31 vs .33 AF UNIX), or may cost a bit if there is little to no execution overlap (see pipe).

Now, I _could_ change wake_affine() to globally succeed in the one task case, but am loath to do so because that very well may upset delicate load balancing apple cart. I think it's much safer to target the spot that I know hurts like hell. Thoughts?

include/net/tcp.h +1 -1
index 34f5cc2..ba3fc64 100644
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,

That was the RFC bit, now moving along to the RFT. Busy folks can fast forward to relevant numbers, or just poke 'D' :)

Below the lmbench numbers is a diff from my experimental fast-path cycle recovery tree against .33. That's me hoping that some folks with some time on their hands to test the high speed low drag things that network folks do, and who wouldn't mind seeing such things go a bit faster, will become curious and give it a go.

Should anyone do any testing, I'd be particularly interested in seeing _negative_ results, and a way to reproduce them if it can be done on localhost + generic Q6600 box. No high speed low drag hardware here unfortunately. Any results highly welcome, food for thought comes in many flavors.

(offline preferred... may be considered spam)

Oh, almost forgot. Second set of three runs for 33-smpx is with SD_SHARE_PKG_RESOURCES disabled on MC domain to disallow use of shared cache and compare cost/benefit. TCP numbers is what's relevant to $subject.

*Local* Communication latencies in microseconds - smaller is better
---------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----

marge     2.6.31.12-smp 0.730 2.845 4.85 6.463  11.3  26.2  14.9  31.
marge     2.6.31.12-smp 0.750 2.864 4.78 6.460  11.2  22.9  14.6  31.
marge     2.6.31.12-smp 0.710 2.835 4.81 6.478  11.5  11.0  14.5  30.
marge        2.6.33-smp 1.320 4.552 5.02 9.169  12.5  26.5  15.4  18.
marge        2.6.33-smp 1.450 4.621 5.45 9.286  12.5  11.4  15.4  18.
marge        2.6.33-smp 1.450 4.589 5.53 9.168  12.6  27.5  15.4  18.
marge       2.6.33-smpx 1.160 3.565 5.97 7.513  11.3 9.776  13.9  18.
marge       2.6.33-smpx 1.140 3.569 6.02 7.479  11.2 9.849  14.0  18.
marge       2.6.33-smpx 1.090 3.563 6.39 7.450  11.2 9.785  14.0  18.
marge       2.6.33-smpx 0.730 2.665 4.85 6.565  11.9  10.3  15.2  31.
marge       2.6.33-smpx 0.740 2.701 4.03 6.573  11.7  10.3  15.4  31.
marge       2.6.33-smpx 0.710 2.753 4.86 6.533  11.7  10.3  15.3  31.

File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------

Host                 OS   0K File      10K File     Mmap    Prot   Page   100fd
                        Create Delete Create Delete Latency Fault  Fault  selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
marge     2.6.31.12-smp                               798.0 0.503 0.96940 1.434
marge     2.6.31.12-smp                               795.0 0.523 0.96630 1.435
marge     2.6.31.12-smp                               791.0 0.536 0.97410 1.438
marge        2.6.33-smp                               841.0 0.603 1.00320 1.492
marge        2.6.33-smp                               818.0 0.582 0.99960 1.496
marge        2.6.33-smp                               814.0 0.578 1.00750 1.492
marge       2.6.33-smpx                               782.0 0.486 0.97290 1.489
marge       2.6.33-smpx                               787.0 0.491 0.97950 1.498
marge       2.6.33-smpx                               801.0 0.469 0.97690 1.492
marge       2.6.33-smpx                               783.0 0.496 0.97200 1.497
marge       2.6.33-smpx                               809.0 0.505 0.98070 1.504
marge       2.6.33-smpx                               796.0 0.501 0.98820 1.502

*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----

marge     2.6.31.12-smp 2821 2971 762. 2829.2 4799.0 1243.0 1230.3 4469 1682.
marge     2.6.31.12-smp 2824 2931 760. 2833.3 4736.5 1239.5 1235.8 4462 1678.
marge     2.6.31.12-smp 2796 2936 1139 2843.3 4815.7 1242.8 1234.6 4471 1685.
marge        2.6.33-smp 2670 5151 739. 2816.6 4768.5 1243.7 1237.2 4389 1684.
marge        2.6.33-smp 2627 5126 1135 2806.9 4783.1 1245.1 1236.1 4413 1684.
marge        2.6.33-smp 2582 5037 1137 2799.6 4755.4 1242.0 1239.1 4471 1683.
marge       2.6.33-smpx 2848 5184 2972 2820.5 4804.8 1242.6 1236.9 4315 1686.
marge       2.6.33-smpx 2804 5183 2934 2822.8 4759.3 1245.0 1234.7 4462 1688.
marge       2.6.33-smpx 2729 5177 2920 2837.6 4820.0 1246.9 1238.5 4467 1684.
marge       2.6.33-smpx 2843 2896 1928 2786.5 4751.2 1242.2 1238.6 4493 1682.
marge       2.6.33-smpx 2869 2886 1936 2841.4 4748.9 1244.3 1237.7 4456 1683.
marge       2.6.33-smpx 2845 2895 1947 2836.0 4813.6 1242.7 1236.3 4473 1674.
include/linux/sched.h +4 -6
index 78efe7c..ebbfcba 100644
@@ -278,6 +278,10 @@ static inline int select_nohz_load_balancer(int cpu)
 }
 #endif
 
+#ifdef CONFIG_NO_HZ
+extern int nohz_ratelimit(int cpu);
+#endif
+
 /*
  * Only dump TASK_* tasks. (0 for all tasks)
  */
@@ -1160,14 +1164,8 @@ struct sched_entity {
 	u64			vruntime;
 	u64			prev_sum_exec_runtime;
 
-	u64			last_wakeup;
-	u64			avg_overlap;
-
 	u64			nr_migrations;
 
-	u64			start_runtime;
-	u64			avg_wakeup;
-
 #ifdef CONFIG_SCHEDSTATS
 	u64			wait_start;
 	u64			wait_max;
include/net/tcp.h +1 -1
index 34f5cc2..ba3fc64 100644
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
kernel/sched.c +28 -67

229 lines not on this page · raw message

kernel/sched_debug.c +0 -2
index 67f95aa..e91311d 100644
@@ -407,8 +407,6 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)
 	PN(se.exec_start);
 	PN(se.vruntime);
 	PN(se.sum_exec_runtime);
-	PN(se.avg_overlap);
-	PN(se.avg_wakeup);
 
 	nr_switches = p->nvcsw + p->nivcsw;
 
kernel/sched_fair.c +10 -51

103 lines not on this page · raw message

kernel/sched_features.h +0 -22
index d5059fd..c545e04 100644
@@ -31,12 +31,6 @@ SCHED_FEAT(START_DEBIT, 1)
 SCHED_FEAT(WAKEUP_PREEMPT, 1)
 
 /*
- * Compute wakeup_gran based on task behaviour, clipped to
- *  [0, sched_wakeup_gran_ns]
- */
-SCHED_FEAT(ADAPTIVE_GRAN, 1)
-
-/*
  * When converting the wakeup granularity to virtual time, do it such
  * that heavier tasks preempting a lighter task have an edge.
  */
@@ -48,12 +42,6 @@ SCHED_FEAT(ASYM_GRAN, 1)
 SCHED_FEAT(WAKEUP_SYNC, 0)
 
 /*
- * Wakeup preempt based on task behaviour. Tasks that do not overlap
- * don't get preempted.
- */
-SCHED_FEAT(WAKEUP_OVERLAP, 0)
-
-/*
  * Use the SYNC wakeup hint, pipes and the likes use this to indicate
  * the remote end is likely to consume the data we just wrote, and
  * therefore has cache benefit from being placed on the same cpu, see
@@ -70,16 +58,6 @@ SCHED_FEAT(SYNC_WAKEUPS, 1)
 SCHED_FEAT(AFFINE_WAKEUPS, 1)
 
 /*
- * Weaken SYNC hint based on overlap
- */
-SCHED_FEAT(SYNC_LESS, 1)
-
-/*
- * Add SYNC hint based on overlap
- */
-SCHED_FEAT(SYNC_MORE, 0)
-
-/*
  * Prefer to schedule the task we woke last (assuming it failed
  * wakeup-preemption), since its likely going to consume data we
  * touched, increases cache locality.
kernel/time/tick-sched.c +3 -0
index f992762..f25735a 100644
@@ -262,6 +262,9 @@ void tick_nohz_stop_sched_tick(int inidle)
 		goto end;
 	}
 
+	if (nohz_ratelimit(cpu))
+		goto end;
+
 	ts->idle_calls++;
 	/* Read jiffies and the time when jiffies were updated last */
 	do {
#1 lore raw
Quoted 31 lines · Mike Galbraith

Greetings network land.

The reason for this query is that wake_affine() fails if there is one and only one task on a runqueue to encourage tasks spreading out, which increases cpu utilization. However, for tasks which are communicating at high frequency, the cost of the resulting cache misses, should partners land in non-shared caches, is horrible to behold. My Q6600 has shared caches, which may or may not be hit IFF something perturbs the system, and bounces partner to the right core. That won't happen on a box with no shared caches of course, and even with shared caches available, the pain is highly visible in the TCP numbers below.

The sync hint tells wake_affine() that the waker is likely going to sleep soonish, so it subtracts the waker from the load imbalance calculation, allowing the partner task to be awakened affine. In the shared cache available case, that is also an enabler that the task be placed in an idle shared cache, which can increase throughput quite a bit (see .31 vs .33 AF UNIX), or may cost a bit if there is little to no execution overlap (see pipe).

Now, I _could_ change wake_affine() to globally succeed in the one task case, but am loath to do so because that very well may upset delicate load balancing apple cart. I think it's much safer to target the spot that I know hurts like hell. Thoughts?

include/net/tcp.h +0 -0
index 34f5cc2..ba3fc64 100644

include/net/tcp.h static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb) · line 945

Quoted 8 lines
include/net/tcp.h +1 -1
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,

I suspect this discussion is more a lkml topic but anyway...

This wake_up_interruptible_sync_poll() change might be good for loopback communications (and pleases tbench), but is it desirable for regular multi flows NIC traffic ?

Ingo probably can answer to this question, since he changed sock_def_readable() (and others) in commit 6f3d09291b498299 I suspect he missed tcp_prequeue() case, maybe not...

sched, net: socket wakeups are sync

'sync' wakeups are a hint towards the scheduler that (certain) networking related wakeups likely create coupling between tasks.

maintainer
#2 lore raw
Quoted 1 line · Eric Dumazet

My Q6600 has shared caches

Isn't core2quad a dual die construct, having 2 cache domains?

#3 lore raw
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Eric Dumazet

My Q6600 has shared caches

Isn't core2quad a dual die construct, having 2 cache domains?

(yes.. am I using wrong terminology?)

#4 lore raw
In reply to Eric Dumazet
Quoted 3 lines · Eric Dumazet

This wake_up_interruptible_sync_poll() change might be good for loopback communications (and pleases tbench), but is it desirable for regular multi flows NIC traffic ?

If you can take high frequency cache misses, likely yes. For localhost, that's pretty much guaranteed currently (ergo the question).

-Mike

maintainer
#5 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Tue, 02 Mar 2010 12:27:53 +0100

Quoted 15 lines · Mike Galbraith
Quoted 14 lines
include/net/tcp.h +1 -1
index 34f5cc2..ba3fc64 100644
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,

...

Quoted 3 lines

Ingo probably can answer to this question, since he changed sock_def_readable() (and others) in commit 6f3d09291b498299 I suspect he missed tcp_prequeue() case, maybe not...

It definitely looks like an oversight to me and we should add be using wake_up_interruptible_sync_poll() in tcp_prequeue() as above.

Ingo?

#6 lore raw
Quoted 5 lines · David Miller

It definitely looks like an oversight to me and we should add be using wake_up_interruptible_sync_poll() in tcp_prequeue() as above.

Ingo?

Implied ack from network maintainer implies ship it to me.

net: add scheduler sync hint to tcp_prequeue().

Decreases the odds wakee will suffer from frequent cache misses.

LKML-Reference: <new-submission>

include/net/tcp.h +1 -1
index 34f5cc2..ba3fc64 100644
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
maintainer Applied
#7 lore raw

From: Mike Galbraith <[email protected]>
Date: Wed, 03 Mar 2010 15:46:50 +0100

Quoted 8 lines · David Miller
Quoted 6 lines

It definitely looks like an oversight to me and we should add be using wake_up_interruptible_sync_poll() in tcp_prequeue() as above.

Ingo?

Quoted 7 lines

Implied ack from network maintainer implies ship it to me.

net: add scheduler sync hint to tcp_prequeue().

Decreases the odds wakee will suffer from frequent cache misses.

Applied.

#8 lore raw
In reply to David Miller
Quoted 23 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Tue, 02 Mar 2010 12:27:53 +0100

Quoted 15 lines

Le mardi 02 mars 2010 ? 10:41 +0100, Mike Galbraith a ?crit :

Quoted 14 lines
include/net/tcp.h +1 -1
index 34f5cc2..ba3fc64 100644
@@ -939,7 +939,7 @@ static inline int tcp_prequeue(struct sock *sk, struct sk_buff *skb)
 
 		tp->ucopy.memory = 0;
 	} else if (skb_queue_len(&tp->ucopy.prequeue) == 1) {
-		wake_up_interruptible_poll(sk->sk_sleep,
+		wake_up_interruptible_sync_poll(sk->sk_sleep,
 					   POLLIN | POLLRDNORM | POLLRDBAND);
 		if (!inet_csk_ack_scheduled(sk))
 			inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,

...

Quoted 3 lines

Ingo probably can answer to this question, since he changed sock_def_readable() (and others) in commit 6f3d09291b498299 I suspect he missed tcp_prequeue() case, maybe not...

Quoted 5 lines

It definitely looks like an oversight to me and we should add be using wake_up_interruptible_sync_poll() in tcp_prequeue() as above.

Ingo?

Yeah. I remember that patch, it was minimalistic, based on the workload i could trigger back then.

Thanks,

Ingo