Skip to contentSkip to conversation

sched: introduce SD_BALANCE_FORK for ht/mc/smp domains

PATCH In review Siddha, Suresh B 3 participants 7 messages Jul 26, 2007 – Jul 30, 2007

lore
Share & tools
lore

Siddha, Suresh B

author
#0 lore raw

PATCH 1/1 sched: introduce SD_BALANCE_FORK for ht/mc/smp domains

Introduce SD_BALANCE_FORK for HT/MC/SMP domains.

For HT/MC, as caches are shared, SD_BALANCE_FORK is the right thing to do. Given that NUMA domain already has this flag and the
scheduler currently doesn't have the concept of running threads belonging to a process as close as possible(i.e., forking may keep close, but periodic balance later will likely take them far away), introduce SD_BALANCE_FORK for SMP domain too.

Signed-off-by Suresh Siddha <[email protected]>

---

include/linux/topology.h +3 -0
index d0890a7..dc15a9f 100644
@@ -104,6 +104,7 @@
 	.flags			= SD_LOAD_BALANCE	\
 				| SD_BALANCE_NEWIDLE	\
 				| SD_BALANCE_EXEC	\
+				| SD_BALANCE_FORK	\
 				| SD_WAKE_AFFINE	\
 				| SD_WAKE_IDLE		\
 				| SD_SHARE_CPUPOWER,	\

@@ -135,6 +136,7 @@
 	.flags			= SD_LOAD_BALANCE	\
 				| SD_BALANCE_NEWIDLE	\
 				| SD_BALANCE_EXEC	\
+				| SD_BALANCE_FORK	\
 				| SD_WAKE_AFFINE	\
 				| SD_WAKE_IDLE		\
 				| SD_SHARE_PKG_RESOURCES\

@@ -166,6 +168,7 @@
 	.flags			= SD_LOAD_BALANCE	\
 				| SD_BALANCE_NEWIDLE	\
 				| SD_BALANCE_EXEC	\
+				| SD_BALANCE_FORK	\
 				| SD_WAKE_AFFINE	\
 				| SD_WAKE_IDLE		\
 				| BALANCE_FOR_PKG_POWER,\
#1 lore raw
Quoted 10 lines · Siddha, Suresh B

Introduce SD_BALANCE_FORK for HT/MC/SMP domains.

For HT/MC, as caches are shared, SD_BALANCE_FORK is the right thing to do. Given that NUMA domain already has this flag and the scheduler currently doesn't have the concept of running threads belonging to a process as close as possible(i.e., forking may keep close, but periodic balance later will likely take them far away), introduce SD_BALANCE_FORK for SMP domain too.

Signed-off-by Suresh Siddha <[email protected]>

i'm not opposed to this fundamentally, but it would be nice to better map the effects of this change: do you have any particular workload under which you've tested this and under which you've seen it makes a difference? I'd expect this to improve fork-intense half-idle workloads perhaps - things like a make -j3 on a 4-core CPU.

Ingo

author
#2 lore raw
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Siddha, Suresh B

Introduce SD_BALANCE_FORK for HT/MC/SMP domains.

For HT/MC, as caches are shared, SD_BALANCE_FORK is the right thing to do. Given that NUMA domain already has this flag and the scheduler currently doesn't have the concept of running threads belonging to a process as close as possible(i.e., forking may keep close, but periodic balance later will likely take them far away), introduce SD_BALANCE_FORK for SMP domain too.

Quoted 7 lines
Quoted 1 line
Signed-off-by Suresh Siddha <[email protected]>

i'm not opposed to this fundamentally, but it would be nice to better map the effects of this change: do you have any particular workload under which you've tested this and under which you've seen it makes a difference? I'd expect this to improve fork-intense half-idle workloads perhaps - things like a make -j3 on a 4-core CPU.

They might be doing more exec's and probably covered by exec balance.

There was a small pthread test case which was calculating the time to create all the threads and how much time each thread took to start running. It appeared as if the threads ran sequentially one after another on a DP system with four cores leading to this SD_BALANCE_FORK observation.

thanks,
suresh

#3 lore raw
Quoted 21 lines · Suresh B
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Siddha, Suresh B

Introduce SD_BALANCE_FORK for HT/MC/SMP domains.

For HT/MC, as caches are shared, SD_BALANCE_FORK is the right thing to do. Given that NUMA domain already has this flag and the scheduler currently doesn't have the concept of running threads belonging to a process as close as possible(i.e., forking may keep close, but periodic balance later will likely take them far away), introduce SD_BALANCE_FORK for SMP domain too.

Quoted 7 lines
Quoted 1 line
Signed-off-by Suresh Siddha <[email protected]>

i'm not opposed to this fundamentally, but it would be nice to better map the effects of this change: do you have any particular workload under which you've tested this and under which you've seen it makes a difference? I'd expect this to improve fork-intense half-idle workloads perhaps - things like a make -j3 on a 4-core CPU.

Quoted 6 lines

They might be doing more exec's and probably covered by exec balance.

There was a small pthread test case which was calculating the time to create all the threads and how much time each thread took to start running. It appeared as if the threads ran sequentially one after another on a DP system with four cores leading to this SD_BALANCE_FORK observation.

If it helps throughput in a non-trivial microbenchmark it would be helpful. I'm not against it either really, but keep in mind that it can make fork more expensive and less scalable; the reason we do it for the NUMA domain is because today we're basically screwed WRT existing working set if we have to migrage processes over nodes. It is really important to try to minimise that any way we possibly can.

When (if) we get NUMA page replication and automatic migration going, I will be looking at whether we can make NUMA migration more aggressive (and potentially remove SD_BALANCE_FORK). Not that either replication or migration help with kernel allocations, nor are they cheap, so NUMA placement will always be worth spending more cycles on to get right.

author
#4 lore raw
Quoted 24 lines · Nick Piggin
Quoted 23 lines · Suresh B
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Siddha, Suresh B

Introduce SD_BALANCE_FORK for HT/MC/SMP domains.

For HT/MC, as caches are shared, SD_BALANCE_FORK is the right thing to do. Given that NUMA domain already has this flag and the scheduler currently doesn't have the concept of running threads belonging to a process as close as possible(i.e., forking may keep close, but periodic balance later will likely take them far away), introduce SD_BALANCE_FORK for SMP domain too.

Quoted 7 lines
Quoted 1 line
Signed-off-by Suresh Siddha <[email protected]>

i'm not opposed to this fundamentally, but it would be nice to better map the effects of this change: do you have any particular workload under which you've tested this and under which you've seen it makes a difference? I'd expect this to improve fork-intense half-idle workloads perhaps - things like a make -j3 on a 4-core CPU.

They might be doing more exec's and probably covered by exec balance.

Quoted 7 lines
Quoted 4 lines

There was a small pthread test case which was calculating the time to create all the threads and how much time each thread took to start running. It appeared as if the threads ran sequentially one after another on a DP system with four cores leading to this SD_BALANCE_FORK observation.

If it helps throughput in a non-trivial microbenchmark it would be helpful.

We are planning to collect some data with this.

Quoted 6 lines

I'm not against it either really, but keep in mind that it can make fork more expensive and less scalable; the reason we do it for the NUMA domain is because today we're basically screwed WRT existing working set if we have to migrage processes over nodes. It is really important to try to minimise that any way we possibly can.

Quoted 6 lines

When (if) we get NUMA page replication and automatic migration going, I will be looking at whether we can make NUMA migration more aggressive (and potentially remove SD_BALANCE_FORK). Not that either replication or migration help with kernel allocations, nor are they cheap, so NUMA placement will always be worth spending more cycles on to get right.

I agree. Perhaps we can set it only for ht/mc domains.

thanks,
suresh

#5 lore raw
In reply to Siddha, Suresh B
Quoted 7 lines · Siddha, Suresh B

They might be doing more exec's and probably covered by exec balance.

There was a small pthread test case which was calculating the time to create all the threads and how much time each thread took to start running. It appeared as if the threads ran sequentially one after another on a DP system with four cores leading to this SD_BALANCE_FORK observation.

would be nice to dig out that testcase i suspect and quantify the benefits of your patch. Another workload which might perform better would be linpack: it benefits from fast and immediate 'spreading' of freshly forked threads.

Ingo

author
#6 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Siddha, Suresh B

They might be doing more exec's and probably covered by exec balance.

Quoted 8 lines
Quoted 5 lines

There was a small pthread test case which was calculating the time to create all the threads and how much time each thread took to start running. It appeared as if the threads ran sequentially one after another on a DP system with four cores leading to this SD_BALANCE_FORK observation.

would be nice to dig out that testcase i suspect and quantify the benefits of your patch.

That test case doesn't do much other than calculating the time taken for each thread to start running. With this balance on fork patch, that small pthread test case shows that all the threads now start almost at the same time on all cores.

Quoted 3 lines

Another workload which might perform better
would be linpack: it benefits from fast and immediate 'spreading' of freshly forked threads.

My understanding is that linkpack doesn't do fork often(as such difference might not be visible, but will take a look). We were planning to test httperf or some other workloads which probably does fork more often.

thanks,
suresh