cpu_exclusive sched domains on partial nodes temp fix
PATCH 1/1 cpu_exclusive sched domains on partial nodes temp fix
If Dinakar, Hawkes and Nick concur (and no one else complains too loud) then the following should go into 2.6.13, to avoid the potential kernel oops that Hawkes reported in Dinakar's feature to allow user control of dynamic sched domain placement using cpu_exclusive cpusets.
This patch keeps the kernel/cpuset.c routine update_cpu_domains() from invoking the sched.c routine partition_sched_domains() if the cpuset in question doesn't fall on node boundaries.
I have boot tested this on an SN2, and with the help of a couple of ad hoc printk's, determined that it does indeed avoid calling the partition_sched_domains() routine on partial nodes.
I did not directly verify that this avoids setting up bogus sched domains or avoids the oops that Hawkes saw.
Obviously, if the above named parties decide to take some other path, then this patch should be discarded. I submit this patch under the expectation that Hawkes and others fixes to support sched domains not on node boundaries will go into *-mm and 2.6.14. Do not include the following patch in *-mm or 2.6.14 versions which have the real sched domain fixes.
This patch imposes a silent artificial constraint on which cpusets can be used to define dynamic sched domains.
This patch should allow proceeding with this new feature in 2.6.13 for the configurations in which it is useful (node alligned sched domains) while avoiding trying to setup sched domains in the less useful cases that can cause the kernel corruption and oops.
Index: linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c =================================--- linux-2.6.13-cpuset-mempolicy-migrate.orig/kernel/cpuset.c +++ linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
@@ -636,6 +636,23 @@ static void update_cpu_domains(struct cp
return;
/*
+ * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
+ * Require the 'cpu_exclusive' cpuset to include all (or none)
+ * of the CPUs on each node, or return w/o changing sched domains.
+ * Remove this hack when dynamic sched domains fixed.
+ */
+ {
+ int i, j;
+
+ for_each_cpu_mask(i, cur->cpus_allowed) {
+ for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
+ if (!cpu_isset(j, cur->cpus_allowed))
+ return;
+ }
+ }
+ }
+
+ /*
* Get all cpus from parent's cpus_allowed not part of exclusive
* children
*/
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 7 lines · Paul Jackson
(quoted diff) +6
-0/* + * Hack to avoid 2.6.13 partial node dynamic sched domain bug. + * Require the 'cpu_exclusive' cpuset to include all (or none) + * of the CPUs on each node, or return w/o changing sched domains. + * Remove this hack when dynamic sched domains fixed. + */ + {
Review
Quoted 10 lines
(quoted diff) +10
-0+ int i, j; + + for_each_cpu_mask(i, cur->cpus_allowed) { + for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) { + if (!cpu_isset(j, cur->cpus_allowed)) + return; + } + } + } +
certainly looks acceptable from a scheduler POV.
Ingo
Quoted 4 lines · Paul Jackson
If Dinakar, Hawkes and Nick concur (and no one else complains too loud) then the following should go into 2.6.13, to avoid the potential kernel oops that Hawkes reported in Dinakar's feature to allow user control of dynamic sched domain placement using
cpu_exclusivecpusets.
I agree this is the way to go for 2.6.13 before we fix things the right way for 2.6.14. Thanks for the patch Paul.
Quoted 5 lines
This patch should allow proceeding with this new feature in 2.6.13 for the configurations in which it is useful (node alligned sched domains) while avoiding trying to setup sched domains in the less useful cases that can cause the kernel corruption and oops.
Dunno if it is something in my setup (4 CPU Power5 box with NUMA enabled) but this patch causes some hard hangs when I run the attached script. The same script runs for much longer with Ingo's changes but panics as I had described earlier. I am still debugging what causes this.
-Dinakar
Quoted 4 lines
+
for_each_cpu_mask(i, cur->cpus_allowed){
+for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))){
+ if (!cpu_isset(j,cur->cpus_allowed))
+ return;
Looks good to me.
Feel free to add:
John Hawkes