Skip to contentSkip to conversation

cpu_exclusive sched domains on partial nodes temp fix

PATCH In review Paul Jackson 4 participants 4 messages Aug 23, 2005 – Aug 24, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpu_exclusive sched domains on partial nodes temp fix

If Dinakar, Hawkes and Nick concur (and no one else complains too loud) then the following should go into 2.6.13, to avoid the potential kernel oops that Hawkes reported in Dinakar's feature to allow user control of dynamic sched domain placement using cpu_exclusive cpusets.

This patch keeps the kernel/cpuset.c routine update_cpu_domains() from invoking the sched.c routine partition_sched_domains() if the cpuset in question doesn't fall on node boundaries.

I have boot tested this on an SN2, and with the help of a couple of ad hoc printk's, determined that it does indeed avoid calling the partition_sched_domains() routine on partial nodes.

I did not directly verify that this avoids setting up bogus sched domains or avoids the oops that Hawkes saw.

Obviously, if the above named parties decide to take some other path, then this patch should be discarded. I submit this patch under the expectation that Hawkes and others fixes to support sched domains not on node boundaries will go into *-mm and 2.6.14. Do not include the following patch in *-mm or 2.6.14 versions which have the real sched domain fixes.

This patch imposes a silent artificial constraint on which cpusets can be used to define dynamic sched domains.

This patch should allow proceeding with this new feature in 2.6.13 for the configurations in which it is useful (node alligned sched domains) while avoiding trying to setup sched domains in the less useful cases that can cause the kernel corruption and oops.

Index: linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c =================================--- linux-2.6.13-cpuset-mempolicy-migrate.orig/kernel/cpuset.c +++ linux-2.6.13-cpuset-mempolicy-migrate/kernel/cpuset.c
@@ -636,6 +636,23 @@ static void update_cpu_domains(struct cp
return;

/*
+ * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
+ * Require the 'cpu_exclusive' cpuset to include all (or none)
+ * of the CPUs on each node, or return w/o changing sched domains.
+ * Remove this hack when dynamic sched domains fixed.
+ */
+ {
+ int i, j;
+
+ for_each_cpu_mask(i, cur->cpus_allowed) {
+ for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
+ if (!cpu_isset(j, cur->cpus_allowed))
+ return;
+ }
+ }
+ }
+
+ /*
* Get all cpus from parent's cpus_allowed not part of exclusive
* children
*/

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
Acked-by on v1 1/1
#1 lore raw
Quoted 7 lines · Paul Jackson
(quoted diff) +6 -0
 	/*
+	 * Hack to avoid 2.6.13 partial node dynamic sched domain bug.
+	 * Require the 'cpu_exclusive' cpuset to include all (or none)
+	 * of the CPUs on each node, or return w/o changing sched domains.
+	 * Remove this hack when dynamic sched domains fixed.
+	 */
+	{

Review

Quoted 10 lines
(quoted diff) +10 -0
+		int i, j;
+
+		for_each_cpu_mask(i, cur->cpus_allowed) {
+			for_each_cpu_mask(j, node_to_cpumask(cpu_to_node(i))) {
+				if (!cpu_isset(j, cur->cpus_allowed))
+					return;
+			}
+		}
+	}
+

certainly looks acceptable from a scheduler POV.

Ingo

#2 lore raw
In reply to Paul Jackson
Quoted 4 lines · Paul Jackson

If Dinakar, Hawkes and Nick concur (and no one else complains too loud) then the following should go into 2.6.13, to avoid the potential kernel oops that Hawkes reported in Dinakar's feature to allow user control of dynamic sched domain placement using cpu_exclusive cpusets.

I agree this is the way to go for 2.6.13 before we fix things the right way for 2.6.14. Thanks for the patch Paul.

Quoted 5 lines

This patch should allow proceeding with this new feature in 2.6.13 for the configurations in which it is useful (node alligned sched domains) while avoiding trying to setup sched domains in the less useful cases that can cause the kernel corruption and oops.

Dunno if it is something in my setup (4 CPU Power5 box with NUMA enabled) but this patch causes some hard hangs when I run the attached script. The same script runs for much longer with Ingo's changes but panics as I had described earlier. I am still debugging what causes this.

-Dinakar