demonstration of hang in sched domain partitioning code

Nick,

I'm running into a problem trying to use your suggested partition_sched_domains(cpumask_t *partition) routine the way I thought I was supposed to be able to use it.

If I ask to set up the same partition twice in a row having just one CPU in it (a singleton) the kernel hangs.

I don't know enough about kernel/sched.c to be able to debug this very efficiently. Any chance you could take a look?

The following patch can be used to reproduce the problem I'm seeing. I'm on an 8 CPU Altix using sn2_defconfig.

This patch applies to 2.6.23-rc8-mm2 immediately -after- the existing patch:

cpuset-remove-sched-domain-hooks-from-cpusets.patch

To see the problem, do:
1) apply the patch, build and boot it
2) mount -t cpuset cpuset /dev/cpuset
3) cd /dev/cpuset
4) echo 1 > memory_pressure_enabled
5) echo 1 > memory_pressure_enabled
6) system hung ... big red switch time ...

Well, not entirely hung - but that task, and sooner or later other tasks, hang enough so that you can't login anymore or complete a natural reboot.

P.S. - the second writing, step (6) above, apparently is not needed. I just had another boot freeze a couple seconds after the first write.

The patch 'borrows' the infrequently used top cpuset file file memory_pressure_enabled to let you directly invoke partition_sched_domains() with a singleton containing just whatever CPU number you write to it.

Here's the syslog output generated from the above (I wrote "2", not "1" as above, when I generated the following):

================================================================

Oct  5 01:38:27 margin kernel: Invoking partition_sched_domains for singleton on CPU 2
Oct  5 01:38:27 margin kernel: CPU2 attaching NULL sched-domain.
Oct  5 01:38:27 margin kernel: CPU2 attaching sched-domain:
Oct  5 01:38:27 margin kernel:  domain 0: span 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
Oct  5 01:38:27 margin kernel:   groups: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
Oct  5 01:38:27 margin kernel: Returned from partition_sched_domains.
Oct  5 01:38:30 margin kernel: Invoking partition_sched_domains for singleton on CPU 2
Oct  5 01:38:30 margin kernel: CPU2 attaching NULL sched-domain.

================================================================

---

 include/linux/sched.h |    2 ++
 kernel/cpuset.c       |   12 ++++++++++++
 kernel/sched.c        |   14 +++++++++++++-
 3 files changed, 27 insertions(+), 1 deletion(-)

@@ -718,6 +718,8 @@ struct sched_domain {
 #endif
 };
 
+extern int partition_sched_domains(cpumask_t *partition);
+
 #endif	/* CONFIG_SMP */
 
 /*

@@ -1005,10 +1005,22 @@ done:
 
 static int update_memory_pressure_enabled(struct cpuset *cs, char *buf)
 {
+#ifdef THE_USUAL_CODE
 	if (simple_strtoul(buf, NULL, 10) != 0)
 		cpuset_memory_pressure_enabled = 1;
 	else
 		cpuset_memory_pressure_enabled = 0;
+#else /* partition_sched_domains debug hook */
+	int i;
+	cpumask_t singleton;
+
+	i = simple_strtoul(buf, NULL, 10);
+	cpus_clear(singleton);
+	cpu_set(i, singleton);
+	printk("Invoking partition_sched_domains for singleton on CPU %d\n", i);
+	partition_sched_domains(&singleton);
+	printk("Returned from partition_sched_domains.\n");
+#endif
 	return 0;
 }
 

@@ -5493,7 +5493,7 @@ int __init migration_init(void)
 int nr_cpu_ids __read_mostly = NR_CPUS;
 EXPORT_SYMBOL(nr_cpu_ids);
 
-#undef SCHED_DOMAIN_DEBUG
+#define SCHED_DOMAIN_DEBUG
 #ifdef SCHED_DOMAIN_DEBUG
 static void sched_domain_debug(struct sched_domain *sd, int cpu)
 {

@@ -6327,6 +6327,18 @@ static void detach_destroy_domains(const
 	arch_destroy_sched_domains(cpu_map);
 }
 
+int partition_sched_domains(cpumask_t *partition)
+{
+	cpumask_t m;
+
+	cpus_and(m, *partition, cpu_online_map);
+	cpus_andnot(m, m, cpu_isolated_map);
+
+	/* Detach sched domains from all of the affected cpus */
+	detach_destroy_domains(&m);
+	return build_sched_domains(&m);
+}
+
 #if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
 static int arch_reinit_sched_domains(void)
 {

                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373

Hi Paul,

Yeah I'll take a look. Meanwhile, as Ingo said, the recent patch you sent is reasonable enough because the most critical thing to do is to allow users to move away from the old interface.