stopmachine: add stopmachine_timeout

From: Rusty Russell <[email protected]>
To: [email protected]
Subject: [PATCH 0/3] stop_machine enhancements and simplifications
Date: Tue, 8 Jul 2008 17:50:55 +1000

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

I took your version and modified it slightly. What do you think?

(BTW, the "warning: passing argument 1 of 'kthread_create' from incompatible pointer type" is caused by a kernel without the typesafe patches: this will be fixed soon, do I've removed your fix for that).

Thanks,
Rusty.

===
stopmachine: add stopmachine_timeout

This patch is based on Hidetoshi Seto's patch to make stop_machine() more robust in the face of wedged CPUs.

This version does not fail unless the stuck CPU is one we wanted to do something on, and does not fail all future stop_machines.

Signed-off-by Rusty Russell <[email protected]>

---
 kernel/stop_machine.c |   52 +++++++++++++++++++++++++++++++++++++++++++++---
 kernel/sysctl.c       |   15 ++++++++++++++
 2 files changed, 63 insertions(+), 4 deletions(-)

kernel/stop_machine.c +11 -1

@@ -35,10 +35,14 @@ struct stop_machine_data {
 };
 
 /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
-static unsigned int num_threads;
+static unsigned num_threads;
 static atomic_t thread_ack;
+static cpumask_t prepared_cpus;
 static struct completion finished;
 static DEFINE_MUTEX(lock);
+
+static unsigned long limit;
+unsigned long stopmachine_timeout = 5; /* secs, arbitrary */
 
 static void set_state(enum stopmachine_state newstate)
 {

@@ -66,6 +70,10 @@ static int stop_cpu(struct stop_machine_
 {
 	enum stopmachine_state curstate = STOPMACHINE_NONE;
 	int uninitialized_var(ret);
+
+	/* If we've been shoved off the normal CPU, abort. */
+	if (cpu_test_and_set(smp_processor_id(), prepared_cpus))
+		do_exit(0);
 
 	/* Simple state machine */
 	do {

@@ -100,6 +108,44 @@ static int chill(void *unused)
 	return 0;
 }
 
+static bool fixup_timeout(struct task_struct **threads, const cpumask_t *cpus)
+{

kernel/stop_machine.c +20 -0

+	}
+
+	return stagger_onwards;
+}
+
 int __stop_machine(int (*fn)(void *), void *data, const cpumask_t *cpus)
 {
 	int i, err;

@@ -121,6 +167,7 @@ int __stop_machine(int (*fn)(void *), vo
 	mutex_lock(&lock);
 	init_completion(&finished);
 	num_threads = num_online_cpus();
+	limit = jiffies + msecs_to_jiffies(stopmachine_timeout * MSEC_PER_SEC);
 	set_state(STOPMACHINE_PREPARE);
 
 	for_each_online_cpu(i) {

@@ -152,9 +199,23 @@ int __stop_machine(int (*fn)(void *), vo
 
 	/* We've created all the threads.  Wake them all: hold this CPU so one
 	 * doesn't hit this CPU until we're ready. */
+	cpus_clear(prepared_cpus);
 	get_cpu();
 	for_each_online_cpu(i)
 		wake_up_process(threads[i]);
+
+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit)) {
+			if (!fixup_timeout(threads, cpus)) {
+				/* Tell them all to exit. */
+				set_state(STOPMACHINE_EXIT);
+				active.fnret = -EIO;
+			}
+			break;
+		}
+		cpu_relax();
+	}
 
 	/* This will release the thread on our CPU. */
 	put_cpu();

kernel/sysctl.c +15 -1

@@ -144,6 +144,10 @@ extern int no_unaligned_warning;
 
 #ifdef CONFIG_RT_MUTEXES
 extern int max_lock_depth;
+#endif
+
+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
 #endif
 
 #ifdef CONFIG_PROC_SYSCTL

@@ -811,6 +815,17 @@ static struct ctl_table kern_table[] = {
 		.procname	= "keys",
 		.mode		= 0555,
 		.child		= key_sysctls,
+	},
+#endif
+#ifdef CONFIG_STOP_MACHINE
+	{
+		.ctl_name       = CTL_UNNUMBERED,
+		.procname       = "stopmachine_timeout",
+		.data           = &stopmachine_timeout,
+		.maxlen         = sizeof(unsigned long),
+		.mode           = 0644,
+		.proc_handler   = &proc_doulongvec_minmax,
+		.strategy       = &sysctl_intvec,
 	},
 #endif
 /*
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi Rusty,

However we need to be careful that the stuck CPU can restart unexpectedly.

kernel/stop_machine.c static int chill(void *unused) · line 122

Quoted 4 lines

kernel/stop_machine.c +4 -0

+	for_each_online_cpu(i) {
+		if (!cpu_isset(i, prepared_cpus) && i != smp_processor_id()) {
+			bool ignore;
+

Note that here is a window that a not-prepared frozen cpu can be thawed and become be prepared.

kernel/stop_machine.c static int chill(void *unused) · line 131

Quoted 9 lines

kernel/stop_machine.c +9 -0

+			/* If we wanted to run on a particular CPU, and that's
+			 * the one which is stuck, it's a real failure. */
+			ignore = !cpus || !cpu_isset(i, *cpus);
+			printk(KERN_CRIT "stopmachine: cpu#%d seems to be "
+			       "stuck, %s.\n",
+			       i, ignore ? "ignoring" : "FAILING");
+			/* Unbind thread: it will exit when it sees
+			 * that prepared_cpus bit set. */
+			set_cpus_allowed(threads[i], cpu_online_map);

Unbinded threads still can wake up on a cpu where they originally targeted.

Quoted 1 line · On Tuesday 15 July 2008 11:11:34 Hidetoshi Seto wrote:

However we need to be careful that the stuck CPU can restart unexpectedly.

OK, if you are worried about that race, I think we can still fix it...

Hmm, there's still the vague possibility that the thread doesn't schedule until we start a new stop_machine (and clear prepared_cpus). We could simply loop in the main thread if any threads are alive, before freeing them (inside the lock). A counter and notifier is the other way, but it seems like overkill for a very unlikely event.

Hi Rusty,

Quoted 4 lines · Rusty Russell

Quoted 2 lines · On Tuesday 15 July 2008 11:11:34 Hidetoshi Seto wrote:

However we need to be careful that the stuck CPU can restart unexpectedly.

Quoted 1 line

OK, if you are worried about that race, I think we can still fix it...

After having a relaxing day, once I said:
"I like your idea that if we did not want to do something on the stuck CPU
then treat the CPU as stopped."
but now I noticed that the stuck CPU can harm what we want to do if it is not real stuck... ex. busy loop in a subsystem, and we want to touch the core of the subsystem exclusively.

The scenario we are addressing is a stuck CPU and module load. If we fail stop machine, module load fails.

That is why we should continue if we can. It is also why the default timeout cannot be 0. You can't turn this on once you notice there's a problem: it's too late.

If we don't want to handle this case, let's not apply any patch at all. Rusty.

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 7 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 7 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Quoted 15 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Quoted 4 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Quoted 15 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Quoted 4 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Quoted 15 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Quoted 6 lines

Quoted 3 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Hm, yeah. But in this case, it's tricky. CPU time is an inherently per-cpu quantity. If cpu A is waiting for cpu B, and wants to do the timeout in cpu-seconds, then it has to be in *B*s cpu-seconds (and if A is waiting on B,C,D,E,F... it needs to measure separate timeouts with separate timebases for each other CPU). It also means that if B is unresponsive but also not consuming any time (blocked in IO, administratively paused, etc), then the timeout will never trigger.

So I think monotonic wallclock time actually makes the most sense here.

The other issue is whether cpu_relax() is the right thing to put in the busywait. We don't hook it in pvops, so it's just an x86 "pause" instruction, so from the hypervisor's perspective it just looks like a spinning CPU. We could either hook cpu_relax() into a hypervisor yield, or come up with a heavier-weight cpu_snooze() (cpu_relax() is often used in loops which are expected to have a short duration, where doing a hypercall+yield would be overkill).

Quoted 15 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Quoted 6 lines

Quoted 3 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Hm, yeah. But in this case, it's tricky. CPU time is an inherently per-cpu quantity. If cpu A is waiting for cpu B, and wants to do the timeout in cpu-seconds, then it has to be in *B*s cpu-seconds (and if A is waiting on B,C,D,E,F... it needs to measure separate timeouts with separate timebases for each other CPU). It also means that if B is unresponsive but also not consuming any time (blocked in IO, administratively paused, etc), then the timeout will never trigger.

So I think monotonic wallclock time actually makes the most sense here.

The other issue is whether cpu_relax() is the right thing to put in the busywait. We don't hook it in pvops, so it's just an x86 "pause" instruction, so from the hypervisor's perspective it just looks like a spinning CPU. We could either hook cpu_relax() into a hypervisor yield, or come up with a heavier-weight cpu_snooze() (cpu_relax() is often used in loops which are expected to have a short duration, where doing a hypercall+yield would be overkill).

Quoted 17 lines · Jeremy Fitzhardinge

Quoted 16 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 7 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

cpu_relax() translates to a hypervisor yield on s390. Probably makes sense if other architectures would do the same.

Quoted 17 lines · Jeremy Fitzhardinge

Quoted 16 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 7 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

cpu_relax() translates to a hypervisor yield on s390. Probably makes sense if other architectures would do the same.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 16 lines · Jeremy Fitzhardinge

Quoted 15 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

Quoted 13 lines

Quoted 5 lines

Quoted 3 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Hm, yeah. But in this case, it's tricky. CPU time is an inherently per-cpu quantity. If cpu A is waiting for cpu B, and wants to do the timeout in cpu-seconds, then it has to be in *B*s cpu-seconds (and if A is waiting on B,C,D,E,F... it needs to measure separate timeouts with separate timebases for each other CPU). It also means that if B is unresponsive but also not consuming any time (blocked in IO, administratively paused, etc), then the timeout will never trigger.

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 16 lines · Jeremy Fitzhardinge

Quoted 15 lines · Rusty Russell

Quoted 14 lines · On Monday 14 July 2008 21:51:25 Christian Borntraeger wrote:

Am Montag, 14. Juli 2008 schrieb Hidetoshi Seto:

kernel/stop_machine.c int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus) · line 183

Quoted 8 lines

kernel/stop_machine.c +7 -0

+	/* Wait all others come to life */
+	while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+		if (time_is_before_jiffies(limit))
+			goto timeout;
+		cpu_relax();
+	}
+

Hmm. I think this could become interesting on virtual machines. The hypervisor might be to busy to schedule a specific cpu at certain load scenarios. This would cause a failure even if the cpu is not really locked up. We had similar problems with the soft lockup daemon on s390.

Quoted 2 lines

Quoted 2 lines

5 seconds is a fairly long time. If all else fails we could have a config option to simply disable this code.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

Quoted 13 lines

Quoted 5 lines

Quoted 3 lines

It would be good to not-use wall-clock time, but really used cpu time instead. Unfortunately I have no idea, if that is possible in a generic way. Heiko, any ideas?

Ah, cpu time comes up again. Perhaps we should actually dig that up again; Zach and Jeremy CC'd.

Hm, yeah. But in this case, it's tricky. CPU time is an inherently per-cpu quantity. If cpu A is waiting for cpu B, and wants to do the timeout in cpu-seconds, then it has to be in *B*s cpu-seconds (and if A is waiting on B,C,D,E,F... it needs to measure separate timeouts with separate timebases for each other CPU). It also means that if B is unresponsive but also not consuming any time (blocked in IO, administratively paused, etc), then the timeout will never trigger.

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 5 lines

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Quoted 5 lines

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

Quoted 6 lines · On Tuesday 15 July 2008 12:24:54 Max Krasnyansky wrote:

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 3 lines

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Indeed, this was my thought too. s390 can initialize it to zero somewhere in their boot code.

Quoted 6 lines · On Tuesday 15 July 2008 12:24:54 Max Krasnyansky wrote:

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 3 lines

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Indeed, this was my thought too. s390 can initialize it to zero somewhere in their boot code.

Quoted 8 lines · On Tuesday 15 July 2008 12:24:54 Max Krasnyansky wrote:

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Indeed, this was my thought too. s390 can initialize it to zero somewhere in their boot code.

Quoted 7 lines

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

We can make it ms, sure. 200ms should be plenty of time: worst I ever saw was 150ms, and that was some weird Power box doing crazy stuff. I wouldn't be surprised if you'd never see 1ms on your hardware.

The ipi idea would handle your case a little more nicely, too, but that's probably not going to hit this merge window.

Quoted 8 lines · On Tuesday 15 July 2008 12:24:54 Max Krasnyansky wrote:

Quoted 4 lines · Heiko Carstens

On Mon, Jul 14, 2008 at 11:56:18AM -0700, Jeremy Fitzhardinge wrote:
This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

How about this. We'll make this a sysctl, as Rusty already did, and set the default to 0 which means "never timeout". That way crazy people like me who care about this scenario can enable this feature.

Indeed, this was my thought too. s390 can initialize it to zero somewhere in their boot code.

Quoted 7 lines

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

We can make it ms, sure. 200ms should be plenty of time: worst I ever saw was 150ms, and that was some weird Power box doing crazy stuff. I wouldn't be surprised if you'd never see 1ms on your hardware.

The ipi idea would handle your case a little more nicely, too, but that's probably not going to hit this merge window.

Quoted 7 lines

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

We can make it ms, sure. 200ms should be plenty of time: worst I ever saw

150ms, and that was some weird Power box doing crazy stuff. I wouldn't be surprised if you'd never see 1ms on your hardware.

Quoted 7 lines

btw Rusty, I just had this "why didn't I think of that" moments. This is actually another way of handling my workload. I mean it certainly does not fix the root case of the problems and we still need other things that we talked about (non-blocking module delete, lock-free module insertion, etc) but at least in the mean time it avoids wedging the machines for good. btw I'd like that timeout in milliseconds. I think 5 seconds is way tooooo long :).

We can make it ms, sure. 200ms should be plenty of time: worst I ever saw

150ms, and that was some weird Power box doing crazy stuff. I wouldn't be surprised if you'd never see 1ms on your hardware.

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

Quoted 1 line

So I think monotonic wallclock time actually makes the most sense here.

This is asking for trouble... a config option to disable this would be nice. But as I don't know which problem this patch originally addresses it might be that this is needed anyway. So lets see why we need it first.

Quoted 3 lines · Heiko Carstens

Hmm.. probably a stupid question: but what could happen that a real cpu (not virtual) becomes unresponsive so that it won't schedule a MAX_RT_PRIO-1 prioritized task for 5 seconds?

The original problem (once I heard and easily reproduced) was there was an another MAX_RT_PRIO-1 task and the task was spinning in itself by a bug. (Now this would not be a problem since RLIMIT_RTTIME will work for it, but I cannot deny that there are some situations which cannot set the limit.)

Thank you for useful feedbacks!
Here is the updated version.
Could you put this on top of your patches, Rusty?

Thanks,
H.Seto

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu.

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs, and set default to 200 msec (since v1's arbitrary 5 sec is too long)
- allow disabling timeout by setting the stopmachine_timeout to 0

Thank you for useful feedbacks!
Here is the updated version.
Could you put this on top of your patches, Rusty?

Thanks,
H.Seto

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu.

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs, and set default to 200 msec (since v1's arbitrary 5 sec is too long)
- allow disabling timeout by setting the stopmachine_timeout to 0

I'd set the default to zero. I beleive that's what Heiko suggested too.

I'd set the default to zero. I beleive that's what Heiko suggested too.

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu in time. You can enable this timeout via sysctl.

v3:
- set stopmachine_timeout default to 0 (= never timeout)

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs
- allow disabling timeout by setting the stopmachine_timeout to 0

Signed-off-by Hidetoshi Seto <[email protected]>

---
 kernel/stop_machine.c |   54 ++++++++++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c       |   15 +++++++++++++
 2 files changed, 66 insertions(+), 3 deletions(-)

kernel/stop_machine.c +51 -3

index 5b72c2b..77b7944 100644
@@ -35,15 +35,18 @@ struct stop_machine_data {
 };
 
 /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
-static unsigned int num_threads;
+static atomic_t num_threads;
 static atomic_t thread_ack;
+static cpumask_t prepared_cpus;
 static struct completion finished;
 static DEFINE_MUTEX(lock);
 
+unsigned long stopmachine_timeout = 0; /* msecs, 0 = "never timeout" */
+
 static void set_state(enum stopmachine_state newstate)
 {
 	/* Reset ack counter. */
-	atomic_set(&thread_ack, num_threads);
+	atomic_set(&thread_ack, atomic_read(&num_threads));
 	smp_wmb();
 	state = newstate;
 }

@@ -67,6 +70,8 @@ static int stop_cpu(struct stop_machine_data *smdata)
 	enum stopmachine_state curstate = STOPMACHINE_NONE;
 	int uninitialized_var(ret);
 
+	cpu_set(smp_processor_id(), prepared_cpus);
+
 	/* Simple state machine */
 	do {
 		/* Chill out and ensure we re-read stopmachine_state. */

@@ -90,6 +95,7 @@ static int stop_cpu(struct stop_machine_data *smdata)
 		}
 	} while (curstate != STOPMACHINE_EXIT);
 
+	atomic_dec(&num_threads);
 	local_irq_enable();
 	do_exit(0);
 }

@@ -105,6 +111,15 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 	int i, err;
 	struct stop_machine_data active, idle;
 	struct task_struct **threads;
+	unsigned long limit;
+
+	if (atomic_read(&num_threads)) {
+		/*
+		 * previous stop_machine was timeout, and still there are some
+		 * unfinished thread (dangling stucked CPU?).
+		 */
+		return -EBUSY;
+	}
 
 	active.fn = fn;
 	active.data = data;

@@ -120,7 +135,7 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 	/* Set up initial state. */
 	mutex_lock(&lock);
 	init_completion(&finished);
-	num_threads = num_online_cpus();
+	atomic_set(&num_threads, num_online_cpus());
 	set_state(STOPMACHINE_PREPARE);
 
 	for_each_online_cpu(i) {

@@ -152,10 +167,21 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 
 	/* We've created all the threads.  Wake them all: hold this CPU so one
 	 * doesn't hit this CPU until we're ready. */
+	cpus_clear(prepared_cpus);
 	get_cpu();
 	for_each_online_cpu(i)
 		wake_up_process(threads[i]);
 
+	/* Wait all others come to life */
+	if (stopmachine_timeout) {
+		limit = jiffies + msecs_to_jiffies(stopmachine_timeout);
+		while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+			if (time_is_before_jiffies(limit))
+				goto timeout;
+			cpu_relax();
+		}
+	}
+
 	/* This will release the thread on our CPU. */
 	put_cpu();
 	wait_for_completion(&finished);

@@ -169,10 +195,32 @@ kill_threads:
 	for_each_online_cpu(i)
 		if (threads[i])
 			kthread_stop(threads[i]);
+	atomic_set(&num_threads, 0);
 	mutex_unlock(&lock);
 
 	kfree(threads);
 	return err;
+
+timeout:
+	printk(KERN_CRIT "stopmachine: Failed to stop machine in time(%lds).\n",
+			stopmachine_timeout);
+	for_each_online_cpu(i) {
+		if (!cpu_isset(i, prepared_cpus) && i != smp_processor_id())
+			printk(KERN_CRIT "stopmachine: cpu#%d seems to be "
+					"stuck.\n", i);
+		/* Unbind threads */
+		set_cpus_allowed(threads[i], cpu_online_map);
+	}
+
+	/* Let threads go exit */
+	set_state(STOPMACHINE_EXIT);
+
+	put_cpu();
+	/* no wait for completion */
+	mutex_unlock(&lock);
+	kfree(threads);
+
+	return -EBUSY;	/* canceled */
 }
 
 int stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)

kernel/sysctl.c +15 -0

index 2911665..3c7ca98 100644
@@ -146,6 +146,10 @@ extern int no_unaligned_warning;
 extern int max_lock_depth;
 #endif
 
+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif
+
 #ifdef CONFIG_PROC_SYSCTL
 static int proc_do_cad_pid(struct ctl_table *table, int write, struct file *filp,
 		  void __user *buffer, size_t *lenp, loff_t *ppos);

@@ -813,6 +817,17 @@ static struct ctl_table kern_table[] = {
 		.child		= key_sysctls,
 	},
 #endif
+#ifdef CONFIG_STOP_MACHINE
+	{
+		.ctl_name       = CTL_UNNUMBERED,
+		.procname       = "stopmachine_timeout",
+		.data           = &stopmachine_timeout,
+		.maxlen         = sizeof(unsigned long),
+		.mode           = 0644,
+		.proc_handler   = &proc_doulongvec_minmax,
+		.strategy       = &sysctl_intvec,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

I really don't like this, it means the system is really screwed up and doesn't deserve to continue.

I really don't like this, it means the system is really screwed up and doesn't deserve to continue.

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu in time. You can enable this timeout via sysctl.

v3:
- set stopmachine_timeout default to 0 (= never timeout)

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs
- allow disabling timeout by setting the stopmachine_timeout to 0

Signed-off-by Hidetoshi Seto <[email protected]>

---
 kernel/stop_machine.c |   54 ++++++++++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c       |   15 +++++++++++++
 2 files changed, 66 insertions(+), 3 deletions(-)

kernel/stop_machine.c +51 -3

index 5b72c2b..77b7944 100644
@@ -35,15 +35,18 @@ struct stop_machine_data {
 };
 
 /* Like num_online_cpus(), but hotplug cpu uses us, so we need this. */
-static unsigned int num_threads;
+static atomic_t num_threads;
 static atomic_t thread_ack;
+static cpumask_t prepared_cpus;
 static struct completion finished;
 static DEFINE_MUTEX(lock);
 
+unsigned long stopmachine_timeout = 0; /* msecs, 0 = "never timeout" */
+
 static void set_state(enum stopmachine_state newstate)
 {
 	/* Reset ack counter. */
-	atomic_set(&thread_ack, num_threads);
+	atomic_set(&thread_ack, atomic_read(&num_threads));
 	smp_wmb();
 	state = newstate;
 }

@@ -67,6 +70,8 @@ static int stop_cpu(struct stop_machine_data *smdata)
 	enum stopmachine_state curstate = STOPMACHINE_NONE;
 	int uninitialized_var(ret);
 
+	cpu_set(smp_processor_id(), prepared_cpus);
+
 	/* Simple state machine */
 	do {
 		/* Chill out and ensure we re-read stopmachine_state. */

@@ -90,6 +95,7 @@ static int stop_cpu(struct stop_machine_data *smdata)
 		}
 	} while (curstate != STOPMACHINE_EXIT);
 
+	atomic_dec(&num_threads);
 	local_irq_enable();
 	do_exit(0);
 }

@@ -105,6 +111,15 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 	int i, err;
 	struct stop_machine_data active, idle;
 	struct task_struct **threads;
+	unsigned long limit;
+
+	if (atomic_read(&num_threads)) {
+		/*
+		 * previous stop_machine was timeout, and still there are some
+		 * unfinished thread (dangling stucked CPU?).
+		 */
+		return -EBUSY;
+	}
 
 	active.fn = fn;
 	active.data = data;

@@ -120,7 +135,7 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 	/* Set up initial state. */
 	mutex_lock(&lock);
 	init_completion(&finished);
-	num_threads = num_online_cpus();
+	atomic_set(&num_threads, num_online_cpus());
 	set_state(STOPMACHINE_PREPARE);
 
 	for_each_online_cpu(i) {

@@ -152,10 +167,21 @@ int __stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)
 
 	/* We've created all the threads.  Wake them all: hold this CPU so one
 	 * doesn't hit this CPU until we're ready. */
+	cpus_clear(prepared_cpus);
 	get_cpu();
 	for_each_online_cpu(i)
 		wake_up_process(threads[i]);
 
+	/* Wait all others come to life */
+	if (stopmachine_timeout) {
+		limit = jiffies + msecs_to_jiffies(stopmachine_timeout);
+		while (cpus_weight(prepared_cpus) != num_online_cpus() - 1) {
+			if (time_is_before_jiffies(limit))
+				goto timeout;
+			cpu_relax();
+		}
+	}
+
 	/* This will release the thread on our CPU. */
 	put_cpu();
 	wait_for_completion(&finished);

@@ -169,10 +195,32 @@ kill_threads:
 	for_each_online_cpu(i)
 		if (threads[i])
 			kthread_stop(threads[i]);
+	atomic_set(&num_threads, 0);
 	mutex_unlock(&lock);
 
 	kfree(threads);
 	return err;
+
+timeout:
+	printk(KERN_CRIT "stopmachine: Failed to stop machine in time(%lds).\n",
+			stopmachine_timeout);
+	for_each_online_cpu(i) {
+		if (!cpu_isset(i, prepared_cpus) && i != smp_processor_id())
+			printk(KERN_CRIT "stopmachine: cpu#%d seems to be "
+					"stuck.\n", i);
+		/* Unbind threads */
+		set_cpus_allowed(threads[i], cpu_online_map);
+	}
+
+	/* Let threads go exit */
+	set_state(STOPMACHINE_EXIT);
+
+	put_cpu();
+	/* no wait for completion */
+	mutex_unlock(&lock);
+	kfree(threads);
+
+	return -EBUSY;	/* canceled */
 }
 
 int stop_machine_run(int (*fn)(void *), void *data, const cpumask_t *cpus)

kernel/sysctl.c +15 -0

index 2911665..3c7ca98 100644
@@ -146,6 +146,10 @@ extern int no_unaligned_warning;
 extern int max_lock_depth;
 #endif
 
+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif
+
 #ifdef CONFIG_PROC_SYSCTL
 static int proc_do_cad_pid(struct ctl_table *table, int write, struct file *filp,
 		  void __user *buffer, size_t *lenp, loff_t *ppos);

@@ -813,6 +817,17 @@ static struct ctl_table kern_table[] = {
 		.child		= key_sysctls,
 	},
 #endif
+#ifdef CONFIG_STOP_MACHINE
+	{
+		.ctl_name       = CTL_UNNUMBERED,
+		.procname       = "stopmachine_timeout",
+		.data           = &stopmachine_timeout,
+		.maxlen         = sizeof(unsigned long),
+		.mode           = 0644,
+		.proc_handler   = &proc_doulongvec_minmax,
+		.strategy       = &sysctl_intvec,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

kernel/sysctl.c extern int no_unaligned_warning; · line 151

Quoted 3 lines · Hidetoshi Seto

kernel/sysctl.c +3 -0

+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif

No externs in C files. Put it in an appropriate header.

I'll do a proper review soon.

kernel/sysctl.c extern int no_unaligned_warning; · line 151

Quoted 3 lines · Hidetoshi Seto

kernel/sysctl.c +3 -0

+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif

No externs in C files. Put it in an appropriate header.

I'll do a proper review soon.

sysctl.c already has many externs... but I can fix at least the above.

Quoted 2 lines

I'll do a proper review soon.

Is it better to postpone v4 until your comment comes?

sysctl.c already has many externs... but I can fix at least the above.

Quoted 2 lines

I'll do a proper review soon.

Is it better to postpone v4 until your comment comes?

Quoted 6 lines · Jeremy Fitzhardinge

kernel/sysctl.c extern int no_unaligned_warning; · line 151

Quoted 3 lines · Hidetoshi Seto

kernel/sysctl.c +3 -0

+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif

No externs in C files. Put it in an appropriate header.

sysctl.c already has many externs... but I can fix at least the above.

Quoted 1 line

I'll do a proper review soon.

Is it better to postpone v4 until your comment comes?

Quoted 6 lines · Jeremy Fitzhardinge

kernel/sysctl.c extern int no_unaligned_warning; · line 151

Quoted 3 lines · Hidetoshi Seto

kernel/sysctl.c +3 -0

+#ifdef CONFIG_STOP_MACHINE
+extern unsigned long stopmachine_timeout;
+#endif

No externs in C files. Put it in an appropriate header.

sysctl.c already has many externs... but I can fix at least the above.

Quoted 1 line

I'll do a proper review soon.

Is it better to postpone v4 until your comment comes?

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu in time. You can enable this timeout via sysctl.

v4:
- move extern into linux/stop_machine.h and add include of the header to kernel/sysctl.c. Now stopmachine_timeout is available only on smp.

v3:
- set stopmachine_timeout default to 0 (= never timeout)

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs
- allow disabling timeout by setting the stopmachine_timeout to 0

Signed-off-by Hidetoshi Seto <[email protected]>

---
 include/linux/stop_machine.h |    3 ++
 kernel/stop_machine.c        |   54 +++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              |   12 +++++++++
 3 files changed, 66 insertions(+), 3 deletions(-)

If stop_machine() invoked while one of onlined cpu is locked up by some reason, stop_machine cannot finish its work because the locked cpu cannot stop. This means all other healthy cpus will be blocked infinitely by one dead cpu.

This patch allows stop_machine to return -EBUSY with some printk messages if any of stop_machine's threads cannot start running on its target cpu in time. You can enable this timeout via sysctl.

v4:
- move extern into linux/stop_machine.h and add include of the header to kernel/sysctl.c. Now stopmachine_timeout is available only on smp.

v3:
- set stopmachine_timeout default to 0 (= never timeout)

v2:
- remove fix for warning since it will be fixed upcoming typesafe patches
- make stopmachine_timeout from secs to msecs
- allow disabling timeout by setting the stopmachine_timeout to 0

Signed-off-by Hidetoshi Seto <[email protected]>

---
 include/linux/stop_machine.h |    3 ++
 kernel/stop_machine.c        |   54 +++++++++++++++++++++++++++++++++++++++--
 kernel/sysctl.c              |   12 +++++++++
 3 files changed, 66 insertions(+), 3 deletions(-)