Skip to contentSkip to conversation

Booting when CPUs fail to come up.

PATCH In review Rusty Russell 3 participants 5 messages Feb 13, 2004 – Feb 22, 2004

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Booting when CPUs fail to come up.

I recently played with setting a bit in cpu_possible_map that wasn't in cpu_online_map: this can happen without hotplug CPU when a CPU fails to boot, for example.

1) i386 should use cpu_callin_map for num_booting_cpus() (an x86-ism anyway): if a CPU doesn't come up, it will be set in cpu_possible_map (aka cpu_callout_map) but not cpu_callin_map.

2) When the cpu fails to come up, some callbacks do kthread_stop(), which doesn't work without keventd (which hasn't started yet). Call it directly, and take care that it restores signal state (note: do_sigaction does a flush on blocked signals, so we don't need to repeat it).

arch/i386/kernel/smpboot.c +1 -1
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .13580-linux-2.6.3-rc2-mm1/arch/i386/kernel/smpboot.c .13580-linux-2.6.3-rc2-mm1.updated/arch/i386/kernel/smpboot.c
@@ -67,7 +67,7 @@ int smp_num_siblings = 1;
 int phys_proc_id[NR_CPUS]; /* Package ID of each logical CPU */
 
 
-static cpumask_t cpu_callin_map;
+cpumask_t cpu_callin_map;
 cpumask_t cpu_callout_map;
 static cpumask_t smp_commenced_mask;
 
include/asm-i386/smp.h +2 -1
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .13580-linux-2.6.3-rc2-mm1/include/asm-i386/smp.h .13580-linux-2.6.3-rc2-mm1.updated/include/asm-i386/smp.h
@@ -58,7 +58,8 @@ extern cpumask_t cpu_callout_map;
 /* We don't mark CPUs online until __cpu_up(), so we need another measure */
 static inline int num_booting_cpus(void)
 {
-	return cpus_weight(cpu_callout_map);
+	extern cpumask_t cpu_callin_map;
+	return cpus_weight(cpu_callin_map);
 }
 
 extern void map_cpu_to_logical_apicid(void);
kernel/kthread.c +15 -12
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal .13580-linux-2.6.3-rc2-mm1/kernel/kthread.c .13580-linux-2.6.3-rc2-mm1.updated/kernel/kthread.c
@@ -100,15 +100,16 @@ static void keventd_stop_kthread(void *_
 {
 	struct kthread_stop_info *stop = _stop;
 	int status, pid;
-	sigset_t blocked;
-	struct k_sigaction sa;
+	sigset_t chldonly, oldset;
+	struct k_sigaction sa, oldsa;
 
 	/* Install a handler so SIGCHLD is actually delivered */
 	sa.sa.sa_handler = SIG_DFL;
 	sa.sa.sa_flags = 0;
 	siginitset(&sa.sa.sa_mask, sigmask(SIGCHLD));
-	do_sigaction(SIGCHLD, &sa, (struct k_sigaction *)0);
-	allow_signal(SIGCHLD);
+	siginitset(&chldonly, sigmask(SIGCHLD));
+	do_sigaction(SIGCHLD, &sa, &oldsa);
+	sigprocmask(SIG_UNBLOCK, &chldonly, &oldset);
 
 	adopt_kthread(stop->k);
 	/* Grab pid now: after waitpid(), stop->k is invalid. */
@@ -123,12 +124,9 @@ static void keventd_stop_kthread(void *_
 	stop->result = -((status >> 8) & 0xFF);
 	complete(&stop->done);
 
-	/* Back to normal: block and flush all signals */
-	sigfillset(&blocked);
-	sigprocmask(SIG_BLOCK, &blocked, NULL);
-	flush_signals(current);
-	sa.sa.sa_handler = SIG_IGN;
-	do_sigaction(SIGCHLD, &sa, (struct k_sigaction *)0);
+	/* Return to normal, then reap any children who died in the race. */
+	sigprocmask(SIG_SETMASK, &oldset, NULL);
+	do_sigaction(SIGCHLD, &oldsa, NULL);
 	while (waitpid(-1, &status, __WALL|WNOHANG) > 0);
 }
 
@@ -179,7 +177,12 @@ int kthread_stop(struct task_struct *k)
 	stop.k = k;
 	init_completion(&stop.done);
 
-	schedule_work(&work);
-	wait_for_completion(&stop.done);
+	/* At boot, if CPUs fail to come up, this happens. */
+	if (!keventd_up())
+		work.func(work.data);
+	else {
+		schedule_work(&work);
+		wait_for_completion(&stop.done);
+	}
 	return stop.result;
 }

--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#1 lore raw

Hi!

Quoted 8 lines

I recently played with setting a bit in cpu_possible_map that wasn't in cpu_online_map: this can happen without hotplug CPU when a CPU fails to boot, for example.

1) i386 should use cpu_callin_map for num_booting_cpus() (an x86-ism anyway): if a CPU doesn't come up, it will be set in cpu_possible_map (aka cpu_callout_map) but not cpu_callin_map.

Quoted 5 lines

2) When the cpu fails to come up, some callbacks do kthread_stop(), which doesn't work without keventd (which hasn't started yet). Call it directly, and take care that it restores signal state (note: do_sigaction does a flush on blocked signals, so we don't need to repeat it).

Is it safe to continue when one cpu is apparently malfunctioning?

Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#2 lore raw

In message <[email protected]> you write:

Quoted 7 lines

Hi!

Quoted 3 lines

I recently played with setting a bit in cpu_possible_map that wasn't in cpu_online_map: this can happen without hotplug CPU when a CPU fails to boot, for example.

Is it safe to continue when one cpu is apparently malfunctioning?

Well, patch was overzealous and no longer required.

But we shouldn't crash when this happens just because a CPU didn't come up.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#3 lore raw

Hi!

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

I recently played with setting a bit in cpu_possible_map that wasn't in cpu_online_map: this can happen without hotplug CPU when a CPU fails to boot, for example.

Is it safe to continue when one cpu is apparently malfunctioning?

Well, patch was overzealous and no longer required.

But we shouldn't crash when this happens just because a CPU didn't come up.

I still do not agree.

You have a system you tried to kick CPU #13 alive, and something very wrong happened, CPU #13 did not come up. It is there, has full access to memory, it is probably running some kind of program.... I'd not dare mount disks read-write in such situation and I believe crashing early is actually right thing to do.
Pavel

Signature · 2 lines
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
#4 lore raw

In message <[email protected]> you write:

Quoted 6 lines

Hi!

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I recently played with setting a bit in cpu_possible_map that wasn't in cpu_online_map: this can happen without hotplug CPU when a CPU fails to boot, for example.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Is it safe to continue when one cpu is apparently malfunctioning?

Well, patch was overzealous and no longer required.

But we shouldn't crash when this happens just because a CPU didn't come up.

I still do not agree.

You're entitled. However, on x86 we booted before when a secondary CPU didn't come up, and the patch was designed to ensure that we still did so.

Quoted 3 lines

You have a system you tried to kick CPU #13 alive, and something very wrong happened, CPU #13 did not come up. It is there, has full access to memory, it is probably running some kind of program....

No, it's possible, but not online. This actually happens on archs where you have hotplug cpus, as well as x86 boot failures.

Quoted 2 lines

I'd not dare mount disks read-write in such situation and I believe crashing early is actually right thing to do.

Sure, send a patch for x86 to do that, and we can discuss that. I'm not going to break existing behavior by stealth though.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.