Skip to contentSkip to conversation

2.6.17 x86_64 regression - reboot fails due to deadlock

DISCUSSION Mr. Berkley Shands 3 participants 3 messages Jul 6, 2006 – Jul 7, 2006

lore
Share & tools
lore
#0 lore raw

With a SuperMicro H8DC8 (nvidia chipset), Dual Opteron 285's, 16GB, Centos 4.3 -

Under 2.6.16 both the tyan 2895 and the supermicro H8DC8 both will reboot corectly, in kernel/sys.c machine_restart() gets called. But with the changes to sys.c under 2.6.17, a new path is introduced, calling void kernel_restart_prepare(char *cmd)
which calls blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd); (line 588) Which looks at the first element of the notifier list, and blocks forever. But ONLY on the supermicro. The tyan, a very similar motherboard does not deadlock. It returns and still calls machine_restart(). So neither reboot nor "shutdown -fh now" actually get to the bios calls.

on the supermicro, (linux-2.6.17/kernel/sys.c)

static int __kprobes notifier_call_chain(struct notifier_block **nl,
unsigned long val, void *v)

{
       int ret = NOTIFY_DONE;
       struct notifier_block *nb;

       nb = rcu_dereference(*nl);
       while (nb) {
               ret = nb->notifier_call(nb, val, v);         /* this is 
the deadlock for the first entry */
               if ((ret & NOTIFY_STOP_MASK) == NOTIFY_STOP_MASK)
                       break;
               nb = rcu_dereference(nb->next);
       }
       return ret;
}

I see that 2.6.18 reworks this code further.

If I want to hurt myself really, really badly, disabling the call to blocking_notifier_call_chain(&reboot_notifier_list,... restores the reboot/power off functions.

In kdb, the system sits idle awaiting something to schedule, but nothing will schedule since there is a deadlock on the supermicro. Any clues as to how to find which notifier is deadlocked?

berkley

#1 lore raw
Quoted 4 lines

In kdb, the system sits idle awaiting something to schedule, but nothing will schedule since there is
a deadlock on the supermicro. Any clues as to how to find which notifier is deadlocked?

Hi,

if it's really a deadlock, then lockdep (new in 2.6.18-rc1) ought to find it... just enable the various locking debug options in the -rc1 kernel and... it's active.

Greetings,
Arjan van de Ven

#2 lore raw
In reply to Mr. Berkley Shands
Quoted 38 lines · Mr. Berkley Shands

With a SuperMicro H8DC8 (nvidia chipset), Dual Opteron 285's, 16GB, Centos 4.3 -

Under 2.6.16 both the tyan 2895 and the supermicro H8DC8 both will reboot corectly,
in kernel/sys.c machine_restart() gets called. But with the changes to sys.c under 2.6.17,
a new path is introduced, calling void kernel_restart_prepare(char *cmd) which calls blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd); (line 588)
Which looks at the first element of the notifier list, and blocks forever. But ONLY on the supermicro.
The tyan, a very similar motherboard does not deadlock. It returns and still calls machine_restart().
So neither reboot nor "shutdown -fh now" actually get to the bios calls.

on the supermicro, (linux-2.6.17/kernel/sys.c)

static int __kprobes notifier_call_chain(struct notifier_block **nl,
unsigned long val, void *v)

{
        int ret = NOTIFY_DONE;
        struct notifier_block *nb;

        nb = rcu_dereference(*nl);
        while (nb) {
                ret = nb->notifier_call(nb, val, v);         /* this is 
the deadlock for the first entry */
                if ((ret & NOTIFY_STOP_MASK) == NOTIFY_STOP_MASK)
                        break;
                nb = rcu_dereference(nb->next);
        }
        return ret;
}

I see that 2.6.18 reworks this code further.

Quoted 9 lines

If I want to hurt myself really, really badly, disabling the call to blocking_notifier_call_chain(&reboot_notifier_list,... restores the reboot/power off functions.

In kdb, the system sits idle awaiting something to schedule, but nothing will schedule since there is
a deadlock on the supermicro. Any clues as to how to find which notifier is deadlocked?

Are you able to do sysrq-T when it's stuck?

Something like this...
diff -puN kernel/sys.c~a kernel/sys.c

kernel/sys.c renamed from kernel/sys.c~a +6 -0
@@ -70,6 +70,8 @@
 int overflowuid = DEFAULT_OVERFLOWUID;
 int overflowgid = DEFAULT_OVERFLOWGID;
 
+static int foo;
+
 #ifdef CONFIG_UID16
 EXPORT_SYMBOL(overflowuid);
 EXPORT_SYMBOL(overflowgid);
@@ -141,6 +143,9 @@ static int __kprobes notifier_call_chain
 	nb = rcu_dereference(*nl);
 	while (nb) {
 		next_nb = rcu_dereference(nb->next);
+		if (foo)
+			print_symbol("calling %s()\n",
+				(unsigned long)nb->notifier_call);
 		ret = nb->notifier_call(nb, val, v);
 		if ((ret & NOTIFY_STOP_MASK) == NOTIFY_STOP_MASK)
 			break;
@@ -590,6 +595,7 @@ EXPORT_SYMBOL_GPL(emergency_restart);
 
 static void kernel_restart_prepare(char *cmd)
 {
+	foo = 1;
 	blocking_notifier_call_chain(&reboot_notifier_list, SYS_RESTART, cmd);
 	system_state = SYSTEM_RESTART;
 	device_shutdown();

_

Be aware that there's a known lock_cpu_hotplug()-vs-cpufreq deadlock, but afaik it's only been reported during suspend. Disabling CONFIG_HOTPLUG_CPU might make a difference.