Hang on bootup when installing cpu idle menu
Hi,
I've notice this hang on bootup every so often. Sometimes it hangs for a second and sometimes it can hang for hours. Always at the same location. This is a soft hang. That is, interrupts are still enabled, and I'm able to perform a sysrq-t.
Here's the problem child:
Call Trace: [<c062cd67>] schedule_timeout+0x75/0x93 [<c062cd9f>] schedule_timeout_uninterruptible+0x1a/0x1c [<c043ca12>] msleep+0x15/0x1b [<c0407726>] cpu_idle_wait+0x80/0xe0 [<c05bbfa7>] cpuidle_uninstall_idle_handler+0x28/0x2a [<c05bc148>] cpuidle_switch_governor+0x20/0xd9 [<c05bc2eb>] cpuidle_register_governor+0x6d/0xa6 [<c076fb04>] init_menu+0x12/0x14 [<c074e535>] kernel_init+0x154/0x2d1 [<c0409dc7>] kernel_thread_helper+0x7/0x10
This happens after it installed cpu_idle_ladder, and later while installing cpu_idle_menu it hangs. But it's not the menu that is the problem. But it looks like the cpu_idle_wait is.
In cpu_idle from process_32.c we have:
while (!need_resched()) {
void (*idle)(void);
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
check_pgt_cache();
rmb();
idle = pm_idle;
if (!idle)
idle = default_idle;
if (cpu_is_offline(cpu))
play_dead();
__get_cpu_var(irq_stat).idle_timestamp = jiffies;
idle();
}and in cpu_idle_wait:
set_cpus_allowed(current, cpumask_of_cpu(this_cpu));
put_cpu();
cpus_clear(map);
for_each_online_cpu(cpu) {
per_cpu(cpu_idle_state, cpu) = 1;
cpu_set(cpu, map);
}
__get_cpu_var(cpu_idle_state) = 0;
wmb();
do {
ssleep(1);
for_each_online_cpu(cpu) {
if (cpu_isset(cpu, map) && !per_cpu(cpu_idle_state, cpu))
cpu_clear(cpu, map);
}
cpus_and(map, map, cpu_online_map);
} while (!cpus_empty(map));Basically, we set cpu_idle_state to one for each online cpu. Clear the cpu that is currently running. Then keep sleeping and wait till each cpu passes the cpu_idle and clears the cpu_idle_state per_cpu variable.
The problem I see here, especially in boot up, is that it doesn't check to see if a cpu is already idle. So if a CPU is idle and NO_HZ is set (yes I have that set), we can shut down an online CPU if there's no interrupts to run nor any threads to run. Which is the case on bootup.
I first thought about adding an "in_idle" per_cpu variable to check, but then I'm thinking that this can cause other races. Because this is used in the idle governor code, and that code can also be a module, we might remove the idle code when CPUS are still using it. So perhaps a simple smp_call_function to an empty function might be good enough. Something like this?
--- arch/x86/kernel/process_32.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/process_32.c +8 -0
@@ -204,6 +204,10 @@ void cpu_idle(void) } } +static void do_nothing (void *unused) +{ +} +void cpu_idle_wait (void ) {unsigned int cpu, this_cpu =get_cpu ();
@@ -221,6 +225,10 @@ void cpu_idle_wait(void)__get_cpu_var (cpu_idle_state) =0 ;wmb (); + +/* make sure all CPUs are awake */ +smp_call_function (do_nothing,NULL ,0 ,0 ); + do { ssleep(1); for_each_online_cpu(cpu) {
Thoughts?
Disclaimer: I've been seeing this when working with the mcount patches. So I haven't seen it on a vanilla git repo. But... I haven't been booting vanilla git repos, and this hang doesn't look related to mcount. Another problem is that this hang only happens once every 100 boots or so.
-- Steve