cpu hotplug support broken in 2.6.23-rc3
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
Plus I guess it would be nice to add CPU HOTPLUG into MAINTAINERS file:
pavel@amd:/data/l/linux$ grep CPU MAINTAINERS CPU FREQUENCY DRIVERS CPUID/MSR DRIVER CPUSETS i386 SETUP CODE / CPU ERRATA WORKAROUNDS SCx200 CPU SUPPORT Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 8 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
Plus I guess it would be nice to add CPU HOTPLUG into
MAINTAINERSfile:
Quoted 6 lines
pavel@amd:/data/l/linux$ grep CPU MAINTAINERS CPU FREQUENCY DRIVERS CPUID/MSR DRIVER CPUSETS i386 SETUP CODE / CPU ERRATA WORKAROUNDS SCx200 CPU SUPPORT
...plus it actually breaks suspend, and it is regression from 2.6.22.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Jeff.
Hi,
Quoted 9 lines · Jeff Chua
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
So maybe diff between your and Pavel's config file will give an answer.
Any details about the software environment?
Regards,
Michal
Signature · 2 lines
LOG http://www.stardust.webpages.pl/log/
Quoted 9 lines · Jeff Chua
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
...or maybe difference is in the .config, or maybe I broken something in my kernel sources....
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 8 lines · Pavel Machek
Quoted 7 lines · Jeff Chua
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
Quoted 8 lines
Quoted 2 lines
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
...or maybe difference is in the .config, or maybe I broken something in my kernel sources....
Well, something seems to be wrong with the CPU hotplug, but it's insanely difficult to reproduce on my boxes.
I bet on one of the notifiers blocking while waiting on a frozen task.
Greetings,
Rafael
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Pavel Machek
Quoted 9 lines · Jeff Chua
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
Quoted 7 lines
Quoted 2 lines
...or maybe difference is in the .config, or maybe I broken something in my kernel sources....
Well, something seems to be wrong with the CPU hotplug, but it's insanely difficult to reproduce on my boxes.
I bet on one of the notifiers blocking while waiting on a frozen task.
It happens reliably for me, with this script... and randomly, when I just echo 0/1 > online from commandline... so it should not be anything with the frozen tasks.
echo test > /sys/power/disk
echo disk > /sys/power/state
reliably hangs on resume in the attached script. It works ok with nosmp.
Pavel
#!/bin/bash
killall klogd
echo -n "testing refrigerator (testproc)..."
echo testproc > /sys/power/disk
echo disk > /sys/power/state
echo "okay"
sleep 2
echo -n "testing drivers (test)..."
echo test > /sys/power/disk
echo disk > /sys/power/state
echo "okay"
sleep 2
echo -n "testing swsusp (reboot)..."
echo reboot > /sys/power/disk
echo disk > /sys/power/state
echo "okay"
sleep 2
echo -n "testing s2ram..."
s2ram
echo "okay"
sleep 2
echo -n "testing swsusp (shutdown)..."
echo shutdown > /sys/power/disk
echo disk > /sys/power/state
echo "okay"
sleep 2
echo -n "testing swsusp (platform)..."
echo platform > /sys/power/disk
echo disk > /sys/power/state
echo "okay"
sleep 2
echo -n "testing s2ram..."
s2ram
echo "okay"
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Quoted 18 lines · Rafael J. Wysocki
Quoted 16 lines · Pavel Machek
Quoted 9 lines · Jeff Chua
Quoted 5 lines · Pavel Machek
Quoted 4 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
...or maybe difference is in the .config, or maybe I broken something in my kernel sources....
Quoted 8 lines
Quoted 4 lines
Well, something seems to be wrong with the CPU hotplug, but it's insanely difficult to reproduce on my boxes.
I bet on one of the notifiers blocking while waiting on a frozen task.
It happens reliably for me, with this script... and randomly, when I just echo 0/1 > online from commandline... so it should not be anything with the frozen tasks.
That suggests the CPU hotplug just deadlocks internally.
Can you put some printk's into _cpu_down() and see where exactly it hangs?
Quoted 5 lines
echo test >
/sys/power/disk
echo disk >/sys/power/statereliably hangs on resume in the attached script. It works ok with nosmp.
Which step hangs it? Or is it at random?
Rafael
Quoted 43 lines
#!/bin/bash
killall klogdecho -n "testing refrigerator (testproc)..."
echo testproc >/sys/power/disk
echo disk >/sys/power/state
echo "okay"sleep 2
echo -n "testing drivers (test)..."
echo test >/sys/power/disk
echo disk >/sys/power/state
echo "okay"sleep 2
echo -n "testing swsusp (reboot)..."
echo reboot >/sys/power/disk
echo disk >/sys/power/state
echo "okay"sleep 2
echo -n "testing s2ram..."
s2ram
echo "okay"sleep 2
echo -n "testing swsusp (shutdown)..."
echo shutdown >/sys/power/disk
echo disk >/sys/power/state
echo "okay"sleep 2
echo -n "testing swsusp (platform)..."
echo platform >/sys/power/disk
echo disk >/sys/power/state
echo "okay"sleep 2
echo -n "testing s2ram..."
s2ram
echo "okay"
Signature · 1 line
"Premature optimization is the root of all evil." - Donald Knuth
2007/8/28, Rafael J. Wysocki <[email protected]>:
Quoted 11 lines · Pavel Machek
Quoted 10 lines · Rafael J. Wysocki
Quoted 9 lines · Pavel Machek
Quoted 8 lines · Jeff Chua
Quoted 7 lines · Pavel Machek
Quoted 5 lines · Pavel Machek
Quoted 4 lines
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
I just 3 cycles of on-line/off-line on 2.6.23-rc3 on ThinkPad x60s, and my system still survives.
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
...or maybe difference is in the .config, or maybe I broken something in my kernel sources....
I have been doing enough CPU offline/online test these days and it works fine. But there is no cpufreq driver which supports my machine. So my test didn't cover test cpu hotplug code in cpufreq.
If you have cpufreq driver and it is built as module, it is worth trying same test after unloading cpufreq driver in order to narrow down the problem area.
Quoted 2 lines · Pavel Machek
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
Mine still survives with this ... with sleep 1 ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; sleep 1; done
and this as well ... without sleep ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; done
I'm on reiserfs. gcc 3.4.5. Config sent to you seperately so as not to cloud lkml. If anyone wants the config, please let me know. Is mime "attachment" acceptable now on lkml?
Thanks,
Jeff.
Hi!
Quoted 8 lines
Quoted 2 lines
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
Mine still survives with this ... with sleep 1 ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; sleep 1; done
Quoted 8 lines
and this as well ... without sleep ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; doneI'm on reiserfs. gcc 3.4.5. Config sent to you seperately so as not to cloud lkml. If anyone wants the config, please let me know. Is mime "attachment" acceptable now on lkml?
Ok, so it gets weirder. I have now machine in "hung" state; other consoles still work, but there are no timers - sleep 1 hangs forever.
sysrq-t shows kstopmachine hung in hrtimer_try_to_cancel.
So I indeed suspect difference-in-kconfig to trigger this, and will try disabling noidlehz.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Hi!
Quoted 8 lines
Quoted 2 lines
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
Mine still survives with this ... with sleep 1 ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; sleep 1; doneQuoted 8 lines
and this as well ... without sleep ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; doneI'm on reiserfs. gcc 3.4.5. Config sent to you seperately so as not to cloud lkml. If anyone wants the config, please let me know. Is mime "attachment" acceptable now on lkml?
Quoted 7 lines
Ok, so it gets weirder. I have now machine in "hung" state; other consoles still work, but there are no timers - sleep 1 hangs forever.
sysrq-t shows kstopmachine hung in
hrtimer_try_to_cancel.So I indeed suspect difference-in-kconfig to trigger this, and will try disabling noidlehz.
I would unset CONFIG_HIGH_RES_TIMERS for starters.
Well, I guess Thomas should know about that. ;-)
Greetings,
Rafael
Quoted 11 lines · Rafael J. Wysocki
Quoted 7 lines
Ok, so it gets weirder. I have now machine in "hung" state; other consoles still work, but there are no timers - sleep 1 hangs forever.
sysrq-t shows kstopmachine hung in
hrtimer_try_to_cancel.So I indeed suspect difference-in-kconfig to trigger this, and will try disabling noidlehz.
I would unset
CONFIG_HIGH_RES_TIMERSfor starters.Well, I guess Thomas should know about that. ;-)
What was the last known to work version ?
tglx
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines
Quoted 5 lines
Ok, so it gets weirder. I have now machine in "hung" state; other consoles still work, but there are no timers - sleep 1 hangs forever.
sysrq-t shows kstopmachine hung in
hrtimer_try_to_cancel.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So I indeed suspect difference-in-kconfig to trigger this, and will try disabling noidlehz.
I would unset
CONFIG_HIGH_RES_TIMERSfor starters.Well, I guess Thomas should know about that. ;-)
What was the last known to work version ?
I'm afraid I only turned on HIGH_RES_TIMERS in 2.6.23-rc1 timeframe... so I'm not sure if it ever worked for me.
I can confirm it is working in 2.6.23-rc5 with highres disabled, and broken with highres enabled. NOHZ turns "waits for keypress during unplug/replug" into "just plain hangs".
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 13 lines · Pavel Machek
Quoted 13 lines · Rafael J. Wysocki
Quoted 11 lines
Quoted 7 lines
Ok, so it gets weirder. I have now machine in "hung" state; other consoles still work, but there are no timers - sleep 1 hangs forever.
sysrq-t shows kstopmachine hung in
hrtimer_try_to_cancel.So I indeed suspect difference-in-kconfig to trigger this, and will try disabling noidlehz.
I would unset
CONFIG_HIGH_RES_TIMERSfor starters.Well, I guess Thomas should know about that. ;-)
Quoted 8 lines
Quoted 1 line
What was the last known to work version ?
I'm afraid I only turned on
HIGH_RES_TIMERSin 2.6.23-rc1 timeframe... so I'm not sure if it ever worked for me.I can confirm it is working in 2.6.23-rc5 with highres disabled, and broken with highres enabled. NOHZ turns "waits for keypress during unplug/replug" into "just plain hangs".
Ok, I can reproduce it and I tracked down what happens:
When the CPU goes offline, the clock event source for this CPU (lapic) is removed from the clock events framework. This also clears the information that the CPU is using C-States which stop the local APIC timer.
Now you put the CPU online again and the local APIC timer is used, but the C-State information is not evaluated again in ACPI. This means that the clock events code does not know that the APIC might stop. In the worst case this will happen and make the CPU wait for timer interrupts forever.
The problem only appears when you are on battery (c3/c4 available) or on those broken machines, where C2 is in reality C3 (e.g. akpm's VAIO)
I have an yet untested fix, which preserves the broadcast state across the offline state, but Len is looking into it as well, whether we can just reevaluate the power states (and the broadcast flags) when a cpu becomes online again. If Len can do that easily for 2.6.23, I'd prefer that.
tglx
Hi!
Quoted 22 lines
Quoted 8 lines
Quoted 1 line
What was the last known to work version ?
I'm afraid I only turned on
HIGH_RES_TIMERSin 2.6.23-rc1 timeframe... so I'm not sure if it ever worked for me.I can confirm it is working in 2.6.23-rc5 with highres disabled, and broken with highres enabled. NOHZ turns "waits for keypress during unplug/replug" into "just plain hangs".
Ok, I can reproduce it and I tracked down what happens:
When the CPU goes offline, the clock event source for this CPU (lapic) is removed from the clock events framework. This also clears the information that the CPU is using C-States which stop the local APIC timer.
Now you put the CPU online again and the local APIC timer is used, but the C-State information is not evaluated again in ACPI. This means that the clock events code does not know that the APIC might stop. In the worst case this will happen and make the CPU wait for timer interrupts forever.
Quoted 8 lines
The problem only appears when you are on battery (c3/c4 available) or on those broken machines, where C2 is in reality C3 (e.g. akpm's VAIO)
I have an yet untested fix, which preserves the broadcast state across the offline state, but Len is looking into it as well, whether we can just reevaluate the power states (and the broadcast flags) when a cpu becomes online again. If Len can do that easily for 2.6.23, I'd prefer that.
Is there a patch you want me to test? Or does Len have anything to play with?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Pavel,
Quoted 8 lines · Pavel Machek
Quoted 5 lines
I have an yet untested fix, which preserves the broadcast state across the offline state, but Len is looking into it as well, whether we can just reevaluate the power states (and the broadcast flags) when a cpu becomes online again. If Len can do that easily for 2.6.23, I'd prefer that.
Is there a patch you want me to test? Or does Len have anything to play with?
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
Thanks,
tglx
--- kernel/time/tick-broadcast.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
kernel/time/tick-broadcast.c +18 -3
@@ -261,10 +261,25 @@ void tick_broadcast_on_off(unsigned longint cpu =get_cpu ();if (!cpu_isset (*oncpu, cpu_online_map)) { -printk (KERN_ERR "tick-braodcast: ignoring broadcast for " -"offline CPU #%d\n" , *oncpu); - }else { +unsigned long flags; + +spin_lock_irqsave (&tick_broadcast_lock, flags); +/* +* We need to cache the broadcast flag for offline +* CPUs. ACPI currently does not reevaluate the +* broadcast flag when a CPU goes online again. Adding +* a cpu notifier to ACPI is probably the correct +* solution, but it is hard to get this correct due to +* notify ordering problems. So caching the flag is +* the safe solution for now. +*/ +if (reason ==CLOCK_EVT_NOTIFY_BROADCAST_ON ) +cpu_set (*oncpu, tick_broadcast_mask); +else +cpu_clear (*oncpu, tick_broadcast_mask); +spin_unlock_irqrestore (&tick_broadcast_lock, flags); + }else {if (cpu == *oncpu)tick_do_broadcast_on_off (&reason);else
Quoted 9 lines · Thomas Gleixner
Pavel,
Quoted 6 lines · Pavel Machek
Quoted 5 lines
I have an yet untested fix, which preserves the broadcast state across the offline state, but Len is looking into it as well, whether we can just reevaluate the power states (and the broadcast flags) when a cpu becomes online again. If Len can do that easily for 2.6.23, I'd prefer that.
Quoted 5 lines
Quoted 2 lines
Is there a patch you want me to test? Or does Len have anything to play with?
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
Hmmpf, the flag is still cleared when the cpu goes offline. Need to take a closer look.
tglx
Pavel,
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
Hmmpf, the flag is still cleared when the cpu goes offline. Need to take a closer look.
I finally tracked it down. There were several ways to turn the box into a brick. Sigh !
Can you please test the combo patch below ?
The details are available from the for-2.6.23 branch of my hrt git repo:
http://git.kernel.org/?p=linux/kernel/git/tglx/linux-2.6-hrt.git;a=shortlog;h=for-2.6.23
Thanks,
tglx
kernel/time/timekeeping.c +9 -1
@@ -217,6 +217,7 @@ static void change_clocksource(void) }#else static inline void change_clocksource (void ) { } +static inline s64 __get_nsec_offset (void ) {return 0 ; }#endif /**
@@ -280,6 +281,8 @@ void __init timekeeping_init(void)static int timekeeping_suspended;/* time in seconds when suspend began */ static unsigned long timekeeping_suspend_time; +/* xtime offset when we went into suspend */ +static s64 timekeeping_suspend_nsecs;/** * timekeeping_resume - Resumes the generic timekeeping subsystem.
@@ -305,6 +308,8 @@ static int timekeeping_resume(struct sys wall_to_monotonic.tv_sec -= sleep_length; total_sleep_time += sleep_length; } +/* Make sure that we have the correct xtime reference */ +timespec_add_ns (&xtime, timekeeping_suspend_nsecs);/* re-base the last cycle value */ clock->cycle_last =clocksource_read (clock); clock->error =0 ;
@@ -325,9 +330,12 @@ static int timekeeping_suspend(struct sy {unsigned long flags; + timekeeping_suspend_time =read_persistent_clock (); +write_seqlock_irqsave (&xtime_lock, flags); +/* Get the current xtime offset */ + timekeeping_suspend_nsecs =__get_nsec_offset (); timekeeping_suspended =1 ; - timekeeping_suspend_time =read_persistent_clock ();write_sequnlock_irqrestore (&xtime_lock, flags);clockevents_notify (CLOCK_EVT_NOTIFY_SUSPEND ,NULL );
drivers/acpi/processor_core.c +21 -0
@@ -724,6 +724,25 @@ static void acpi_processor_notify(acpi_hreturn ; } +static int acpi_cpu_soft_notify (struct notifier_block *nfb, +unsigned long action,void *hcpu) +{ +unsigned int cpu = (unsigned long )hcpu; +struct acpi_processor *pr = processors[cpu]; + +if (action ==CPU_ONLINE && pr) { +acpi_processor_ppc_has_changed (pr); +acpi_processor_cst_has_changed (pr); +acpi_processor_tstate_has_changed (pr); + } +return NOTIFY_OK ; +} + +static struct notifier_block acpi_cpu_notifier = +{ + .notifier_call = acpi_cpu_soft_notify, +}; +static int acpi_processor_add (struct acpi_device *device) {struct acpi_processor *pr =NULL ;
@@ -987,6 +1006,7 @@ void acpi_processor_install_hotplug_notiACPI_UINT32_MAX , processor_walk_namespace_cb, &action, NULL);#endif +register_hotcpu_notifier (&acpi_cpu_notifier); } static
@@ -999,6 +1019,7 @@ void acpi_processor_uninstall_hotplug_noACPI_UINT32_MAX , processor_walk_namespace_cb, &action, NULL);#endif +unregister_hotcpu_notifier (&acpi_cpu_notifier); }/*
kernel/time/tick-broadcast.c +16 -8
@@ -382,12 +382,23 @@ static int tick_broadcast_set_event(ktimint tick_resume_broadcast_oneshot (struct clock_event_device *bc) { +int cpu =smp_processor_id (); + +/* +* If the CPU is marked for broadcast, enforce oneshot +* broadcast mode. The jinxed VAIO does not resume otherwise. +* No idea why it ends up in a lower C State during resume +* without notifying the clock events layer. +*/ +if (cpu_isset (cpu, tick_broadcast_mask)) +cpu_set (cpu, tick_broadcast_oneshot_mask); +clockevents_set_mode (bc,CLOCK_EVT_MODE_ONESHOT );if (!cpus_empty (tick_broadcast_oneshot_mask))tick_broadcast_set_event (ktime_get (),1 ); -return cpu_isset (smp_processor_id (), tick_broadcast_oneshot_mask); +return cpu_isset (cpu, tick_broadcast_oneshot_mask); }/*
@@ -549,20 +560,17 @@ void tick_broadcast_switch_to_oneshot(vo*/ void tick_shutdown_broadcast_oneshot (unsigned int *cpup) { -struct clock_event_device *bc;unsigned long flags;unsigned int cpu = *cpup;spin_lock_irqsave (&tick_broadcast_lock, flags); - bc = tick_broadcast_device.evtdev; +/* +* Clear the broadcast mask flag for the dead cpu, but do not +* stop the broadcast device! +*/ cpu_clear (cpu, tick_broadcast_oneshot_mask); -if (tick_broadcast_device.mode ==TICKDEV_MODE_ONESHOT ) { -if (bc &&cpus_empty (tick_broadcast_oneshot_mask)) -clockevents_set_mode (bc,CLOCK_EVT_MODE_SHUTDOWN ); - } -spin_unlock_irqrestore (&tick_broadcast_lock, flags); }
kernel/time/tick-sched.c +12 -0
@@ -160,6 +160,18 @@ void tick_nohz_stop_sched_tick(void) cpu =smp_processor_id (); ts = &per_cpu (tick_cpu_sched, cpu); +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =-1 ; + } +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) goto end;
Quoted 7 lines · On Sat, 15 Sep 2007 11:49:41 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
Hmmpf, the flag is still cleared when the cpu goes offline. Need to take a closer look.
Quoted 9 lines
I finally tracked it down. There were several ways to turn the box into a brick. Sigh !
Can you please test the combo patch below ?
The details are available from the for-2.6.23 branch of my hrt git repo:
http://git.kernel.org/?p=linux/kernel/git/tglx/linux-2.6-hrt.git;a=shortlog;h=for-2.6.23
That patch fixes the resume-from-ram and suspend-to-ram regressions on the Vaio.
I dropped the timekeeping.c hunks because they are an older version of timekeeping-prevent-time-going-backwards-on-resume.patch which I already had.
Is this good to go? Needs a bit of changelogging.
drivers/acpi/processor_core.c | 21 +++++++++++++++++++++ kernel/time/tick-broadcast.c | 24 ++++++++++++++++-------- kernel/time/tick-sched.c | 12 ++++++++++++ 3 files changed, 49 insertions(+), 8 deletions(-)
drivers/acpi/processor_core.c renamed from drivers/acpi/processor_core.c~cpu-hotplug-support-broken-in-2623-rc3 +21 -0
@@ -724,6 +724,25 @@ static void acpi_processor_notify(acpi_hreturn ; } +static int acpi_cpu_soft_notify (struct notifier_block *nfb, +unsigned long action,void *hcpu) +{ +unsigned int cpu = (unsigned long )hcpu; +struct acpi_processor *pr = processors[cpu]; + +if (action ==CPU_ONLINE && pr) { +acpi_processor_ppc_has_changed (pr); +acpi_processor_cst_has_changed (pr); +acpi_processor_tstate_has_changed (pr); + } +return NOTIFY_OK ; +} + +static struct notifier_block acpi_cpu_notifier = +{ + .notifier_call = acpi_cpu_soft_notify, +}; +static int acpi_processor_add (struct acpi_device *device) {struct acpi_processor *pr =NULL ;
@@ -987,6 +1006,7 @@ void acpi_processor_install_hotplug_notiACPI_UINT32_MAX , processor_walk_namespace_cb, &action, NULL);#endif +register_hotcpu_notifier (&acpi_cpu_notifier); } static
@@ -999,6 +1019,7 @@ void acpi_processor_uninstall_hotplug_noACPI_UINT32_MAX , processor_walk_namespace_cb, &action, NULL);#endif +unregister_hotcpu_notifier (&acpi_cpu_notifier); }/*
kernel/time/tick-broadcast.c renamed from kernel/time/tick-broadcast.c~cpu-hotplug-support-broken-in-2623-rc3 +16 -8
@@ -382,12 +382,23 @@ static int tick_broadcast_set_event(ktimint tick_resume_broadcast_oneshot (struct clock_event_device *bc) { +int cpu =smp_processor_id (); + +/* +* If the CPU is marked for broadcast, enforce oneshot +* broadcast mode. The jinxed VAIO does not resume otherwise. +* No idea why it ends up in a lower C State during resume +* without notifying the clock events layer. +*/ +if (cpu_isset (cpu, tick_broadcast_mask)) +cpu_set (cpu, tick_broadcast_oneshot_mask); +clockevents_set_mode (bc,CLOCK_EVT_MODE_ONESHOT );if (!cpus_empty (tick_broadcast_oneshot_mask))tick_broadcast_set_event (ktime_get (),1 ); -return cpu_isset (smp_processor_id (), tick_broadcast_oneshot_mask); +return cpu_isset (cpu, tick_broadcast_oneshot_mask); }/*
@@ -549,20 +560,17 @@ void tick_broadcast_switch_to_oneshot(vo*/ void tick_shutdown_broadcast_oneshot (unsigned int *cpup) { -struct clock_event_device *bc;unsigned long flags;unsigned int cpu = *cpup;spin_lock_irqsave (&tick_broadcast_lock, flags); - bc = tick_broadcast_device.evtdev; +/* +* Clear the broadcast mask flag for the dead cpu, but do not +* stop the broadcast device! +*/ cpu_clear (cpu, tick_broadcast_oneshot_mask); -if (tick_broadcast_device.mode ==TICKDEV_MODE_ONESHOT ) { -if (bc &&cpus_empty (tick_broadcast_oneshot_mask)) -clockevents_set_mode (bc,CLOCK_EVT_MODE_SHUTDOWN ); - } -spin_unlock_irqrestore (&tick_broadcast_lock, flags); }
kernel/time/tick-sched.c renamed from kernel/time/tick-sched.c~cpu-hotplug-support-broken-in-2623-rc3 +12 -0
@@ -160,6 +160,18 @@ void tick_nohz_stop_sched_tick(void) cpu =smp_processor_id (); ts = &per_cpu (tick_cpu_sched, cpu); +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =-1 ; + } +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) goto end;
_
Quoted 5 lines · Andrew Morton
On Sat, 15 Sep 2007 11:49:41 +0200 Thomas Gleixner <[email protected]> wrote:
I dropped the
timekeeping.chunks because they are an older version of timekeeping-prevent-time-going-backwards-on-resume.patch which I already had.
Err, no. The timekeeping hunk is redone due to the lockdep fix which I made.
Thanks,
tglx
Quoted 11 lines · Andrew Morton
Quoted 2 lines
http://git.kernel.org/?p=linux/kernel/git/tglx/linux-2.6-hrt.git;a=shortlog;h=for-2.6.23
That patch fixes the resume-from-ram and suspend-to-ram regressions on the Vaio.
I dropped the
timekeeping.chunks because they are an older version of timekeeping-prevent-time-going-backwards-on-resume.patch which I already had.Is this good to go? Needs a bit of changelogging.
Changelog it in the git tree. Please pull from there:
The following changes since commit 53a3f3087be361dacfc02e7a85b6d6142a41ce8a:
Linus Torvalds (1):
Merge branch 'for-linus' of master.kernel.org:/.../cooloney/blackfin-2.6
are available in the git repository at:
ssh://master.kernel.org/pub/scm/linux/kernel/git/tglx/linux-2.6-hrt.git for-2.6.23
Thomas Gleixner (6):
timekeeping: access rtc outside of xtime lock
timekeeping: Prevent time going backwards on resume
ACPI: Reevaluate C/P/T states when a cpu becomes online
clockevents: Enforce oneshot broadcast when broadcast mask is set on resume
clockevents: do not shutdown the oneshot broadcast device
clockevents: prevent stale tick update on offline cpu
drivers/acpi/processor_core.c | 21 +++++++++++++++++++++ kernel/time/tick-broadcast.c | 24 ++++++++++++++++-------- kernel/time/tick-sched.c | 12 ++++++++++++ kernel/time/timekeeping.c | 10 +++++++++- 4 files changed, 58 insertions(+), 9 deletions(-)
Quoted 7 lines · On Sat, 15 Sep 2007 15:28:23 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines
http://git.kernel.org/?p=linux/kernel/git/tglx/linux-2.6-hrt.git;a=shortlog;h=for-2.6.23
That patch fixes the resume-from-ram and suspend-to-ram regressions on the Vaio.
Quoted 7 lines
Quoted 5 lines
I dropped the
timekeeping.chunks because they are an older version of timekeeping-prevent-time-going-backwards-on-resume.patch which I already had.Is this good to go? Needs a bit of changelogging.
Changelog it in the git tree. Please pull from there:
who, me?
Quoted 8 lines
The following changes since commit
53a3f3087be361dacfc02e7a85b6d6142a41ce8a:
Linus Torvalds (1):
Merge branch 'for-linus' of master.kernel.org:/.../cooloney/blackfin-2.6are available in the git repository at:
ssh://master.kernel.org/pub/scm/linux/kernel/git/tglx/linux-2.6-hrt.git for-2.6.23
Quoted 7 lines
Thomas Gleixner (6):
timekeeping: access rtc outside of xtime lock
timekeeping: Prevent time going backwards on resume
ACPI: Reevaluate C/P/T states when a cpu becomes online
clockevents: Enforce oneshot broadcast when broadcast mask is set on resume
clockevents: do not shutdown the oneshot broadcast device
clockevents: prevent stale tick update on offline cpu
please send it to Linus?
Hi!
Quoted 10 lines
Quoted 5 lines
Quoted 2 lines
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
Hmmpf, the flag is still cleared when the cpu goes offline. Need to take a closer look.
I finally tracked it down. There were several ways to turn the box into a brick. Sigh !
Can you please test the combo patch below ?
Sorry, I was on holidays. I assume this is in -rc9 or so, already? Yes, seems so.
Unfortunately, cpu hotplug seems to be still behaving strangely in -rc9. I can echo 0 > online (and cpu will go down). I do echo 0 > online, again, and I get -EBUSY. Good. But I try to do echo 1 > online, and get -EBUSY, too... and that's bad :-(.
root@amd:/sys/devices/system/cpu/cpu1# echo 0 > online root@amd:/sys/devices/system/cpu/cpu1# echo 0 > online -bash: echo: write error: Device or resource busy root@amd:/sys/devices/system/cpu/cpu1# echo 1 > online -bash: echo: write error: Device or resource busy root@amd:/sys/devices/system/cpu/cpu1# uname -a Linux amd 2.6.23-rc9 #507 SMP Tue Oct 2 09:58:40 CEST 2007 i686 GNU/Linux
Kernel says:
Oct 2 11:42:12 amd log1n[1436]: ROOT LOGIN on `tty1' Oct 2 11:42:56 amd kernel: CPU 1 is now offline Oct 2 11:42:56 amd kernel: SMP alternatives: switching to UP code
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 23 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Thomas Gleixner
Sent: Friday, September 14, 2007 5:51 AM
To: Pavel Machek
Cc: Rafael J. Wysocki; Jeff Chua; [email protected]; [email protected]; [email protected]; kernel list; Len Brown
Subject: Re: cpu hotplug support broken in 2.6.23-rc3Pavel,
Quoted 1 line · Pavel Machek
Quoted 1 line
I have an yet untested fix, which preserves the broadcast
state across
Quoted 1 line
Quoted 1 line
the offline state, but Len is looking into it as well,
whether we can
Quoted 1 line
Quoted 1 line
just reevaluate the power states (and the broadcast flags)
when a cpu
Quoted 1 line
Quoted 1 line
becomes online again. If Len can do that easily for
2.6.23, I'd prefer
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
that.
Is there a patch you want me to test? Or does Len have anything to play with?
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
While wirting that patch, I knew solution could not be that simple :(. Does the patch work for online offline case atleast?
Will look at the Suspend/Resume ordering part in that case.
Thanks,
Venki
Quoted 4 lines · Venkatesh
Quoted 4 lines
Quoted 3 lines
Is there a patch you want me to test? Or does Len have anything to play with?
Quoted 8 lines
Quoted 4 lines
Venki sent me an initial patch, but it has issues with the notify ordering. Find below my "cache the broadcast flags" version for testing.
While wirting that patch, I knew solution could not be that simple :(. Does the patch work for online offline case atleast?
Will look at the Suspend/Resume ordering part in that case.
Yup, the online/offline part works and it helped me to decode the other reason (/me needs a dark brown paperbag) why Pavel noticed that his box turned into a brick. I'll send out a full series of fixups (including your online/offline one) tomorrow morning. I want to give that some more testing.
Vs. the resume reevaluation: I don't think it's an urgent problem. It's only my VAIO which does not tell the kernel after resume that the power supply source has changed. All my other boxen do that and we never had a complaint about that from other folks.
tglx
highres timers break cpu hotplug in 2.6.23-rc5 [was Re: cpu hotplug support broken in 2.6.23-rc3]
In reply to Jeff ChuaHi!
Quoted 8 lines
Quoted 2 lines
Can you try 20-or-so tests? Mine hangs randomly, so it survived 4 or so cycles at one point.
Mine still survives with this ... with sleep 1 ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; sleep 1; done
Quoted 8 lines
and this as well ... without sleep ...
# for((i=0; i<100; i++)); do echo $i; echo $((i % 2))
Quoted 1 line
/sys/devices/system/cpu/cpu1/online; doneI'm on reiserfs. gcc 3.4.5. Config sent to you seperately so as not to cloud lkml. If anyone wants the config, please let me know. Is mime "attachment" acceptable now on lkml?
It gets weirder. With "nohz=off" on commandline, I have to press any key (generate interrupt?) for echo 1 > online to finish. 2.6.23-rc5 kernel... but hotplug/unplug works reliably now.
With nohz=off highres=off I can unplug/replug cpus as much as I want... running in tight loop now.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: highres timers break cpu hotplug in 2.6.23-rc5 [was Re: cpu hotplug support broken in 2.6.23-rc3]
Quoted 6 lines · Pavel Machek
It gets weirder. With "nohz=off" on commandline, I have to press any key (generate interrupt?) for echo 1 > online to finish. 2.6.23-rc5 kernel... but hotplug/unplug works reliably now.
With nohz=off highres=off I can unplug/replug cpus as much as I want... running in tight loop now.
Yes. CONFIG_NO_HZ and and CONFIG_HIGH_RES_TIMERS has to be unset or suspend-to-disk would just hang, unless you type something on the keyboard, and then you can suspend to disk. It seems interrupts are not missing.
Thanks,
Jeff.
Hi Pavel,
Quoted 5 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
That's strange.
I've been running cpu offline/online tests with kern bench, cpufreq-ondemand and a few rt-tasks running in the background and it has worked for me.
Something like 100 iterations without a problem. But these were on machines with 4-8 cpus. So may be this could be something specific to the dual cpu machine.
Can you post the .config? I'll try to recreate it?
It's really strange since you mention that it tooks was an echo 1/0 into the sysfs file to break it.
Quoted 3 lines
Plus I guess it would be nice to add CPU HOTPLUG into
MAINTAINERSfile:
There is a list of maintainers in the Documentation/cpu-hotplug.txt, which includes maintainers for different platforms as well.
It's a good idea to add that info to the MAINTAINERS file as well.
Thanks and Regards
gautham.
Signature · 5 lines
Gautham R Shenoy Linux Technology Center IBM India. "Freedom comes with a price tag of responsibility, which is still a bargain, because Freedom is priceless!"
Quoted 8 lines · Gautham R Shenoy
Hi Pavel,
Quoted 5 lines · Pavel Machek
Hi!
Trying to do few onlines/offlines reliably hangs my machine (thinkpad x60, i386 architecture).
Quoted 8 lines
That's strange.
I've been running cpu offline/online tests with kern bench, cpufreq-ondemand and a few rt-tasks running in the background and it has worked for me.
Something like 100 iterations without a problem. But these were on machines with 4-8 cpus. So may be this could be something specific to the dual cpu machine.
Seems like it is specific to nohz/highrestimers.
Quoted 1 line
Can you post the .config? I'll try to recreate it?
Will send privately.
Quoted 8 lines
Quoted 3 lines
Plus I guess it would be nice to add CPU HOTPLUG into
MAINTAINERSfile:There is a list of maintainers in the
Documentation/cpu-hotplug.txt, which includes maintainers for different platforms as well.It's a good idea to add that info to the
MAINTAINERSfile as well.
Yes, please.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
cpu hotplug strangeness in 2.6.24-rc2 (was Re: cpu hotplug support broken in 2.6.23-rc3)
Hi!
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
Plus I guess it would be nice to add CPU HOTPLUG into
MAINTAINERSfile:There is a list of maintainers in the
Documentation/cpu-hotplug.txt, which includes maintainers for different platforms as well.It's a good idea to add that info to the
MAINTAINERSfile as well.Yes, please.
Just an update... In 2.6.24-rc2, cpu hotplug basically works, _but_:
if I do echo 0 > online; echo 0 > online; at same cpu, I get error, and can't up anything any more. It is not serious, but it is not pretty, either.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html