Skip to contentSkip to conversation

nohz and strange sleep latencies

DISCUSSION Pavel Machek 4 participants 23 messages Nov 19, 2007 – Nov 26, 2007 x86

lore
Share & tools
lore
#0 lore raw

Hi!

On unloaded x60 system, 2.6.24-rc3 (tainted-pavel-so if someone can reproduce it, it would be helpful):

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 30.71%CPU
0.00user 0.00system 0.02 (0m0.024s) elapsed 8.36%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.53%CPU
0.00user 0.00system 0.02 (0m0.026s) elapsed 7.75%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.34%CPU
0.00user 0.00system 0.20 (0m0.207s) elapsed 1.45%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.59%CPU
0.00user 0.00system 0.23 (0m0.230s) elapsed 1.30%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.61%CPU
0.00user 0.00system 0.20 (0m0.201s) elapsed 1.48%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.50%CPU
0.00user 0.00system 0.19 (0m0.190s) elapsed 1.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.37%CPU
0.00user 0.00system 0.04 (0m0.049s) elapsed 4.09%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.35%CPU
0.00user 0.00system 0.01 (0m0.017s) elapsed 11.89%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.38%CPU
0.00user 0.00system 0.08 (0m0.081s) elapsed 3.71%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU
0.00user 0.00system 0.01 (0m0.019s) elapsed 15.92%CPU

nohz=off helps a lot. while true; do time sleep 0.0; done does not have unexpected latencies.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#1 lore raw
Quoted 27 lines · Pavel Machek

Hi!

On unloaded x60 system, 2.6.24-rc3 (tainted-pavel-so if someone can reproduce it, it would be helpful):

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 30.71%CPU
0.00user 0.00system 0.02 (0m0.024s) elapsed 8.36%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.53%CPU
0.00user 0.00system 0.02 (0m0.026s) elapsed 7.75%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.34%CPU
0.00user 0.00system 0.20 (0m0.207s) elapsed 1.45%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.59%CPU
0.00user 0.00system 0.23 (0m0.230s) elapsed 1.30%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.61%CPU
0.00user 0.00system 0.20 (0m0.201s) elapsed 1.48%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.50%CPU
0.00user 0.00system 0.19 (0m0.190s) elapsed 1.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.37%CPU
0.00user 0.00system 0.04 (0m0.049s) elapsed 4.09%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.35%CPU
0.00user 0.00system 0.01 (0m0.017s) elapsed 11.89%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.38%CPU
0.00user 0.00system 0.08 (0m0.081s) elapsed 3.71%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU
0.00user 0.00system 0.01 (0m0.019s) elapsed 15.92%CPU
Quoted 2 lines

nohz=off helps a lot. while true; do time sleep 0.0; done does not have unexpected latencies.

does hpet=disable on the boot line also help? (without nohz=off)

Ingo

#2 lore raw

Hi!

Quoted 25 lines
Quoted 25 lines

On unloaded x60 system, 2.6.24-rc3 (tainted-pavel-so if someone can reproduce it, it would be helpful):

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 30.71%CPU
0.00user 0.00system 0.02 (0m0.024s) elapsed 8.36%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.53%CPU
0.00user 0.00system 0.02 (0m0.026s) elapsed 7.75%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.34%CPU
0.00user 0.00system 0.20 (0m0.207s) elapsed 1.45%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.59%CPU
0.00user 0.00system 0.23 (0m0.230s) elapsed 1.30%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.61%CPU
0.00user 0.00system 0.20 (0m0.201s) elapsed 1.48%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.50%CPU
0.00user 0.00system 0.19 (0m0.190s) elapsed 1.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.37%CPU
0.00user 0.00system 0.04 (0m0.049s) elapsed 4.09%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.35%CPU
0.00user 0.00system 0.01 (0m0.017s) elapsed 11.89%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.38%CPU
0.00user 0.00system 0.08 (0m0.081s) elapsed 3.71%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU
0.00user 0.00system 0.01 (0m0.019s) elapsed 15.92%CPU
Quoted 4 lines
Quoted 2 lines

nohz=off helps a lot. while true; do time sleep 0.0; done does not have unexpected latencies.

does hpet=disable on the boot line also help? (without nohz=off)

hpet-disable helps.. a bit. 200msec latencies are gone. (What is used for wakeups in this case?)

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 22.96%CPU
0.00user 0.00system 0.01 (0m0.018s) elapsed 11.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.54%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 8.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.34%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 8.14%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 8.16%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.30%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 5.47%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.25%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.54%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.56%CPU
0.00user 0.00system 0.03 (0m0.036s) elapsed 5.49%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.35%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 8.14%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.35%CPU
0.00user 0.00system 0.03 (0m0.037s) elapsed 8.14%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 23.34%CPU
									Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#3 lore raw
Quoted 9 lines · Pavel Machek
Quoted 8 lines
Quoted 6 lines

0.00user 0.00system 0.08 (0m0.081s) elapsed 3.71%CPU 0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU 0.00user 0.00system 0.01 (0m0.019s) elapsed 15.92%CPU

nohz=off helps a lot. while true; do time sleep 0.0; done does not have unexpected latencies.

does hpet=disable on the boot line also help? (without nohz=off)

Quoted 7 lines

hpet-disable helps.. a bit. 200msec latencies are gone. (What is used for wakeups in this case?)

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 22.96%CPU
0.00user 0.00system 0.01 (0m0.018s) elapsed 11.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.54%CPU

could you run this while doing the sleep test:

http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

Ingo

#4 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 7 lines

hpet-disable helps.. a bit. 200msec latencies are gone. (What is used for wakeups in this case?)

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 22.96%CPU
0.00user 0.00system 0.01 (0m0.018s) elapsed 11.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.54%CPU
Quoted 7 lines

could you run this while doing the sleep test:

http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

another question: do you feel and see these timer latencies? If unsure you can do a timing test from a remote (known-good) box:

while :; do time ssh testbox sleep 0.01; done

there'll be a fair bit of offset cost and noise, but these fluctuations should still be there and should be noticeable.

but i suspect the delays are "real", i.e. caused by timers not firing at the right moment and not by some GTOD anomaly. Once you send the debug data we might be able to tell more.

Ingo

#5 lore raw
In reply to Ingo Molnar

Hi!

Quoted 18 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines

0.00user 0.00system 0.08 (0m0.081s) elapsed 3.71%CPU 0.00user 0.00system 0.01 (0m0.013s) elapsed 23.33%CPU 0.00user 0.00system 0.01 (0m0.019s) elapsed 15.92%CPU

nohz=off helps a lot. while true; do time sleep 0.0; done does not have unexpected latencies.

does hpet=disable on the boot line also help? (without nohz=off)

Quoted 7 lines

hpet-disable helps.. a bit. 200msec latencies are gone. (What is used for wakeups in this case?)

pavel@amd:~$ while true; do time sleep 0.01; done
0.00user 0.00system 0.01 (0m0.013s) elapsed 22.96%CPU
0.00user 0.00system 0.01 (0m0.018s) elapsed 11.05%CPU
0.00user 0.00system 0.01 (0m0.013s) elapsed 15.54%CPU
Quoted 6 lines

could you run this while doing the sleep test:

http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

This was w/o hpet=disable . Do you want me to test with hpet=disable?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#6 lore raw
Quoted 5 lines · Pavel Machek
Quoted 3 lines

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

This was w/o hpet=disable . Do you want me to test with hpet=disable?

no, this is fine. You've got a hpet clockevents driver and two lapic drivers:

Clock Event Device: hpet
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Ingo

#7 lore raw
Quoted 22 lines · Ingo Molnar
Quoted 5 lines · Pavel Machek
Quoted 3 lines

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

This was w/o hpet=disable . Do you want me to test with hpet=disable?

no, this is fine. You've got a hpet clockevents driver and two lapic drivers:

Clock Event Device: hpet
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Quoted 8 lines

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Hmm, lapic is considered unstable in c2 by default. You have to tell the kernel that you trust it in C2 on the command line.

/me goes back to read the logs

tglx

#8 lore raw
Quoted 28 lines · Thomas Gleixner
Quoted 27 lines · Ingo Molnar
Quoted 5 lines · Pavel Machek
Quoted 3 lines

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

This was w/o hpet=disable . Do you want me to test with hpet=disable?

no, this is fine. You've got a hpet clockevents driver and two lapic drivers:

Clock Event Device: hpet
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Quoted 6 lines
Quoted 3 lines

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Hmm, lapic is considered unstable in c2 by default. You have to tell the kernel that you trust it in C2 on the command line.

yeah, i was wondering about that too. ACPI enumerated them properly at a certain stage:

ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ingo

#9 lore raw

Hi!

Quoted 31 lines
Quoted 24 lines
Quoted 24 lines
Quoted 5 lines
Quoted 3 lines

and send us the output? (Enabling CONFIG_TIMER_STATS, CONFIG_SCHED_DEBUG and CONFIG_SCHEDSTATS would maximize the amount of information.)

This was w/o hpet=disable . Do you want me to test with hpet=disable?

no, this is fine. You've got a hpet clockevents driver and two lapic drivers:

Clock Event Device: hpet
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Quoted 6 lines
Quoted 3 lines

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Hmm, lapic is considered unstable in c2 by default. You have to tell the kernel that you trust it in C2 on the command line.

Quoted 8 lines

yeah, i was wondering about that too. ACPI enumerated them properly at a certain stage:

ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

I tried to add nolapic_timer manually, and that helped, but:

checking TSC synchronization [CPU#0 -> CPU#1]:
Measured 627605 cycles TSC warp between CPUs, turning off TSC clock. Marking TSC unstable due to: check_tsc_sync_source failed. Brought up 2 CPUs
CPU0 attaching sched-domain:
domain 0: span 3
groups: 1 2
CPU1 attaching sched-domain:
domain 0: span 3
groups: 2 1
net_namespace: 64 bytes
...
Bluetooth: HCI socket layer initialized
ACPI: RTC can wake from S4
Time: hpet clocksource has been installed.
Clockevents: could not switch to one-shot mode:<6>Clockevents: could not switch to one-shot mode: lapic is not functional. Could not switch to high resolution mode on CPU 1
lapic is not functional.
Could not switch to high resolution mode on CPU 0
system 00:00: iomem range 0x0-0x9ffff could not be reserved

...and result is nohz is not functional :-(.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#10 lore raw
In reply to Ingo Molnar

Hi!

Quoted 22 lines
Quoted 15 lines
Quoted 15 lines

Clock Event Device: hpet
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Clock Event Device: lapic
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Quoted 6 lines
Quoted 3 lines

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Hmm, lapic is considered unstable in c2 by default. You have to tell the kernel that you trust it in C2 on the command line.

Quoted 8 lines

yeah, i was wondering about that too. ACPI enumerated them properly at a certain stage:

ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

I can confirm that setting max-cstate=1 helps. (Plus, problem gets _way_ more visible with stripped down config. With usb and irda off, time sleep .01 easily takes 2 seconds).
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#11 lore raw
In reply to Ingo Molnar

Hi!

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

to me this has the feeling of lapic breakage in C2 mode. Does it get any better if you boot with 'nolapic'? (but that might in turn turn off high-res timers and nohz in essence) Thomas, any ideas?

Hmm, lapic is considered unstable in c2 by default. You have to tell the kernel that you trust it in C2 on the command line.

Quoted 8 lines

yeah, i was wondering about that too. ACPI enumerated them properly at a certain stage:

ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C2] C3[C3])

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

I'm trying to decide when system is idle (lets say that means "no user task is scheduled to wakeup within 10 seconds)... I added some instrumentation to nohz subsystem, but it does not behave like I'd
expect: even if I run "while true; do sleep .01; done" loop, I see nohz preparing for 5 seconds sleep... while it seems obvious that it can only be 10msec sleep, and with max_cstate=1, it works that way... Plus, nte->start_pid seems to contain some random numbers :-(.

What am I doing wrong?

(Patch for illustration, I can generate full diff against vanilla, but...)
Pavel

+++ b/kernel/time/tick-sched.c
@@ -229,11 +232,13 @@ void tick_nohz_stop_sched_tick(void)
if (delta_jiffies > 1)
cpu_set(cpu, nohz_cpu_mask);

+ {
+ int user_wait = get_next_timer_interrupt(last_jiffies, 1) - last_jiffies;
+
+ if ((user_wait > HZ/10) && (num_online_cpus() == 1))
+ printk("NOHZ: user ready for %d:%d sec wait (kernel %d:%d sec wait), naughty %d\n", user_wait/HZ, user_wait%HZ, delta_jiffies/HZ, delta_jiffies%HZ, naughty_pid);
+ }
+
/*
* nohz_stop_sched_tick can be called several times before
* the nohz_restart_sched_tick is called. This happens when
+++ b/kernel/timer.c
@@ -691,6 +693,12 @@ static unsigned long __next_timer_interr
if (tbase_get_deferrable(nte->base))
continue;

+ if (flags && (nte->start_pid < 1))
+ continue;
+
+ if (flags)
+ naughty_pid = nte->start_pid;
+
found = 1;
expires = nte->expires;
/* Look at the cascade bucket(s)? */

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#12 lore raw
Quoted 5 lines · Pavel Machek
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Quoted 8 lines

I'm trying to decide when system is idle (lets say that means "no user task is scheduled to wakeup within 10 seconds)... I added some instrumentation to nohz subsystem, but it does not behave like I'd
expect: even if I run "while true; do sleep .01; done" loop, I see nohz preparing for 5 seconds sleep... while it seems obvious that it can only be 10msec sleep, and with max_cstate=1, it works that way... Plus, nte->start_pid seems to contain some random numbers :-(.

Quoted 4 lines

What am I doing wrong?

(Patch for illustration, I can generate full diff against vanilla, but...)

Just to make sure what we are hunting: Do you have the same problem with an non-pavel-tainted 2.6.24-rc3 ?

tglx

#13 lore raw

Hi!

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Strange.

Quoted 8 lines
Quoted 8 lines

I'm trying to decide when system is idle (lets say that means "no user task is scheduled to wakeup within 10 seconds)... I added some instrumentation to nohz subsystem, but it does not behave like I'd
expect: even if I run "while true; do sleep .01; done" loop, I see nohz preparing for 5 seconds sleep... while it seems obvious that it can only be 10msec sleep, and with max_cstate=1, it works that way... Plus, nte->start_pid seems to contain some random numbers :-(.

Quoted 7 lines
Quoted 4 lines

What am I doing wrong?

(Patch for illustration, I can generate full diff against vanilla, but...)

Just to make sure what we are hunting: Do you have the same problem with an non-pavel-tainted 2.6.24-rc3 ?

The strange sleep latencies were definitely there, I'll check for "offline cpu and get 1000 interrupts", too.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#14 lore raw
In reply to Thomas Gleixner
Quoted 4 lines · Thomas Gleixner
Quoted 3 lines · Pavel Machek
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Quoted 8 lines
Quoted 2 lines

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Strange...

Quoted 2 lines

Just to make sure what we are hunting: Do you have the same problem with an non-pavel-tainted 2.6.24-rc3 ?

The effect described above disappeared from both clean and tainted-pavel trees...?

But I have verified again that time sleep 0.01 is perfectly willing to sleep for 0.9second on untainted 2.6.24-rc3...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#15 lore raw
In reply to Thomas Gleixner

Hi!

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Digging into process_32|64.c...

64:

        while (1) {
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;

                        tick_nohz_stop_sched_tick();

32:

        while (1) {
                tick_nohz_stop_sched_tick();
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;

...eek? Which one is wrong?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#16 lore raw
Quoted 35 lines · Pavel Machek

Hi!

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Digging into process_32|64.c...

64:

        while (1) {
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;

                        tick_nohz_stop_sched_tick();

32:

        while (1) {
                tick_nohz_stop_sched_tick();
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;
Quoted 1 line

...eek? Which one is wrong?

Hm, it looks like you should have quoted more lines ...

In the second case (32), the tick_nohz_stop_sched_tick() seems to be redundant, so I bet it's this one.

Greetings,
Rafael

#17 lore raw
Quoted 36 lines · Rafael J. Wysocki
Quoted 35 lines · Pavel Machek

Hi!

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

Digging into process_32|64.c...

64:

        while (1) {
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;

                        tick_nohz_stop_sched_tick();

32:

        while (1) {
                tick_nohz_stop_sched_tick();
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;
Quoted 6 lines
Quoted 1 line

...eek? Which one is wrong?

Hm, it looks like you should have quoted more lines ...

In the second case (32), the tick_nohz_stop_sched_tick() seems to be redundant, so I bet it's this one.

OTOH, the ARM's process.c is more similar to process_32.c ...

#18 lore raw
In reply to Rafael J. Wysocki
Quoted 21 lines · Rafael J. Wysocki
Quoted 21 lines

Digging into process_32|64.c...

64:

        while (1) {
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;

                        tick_nohz_stop_sched_tick();

32:

        while (1) {
                tick_nohz_stop_sched_tick();
                while (!need_resched()) {
                        void (*idle)(void);

                        if (__get_cpu_var(cpu_idle_state))
                                __get_cpu_var(cpu_idle_state) = 0;
Quoted 6 lines
Quoted 1 line

...eek? Which one is wrong?

Hm, it looks like you should have quoted more lines ...

In the second case (32), the tick_nohz_stop_sched_tick() seems to be redundant, so I bet it's this one.

No. Both are fine. the 64bit version calls into tick_nohz_stop_sched_tick() way to often. This is a leftover from the old theory that we can have non-irq caused wakeups. Will fix it.

tglx

#19 lore raw
In reply to Thomas Gleixner

Hi!

Okay, so the problem seems to be we are using unreliable lapic timer... which is stopped in C3 (and in C2 on broken machines).

I do not see any mechanism to disable lapic; there seems to be some mechanism to work around stopped lapic timer is used (in acpi/processor_idle.c) but it does not seem to work. So what about this?

Pavel

arch/x86/kernel/apic_32.c +2 -2
index 08b07c1..b9bf79c 100644
@@ -63,8 +63,8 @@ static int enable_local_apic __initdata 
 static int local_apic_timer_verify_ok;
 /* Disable local APIC timer from the kernel commandline or via dmi quirk
    or using CPU MSR check */
-int local_apic_timer_disabled;
-/* Local APIC timer works in C2 */
+int local_apic_timer_disabled = 1;
+/* Local APIC timer works in C2: used in processor_idle.c  */
 int local_apic_timer_c2_ok;
 EXPORT_SYMBOL_GPL(local_apic_timer_c2_ok);
 
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#20 lore raw
Quoted 9 lines · Pavel Machek

Hi!

Okay, so the problem seems to be we are using unreliable lapic timer... which is stopped in C3 (and in C2 on broken machines).

I do not see any mechanism to disable lapic; there seems to be some mechanism to work around stopped lapic timer is used (in acpi/processor_idle.c) but it does not seem to work. So what about this?

nolapic_timer on 32bit
noapictimer on 64bit

Sigh, I know that sucks, but it's the history of the diverged trees.

tglx

#21 lore raw
In reply to Thomas Gleixner

Hi!

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

but perhaps somehow we miss this fact and fail to turn off the lapic clockevents drivers?

Ok, I guess I'm lost. If I offline second CPU, I immediately get 1000Hz timer tick... is that expected?

Hmm. No. I have no idea why this is happening.

34196 total events, 55.083 events/sec
echo 0 >/sys/devices/system/cpu/cpu1/online
36073 total events, 54.679 events/sec

It happened again, this time on 2.6.24-pre-rc1 (mostly untainted). Reboot and problem goes away... so this one is random :-(.

I'll try plug/unplug loops, perhaps that can trigger it reliably. (cpu replug does not fix it once triggered).
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#22 lore raw
In reply to Pavel Machek
Quoted 2 lines · Pavel Machek

On unloaded x60 system, 2.6.24-rc3 (tainted-pavel-so if someone can reproduce it, it would be helpful):

Can you please provide your .config ?

tglx