hires timer patchset [was Re: 2.6.19 -mm merge plans]
Sorry about dropping CC-s. I'm reading archived lkml and can't see them. Adding the obvious candidates instead (Andrew, you're not obvious, but if you'll pull in Thomas's stuff, you'll be...)
Quoted 5 lines · On 2006-09-22 20:29:40 Bill Rugolsky Jr. wrote:
Quoted 4 lines · Ingo Molnar
Quoted 1 line · Pavel Machek
Quoted 1 line
would be nice to merge the -hrt queue that goes right ontop
[...]
Quoted 8 lines
Quoted 4 lines
note that this is a new implementation of dynticks though, not Con's older stuff which you probably used, right? But it's fairly low-impact (just a few lines ontop of hrtimers, here and there), ontop of the long-existing -hrt queue.
Just a data point, FWIW; I've made no systematic effort to find regressions.
Quoted 8 lines
We've been running Thomas's (pre-dynticks) 2.6.16-hrt patchset (and HZ=1000) without issue as part of my standard kernel build on x86 and
x86_64UP/SMP production hosts -- workstations, PostgreSQL (and other) servers, and routers --- since I experienced latency/ntp problems with an unpatched kernel usingsata_nvon Tyan 2895 Nvidia CK804-chipset mobo back in March 2006. I've also been running the (originally x86-only) 2.6.17-dynticks patch on a Tyan Athlon SMP workstation mobo, and occasionally on my laptop, so far without issue.
Here's another data point: I tried 2.6.18-rt3 today on a x86_64 notebook. I'm on an eternal quest for extended battery time, so NO_HZ would be perfect. Long story shortened, HIGH_RES_TIMERS (prerequisite for NO_HZ) caused the CPU to never step down from max speed (ondemand, powernow_k8) at 2200 MHz.
In addition, something invisible used very frequent bursts of ~30% SYS CPU. Turning from PREEMPT_RT to PREEMPT_DESKTOP introduced occasional bursts of ~50% USER CPU (mixed with the SYS). Toggling RCU model made no difference.
Going from the default HIGH_RES_RESOLUTION=1000 to 10000 and even 100000 had no impact on the level or frequency of the bursts.
This resulted in the CPU temp never going lower than 53C, and thusly the fan kept spinning at 'level 2' - it needs 49C to fall back to normal speed. That sound tend to drive me insane...
Turning off HIGH_RES_TIMER allowed the CPU to enter the lowest 800 MHz immediately, and fan to slow down after the normal time, post-boot.
I'd be glad to help investigate this issue, but since -rt is completely new territory I'd need exact instructions about which knobs to turn. The CPU usage cause _was_ invisible, at least to "top". And I had all USB stuff unplugged, and no ehci_hcd loaded (that's another story, written in the bugzilla). Completely plain console, no X, no net, no nothing.
Here's a snippet from dmesg pertaining to the timers:
ACPI: PM-Timer IO Port: 0x4008
[...]
Event source pit configured with caps set: 07time.c: Detected 2201.306 MHz processor.
[...]
Calibrating delay using timer specific routine.. 4404.80 BogoMIPS (lpj=2202402)
[...]
Using local APIC timer interrupts.
result 12507441
Detected 12.507 MHz APIC timer.
lapic max_delta_ns: 670689262
Event source pit new caps set: 03
Event source lapic configured with caps set: 04
[...]
Real Time Clock Driver v1.12ac
[...]
Time: tsc clocksource has been installed.
Event source pit disabled
Event source lapic configured with caps set: 08
hrtimers: Switched to high resolution mode CPU 0
Oh, and here's a compiler warning fix (when PREEMPT_RT is off):
kernel/hrtimer.c +1 -1
@@ -786,7 +786,7 @@ static inline void hrtimer_raise_softirqraise_softirq_irqoff (HRTIMER_SOFTIRQ ); } -static inline int hrtimer_adjust_softirq_prio (struct hrtimer_cpu_base *base) +static inline void hrtimer_adjust_softirq_prio (struct hrtimer_cpu_base *base) { }
Mvh
Mats Johannesson
Quoted 11 lines · Voluspa
Here's another data point: I tried 2.6.18-rt3 today on a
x86_64notebook. I'm on an eternal quest for extended battery time, soNO_HZwould be perfect. Long story shortened,HIGH_RES_TIMERS(prerequisite forNO_HZ) caused the CPU to never step down from max speed (ondemand,powernow_k8) at 2200 MHz.In addition, something invisible used very frequent bursts of ~30% SYS CPU. Turning from
PREEMPT_RTtoPREEMPT_DESKTOPintroduced occasional bursts of ~50% USER CPU (mixed with the SYS). Toggling RCU model made no difference.
It seems like you don't need all of 2.6.18-rt3 , you just want dynamic tick .. You can obtain just the HRT/dynamic tick patch from here,
http://www.tglx.de/projects/hrtimers/2.6.18/
Daniel
Quoted 14 lines · On Sat, 23 Sep 2006 11:09:26 -0700 Daniel Walker wrote:
Quoted 11 lines · Voluspa
Here's another data point: I tried 2.6.18-rt3 today on a
x86_64notebook. I'm on an eternal quest for extended battery time, soNO_HZwould be perfect. Long story shortened,HIGH_RES_TIMERS(prerequisite forNO_HZ) caused the CPU to never step down from max speed (ondemand,powernow_k8) at 2200 MHz.In addition, something invisible used very frequent bursts of ~30% SYS CPU. Turning from
PREEMPT_RTtoPREEMPT_DESKTOPintroduced occasional bursts of ~50% USER CPU (mixed with the SYS). Toggling RCU model made no difference.
Quoted 4 lines
It seems like you don't need all of 2.6.18-rt3 , you just want dynamic tick .. You can obtain just the HRT/dynamic tick patch from here,
Thank you pointing that out. I didn't realise that the features were so sharply divided. Perhaps I'll even be able to debug it locally, then - wearing ear-mufflers. I'm not kidding about the fan noise.
At the very least the experiment will tell if it's a -rt combo problem or NO_HZ on its own.
Mvh
Mats Johannesson
Quoted 1 line · On Sat, 23 Sep 2006 11:09:26 -0700 Daniel Walker wrote:
On Sat, 2006-09-23 at 04:17 +0200, Voluspa wrote:
[...]
Quoted 4 lines
It seems like you don't need all of 2.6.18-rt3 , you just want dynamic tick .. You can obtain just the HRT/dynamic tick patch from here,
I'm not good enough at C yet to immediately see what needs fixing. And the hour is too late anyways:
BUILD arch/x86_64/boot/bzImage
Root device is (3, 2)
Boot sector 512 bytes.
Setup is 4709 bytes.
System is 1461 kB
Kernel: arch/x86_64/boot/bzImage is ready (#1)
Building modules, stage 2.
MODPOST
[usual ACPI section mismatch warning deleted]
WARNING: "monotonic_clock" [drivers/char/hangcheck-timer.ko] undefined!
WARNING: "hrtimer_stop_sched_tick" [drivers/acpi/processor.ko] undefined!
WARNING: "hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!
CC arch/x86_64/kernel/cpufreq/powernow-k8.mod.o
LD [M] arch/x86_64/kernel/cpufreq/powernow-k8.ko
Mvh
Mats Johannesson
Quoted 1 line · Voluspa
WARNING: "
monotonic_clock" [drivers/char/hangcheck-timer.ko] undefined!
turn off the CONFIG_HANGCHECK_TIMER option.
Quoted 2 lines
WARNING: "
hrtimer_stop_sched_tick" [drivers/acpi/processor.ko] undefined!
WARNING: "hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!
add these two lins to the end of kernel/hrtimer.c:
EXPORT_SYMBOL_GPL(hrtimer_stop_sched_tick);EXPORT_SYMBOL_GPL(hrtimer_restart_sched_tick);
Ingo
Quoted 8 lines · On Sat, 23 Sep 2006 22:20:27 +0200 Ingo Molnar wrote:
Quoted 2 lines · Voluspa
WARNING: "
monotonic_clock" [drivers/char/hangcheck-timer.ko] undefined!turn off the
CONFIG_HANGCHECK_TIMERoption.
Quoted 8 lines
Quoted 3 lines
WARNING: "
hrtimer_stop_sched_tick" [drivers/acpi/processor.ko] undefined! WARNING:
"hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!add these two lins to the end of
kernel/hrtimer.c:
EXPORT_SYMBOL_GPL(hrtimer_stop_sched_tick);EXPORT_SYMBOL_GPL(hrtimer_restart_sched_tick);
My mind was clouded close to bedtime. I now remember the fix that was published already at 2.6.17 time, from Steven Rostedt:
http://marc.theaimsgroup.com/?l=linux-kernel&m=115124086410874&w=2
Well, result is that NO_HZ indeed is the culprit for this CPU issue. /proc/interrupts showed the timer to be stuck on an initial 3044 triggers after boot, while NMI: counted up almost as fast as LOC: (if that tells any tale). Observing "top -d 1" for awhile revealed SYS bursting (almost regularly alternating) between 50% and 100% CPU each 2 to 3 seconds. In between it was 0. USER also had the same pattern, but with much longer duration. Perhaps 10 seconds from one show to the next.
I've gotten the broken out hrt-dyntick1 patches so will be able to experiment on my own - slowly, on spare time.
I am of course available for any thoughts or trials you can come up with in the meantime.
Mvh
Mats Johannesson