i386 No-Idle-Hz aka Dynamic-Ticks 3

Quoted 6 lines · Con Kolivas

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.

Runs very well here on a (noname) laptop, even with apic timer, the desktop does not "feel" different from static 1000HZ. But dtck reproducibly breaks software-suspend2; the kernel will simply stall on resume. Also stalls with dyntick=noapic. As soon as I set CONFIG_NO_IDLE_HZ = n, resume works again.

My kernel is 2.6.13-rc5 + swsusp-2.1.9.11 + dtck-3, the system is a Mitac 8375 laptop, via chipset, mobile athlon xp, gcc-3.4.4

Regards,
Jim

Jeffrey Hundstad

2005-08-03 19:55 UTC #20 lore raw

In reply to Jeffrey Hundstad

Quoted 7 lines · Con Kolivas

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.
The sysfs interface was moved to its own directory in /sys/devices/system/dyn_tick and split into separate files to enable/disable dynamic ticks and usage of apic on the fly. It makes sense to enable dynamic ticks and usage of apic by default if they're actually built into the kernel so that is now done.

I am successfully running the dynamic tick patch on an old IBM ThinkPad A22m. When I enable the APIC support console beeps, you know bash -c 'echo -e "\a"', takes a REALLY long time to finish. I'm assuming this is a badly written program and not a kernel problem. Correct?

BTW: how do you know what HZ your machine is running at?

Signature · 1 line

Jeffrey Hundstad

[email protected]

2005-08-03 20:08 UTC #21 lore raw

Quoted 1 line · Jeffrey Hundstad

BTW: how do you know what HZ your machine is running at?

% zcat /proc/config.gz | grep -i hz

might do what you thought you wanted.

What rate you're *actually* running at is probably best done by taking the number of timer interrupts from /proc/interrupts and dividing by the uptime in /proc/uptime....

Con Kolivas

author

2005-08-03 21:16 UTC #22 lore raw

Quoted 15 lines · Jeffrey Hundstad

Quoted 8 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.
Quoted 5 lines
The sysfs interface was moved to its own directory
in /sys/devices/system/dyn_tick and split into separate files to enable/disable dynamic ticks and usage of apic on the fly. It makes sense to enable dynamic ticks and usage of apic by default if they're actually built into the kernel so that is now done.

Quoted 4 lines

I am successfully running the dynamic tick patch on an old IBM ThinkPad A22m. When I enable the APIC support console beeps, you know bash -c 'echo -e "\a"', takes a REALLY long time to finish. I'm assuming this is a badly written program and not a kernel problem. Correct?

I think the config option which recommends leaving apic support off and describes this behaviour should make it clearer. Indeed I found running with apic on produced weird results unlike running with PIT.

Quoted 1 line

BTW: how do you know what HZ your machine is running at?

vmstat 5 will show you the average interrupts per second over 5 seconds in the "in" column. The interrupts are obviously more than your Hz but often only a little more.

The pmstats script by Tony will tell you more accurately:
http://www.muru.com/linux/dyntick/tools/pmstats-0.2.gz

Cheers,
Con

Con Kolivas

author

2005-08-03 21:18 UTC #7 lore raw

In reply to Jim MacBaine

Quoted 11 lines · Jim MacBaine

Quoted 7 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.
Runs very well here on a (noname) laptop, even with apic timer, the desktop does not "feel" different from static 1000HZ.

I did find performance dropped off substantially in interbench but usually below the human perceptible range. Real time performance dropped off substantially, but is back to mainline performance when disabled (ie config on but disabled at runtime/boottime).

Quoted 4 lines

But dtck
reproducibly breaks software-suspend2; the kernel will simply stall on resume. Also stalls with dyntick=noapic. As soon as I set CONFIG_NO_IDLE_HZ = n, resume works again.

What happens when you disable it at runtime before suspending?

echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/enable

Cheers,
Con

Jim MacBaine

2005-08-03 22:22 UTC #8 lore raw

Quoted 3 lines · Con Kolivas

What happens when you disable it at runtime before suspending?
echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/enable

This has no effect. The system stalls at exactly the same point. The last lines on my screen are:

...
Software Suspend Core.
Software Suspend Compression Driver loading. Software Suspend Encryption Driver loading. Software Suspend Swap Writer loading.
Software Suspend FileWriter loading.
dyn-tick: Maximum ticks to skip limited to 13
dyn-tick: Timer using dynamic tick
ACPI wakeup devices:
SBTN USB0 USB1 EHCI AC97 MC97 LAN0 FSD0
ACPI: (supports S0 S3 S4 S5)
Software Suspend 2.1.9.11: Swapwriter: Signature found. Software Suspend 2.1.9.11: Suspending enabled.

Regards,
Jim

Con Kolivas

author

2005-08-03 22:59 UTC #9 lore raw

Quoted 7 lines · Jim MacBaine

Quoted 3 lines · Con Kolivas
What happens when you disable it at runtime before suspending?
echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/enable
This has no effect. The system stalls at exactly the same point. The last lines on my screen are:

Ok perhaps on the resume side instead. When trying to resume can you try booting with 'dyntick=disable'. Note this isn't meant to be a long term fix but once we figure out where the problem is we should be able to code around it.

Cheers,
Con

Christian Leber

2005-08-03 23:23 UTC #23 lore raw

Quoted 1 line · Con Kolivas

Patch for 2.6.13-rc5

Just a few numbers:

I tried it on a Laptop (Dell C810, P3m 1133 mhz) and measured the power usage with an external device and it stayed with or without patch at 27W. (HZ was at about 28)

On a desktop with AthlonXP with STOP activated i got a power usage of 57W with and without patch (without STOP activated 94W). (HZ was about at 50)

Christian Leber

Signature · 3 lines

  "Omnis enim res, quae dando non deficit, dum habetur et non datur,
   nondum habetur, quomodo habenda est."       (Aurelius Augustinus)
  Translation: <http://gnuhh.org/work/fsf-europe/augustinus.html>

Jan De Luyck

2005-08-04 05:09 UTC #25 lore raw

Quoted 3 lines · Con Kolivas

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5

On a weird sidenote: my synaptics touchpad seems to not-like dyntick very much. When starting with a dyntick enabled kernel I get when psmouse.ko is loaded:

Aug  4 06:45:47 precious kernel: Synaptics claims to have extended capabilities, but I'm not able to read them.<3>Unable to initialize Synaptics hardware.
Aug  4 06:45:47 precious kernel: input: PS/2 Synaptics TouchPad on isa0060/serio1

subsequently, X fails to start too (touchpad is set as corepointer)

reloading the module right then and there solves the problem:

Aug  4 06:47:47 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8, id: 0x9d48b1, caps: 0x904713/0x4006
Aug  4 06:47:47 precious kernel: input: SynPS/2 Synaptics TouchPad on isa0060/serio1

Also, booting the same (but non-patched) kernel gives me a clean boot:

Aug  4 06:56:42 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8, id: 0x9d48b1, caps: 0x904713/0x4006
Aug  4 06:56:42 precious kernel: input: SynPS/2 Synaptics TouchPad on isa0060/serio1

This is constantly reproducable for me. I guess some timing issue somewhere?

Jan

Signature · 3 lines

A good name lost is seldom regained.  When character is gone,
all is gone, and one of the richest jewels of life is lost forever.
		-- J. Hawes

Con Kolivas

author

2005-08-04 05:11 UTC #26 lore raw

Quoted 22 lines · Jan De Luyck

Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
On a weird sidenote: my synaptics touchpad seems to not-like dyntick very much. When starting with a dyntick enabled kernel I get when psmouse.ko is loaded:
Aug  4 06:45:47 precious kernel: Synaptics claims to have extended
capabilities, but I'm not able to read them.<3>Unable to initialize Synaptics hardware. Aug 4 06:45:47 precious kernel: input: PS/2 Synaptics TouchPad on isa0060/serio1
subsequently, X fails to start too (touchpad is set as corepointer)
reloading the module right then and there solves the problem:
Aug  4 06:47:47 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8, id:
0x9d48b1, caps: 0x904713/0x4006 Aug 4 06:47:47 precious kernel: input:
SynPS/2 Synaptics TouchPad on isa0060/serio1

Quoted 8 lines

Also, booting the same (but non-patched) kernel gives me a clean boot:
Aug  4 06:56:42 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8, id:
0x9d48b1, caps: 0x904713/0x4006 Aug 4 06:56:42 precious kernel: input:
SynPS/2 Synaptics TouchPad on isa0060/serio1
This is constantly reproducable for me. I guess some timing issue somewhere?

Did you try without the apic option or disable it at runtime? The apic option is proving more problems than not so far for people that have tried it.

Cheers,
Con

Jan De Luyck

2005-08-04 05:34 UTC #27 lore raw

Quoted 29 lines · Con Kolivas

Quoted 28 lines · Jan De Luyck
Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
On a weird sidenote: my synaptics touchpad seems to not-like dyntick very much. When starting with a dyntick enabled kernel I get when psmouse.ko is loaded:
Aug  4 06:45:47 precious kernel: Synaptics claims to have extended
capabilities, but I'm not able to read them.<3>Unable to initialize Synaptics hardware. Aug 4 06:45:47 precious kernel: input: PS/2 Synaptics TouchPad on isa0060/serio1
subsequently, X fails to start too (touchpad is set as corepointer)
reloading the module right then and there solves the problem:
Aug  4 06:47:47 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8,
id: 0x9d48b1, caps: 0x904713/0x4006 Aug 4 06:47:47 precious kernel:
input: SynPS/2 Synaptics TouchPad on isa0060/serio1
Also, booting the same (but non-patched) kernel gives me a clean boot:
Aug  4 06:56:42 precious kernel: Synaptics Touchpad, model: 1, fw: 5.8,
id: 0x9d48b1, caps: 0x904713/0x4006 Aug 4 06:56:42 precious kernel:
input: SynPS/2 Synaptics TouchPad on isa0060/serio1

Quoted 6 lines

Quoted 2 lines
This is constantly reproducable for me. I guess some timing issue somewhere?
Did you try without the apic option or disable it at runtime? The apic option is proving more problems than not so far for people that have tried it.

The above was with apic enabled. With apic disabled, same story tho different boot-time message:

$ cat /sys/../state
suitable:       1
enabled:        1
apic suitable:  1
using APIC:     0

dmesg gives:
Unable to query Synaptics hardware.
input: PS/2 Synaptics TouchPad on isa0060/serio1

and X refuses to start. Same resolution, just reload psmouse.

Jan

Signature · 2 lines

The default Magic Word, "Abracadabra", actually is a corruption of the
Hebrew phrase "ha-Bracha dab'ra" which means "pronounce the blessing".

Jim MacBaine

2005-08-04 05:35 UTC #10 lore raw

In reply to Christian Leber

Quoted 4 lines · Con Kolivas

Ok perhaps on the resume side instead. When trying to resume can you try booting with 'dyntick=disable'. Note this isn't meant to be a long term fix but once we figure out where the problem is we should be able to code around it.

Sorry, no luck.

I tried dyntick=disable and dyntick=noapic on resume time. I also tried suspend and resume after the system has been started with dyntick=noapic: Same result.

As soon as I tell swsusp2 to discard its image, the system will come up flawlessly.

Regards,
Jim

Jim MacBaine

2005-08-04 07:00 UTC #11 lore raw

I just borrowed a power meter to see (or not to see) real effects of dyntick. The difference between static 1000 HZ and dynamic HZ is much less than I expected, only a very little about noise. With dyntick disabled at 1000 HZ my laptop needs 31,3 W. With dyntick enabled I get 29.8 W, the pmstats-0.2 script shows me that the system is at 35-45 HZ when it is idle.

The power consumption difference between 250 HZ static and dyntick is below the noise, so maybe hardly worth all the struggle.

Regards,
Jim

Con Kolivas

author

2005-08-04 07:09 UTC #12 lore raw

Quoted 9 lines · Jim MacBaine

I just borrowed a power meter to see (or not to see) real effects of dyntick. The difference between static 1000 HZ and dynamic HZ is much less than I expected, only a very little about noise. With dyntick disabled at 1000 HZ my laptop needs 31,3 W. With dyntick enabled I get 29.8 W, the pmstats-0.2 script shows me that the system is at 35-45 HZ when it is idle.
The power consumption difference between 250 HZ static and dyntick is below the noise, so maybe hardly worth all the struggle.

That's not the point. We want the power savings without sacrificing the extra ticks if we need them.

Cheers,
Con

Con Kolivas

author

2005-08-04 07:16 UTC #13 lore raw

Quoted 8 lines · Con Kolivas

Quoted 7 lines · Jim MacBaine
I just borrowed a power meter to see (or not to see) real effects of dyntick. The difference between static 1000 HZ and dynamic HZ is much less than I expected, only a very little about noise. With dyntick disabled at 1000 HZ my laptop needs 31,3 W. With dyntick enabled I get 29.8 W, the pmstats-0.2 script shows me that the system is at 35-45 HZ when it is idle.

Quoted 5 lines

Quoted 2 lines
The power consumption difference between 250 HZ static and dyntick is below the noise, so maybe hardly worth all the struggle.
That's not the point. We want the power savings without sacrificing the extra ticks if we need them.

Oh but thank you very much for confirming the power savings are around the 5% mark. If we don't measure we won't know (and everything else is mental masturbation according to Linus ;)).

Cheers,
Con

Tony Lindgren

2005-08-04 07:29 UTC #14 lore raw

Quoted 12 lines · Con Kolivas

Quoted 11 lines · Con Kolivas
Quoted 9 lines · Jim MacBaine
I just borrowed a power meter to see (or not to see) real effects of dyntick. The difference between static 1000 HZ and dynamic HZ is much less than I expected, only a very little about noise. With dyntick disabled at 1000 HZ my laptop needs 31,3 W. With dyntick enabled I get 29.8 W, the pmstats-0.2 script shows me that the system is at 35-45 HZ when it is idle.
The power consumption difference between 250 HZ static and dyntick is below the noise, so maybe hardly worth all the struggle.

Quoted 6 lines

Quoted 2 lines
That's not the point. We want the power savings without sacrificing the extra ticks if we need them.
Oh but thank you very much for confirming the power savings are around the 5% mark. If we don't measure we won't know (and everything else is mental masturbation according to Linus ;)).

Dyntick on it's own does not do much. But it allows adding better PM code later on.

Tony

Vojtech Pavlik

2005-08-04 15:07 UTC #4 lore raw

In reply to Jan De Luyck

Quoted 12 lines · Jan De Luyck

Quoted 11 lines · Con Kolivas
Quoted 9 lines · Jan De Luyck
Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
Compiles and runs ok here.
Is there actually any timer frequency that's advisable to set as maximum? (in the kernel config)

Quoted 8 lines

Quoted 1 line
I'd recommend 1000.
Thanks. Currently the system - under X, KDE, no artsd, bottoms at around 300HZ. In total single mode with every module unloaded that I can unload it stops around 22HZ.
I guess I'll have to go hunting whatever thing is causing the pollings. no timertop yet, I guess? :P

i8042 runs a steady periodic 20Hz timer. You can make it slower to get even the total low lower, and it will not affect performance under normal (sane hardware) circumstances.

Signature · 2 lines

Vojtech Pavlik
SuSE Labs, SuSE CR

Marc Ballarin

2005-08-04 16:26 UTC #24 lore raw

Quoted 5 lines · On Thu, 4 Aug 2005 01:22:36 +0200 Christian Leber <[email protected]> wrote:

Just a few numbers:
I tried it on a Laptop (Dell C810, P3m 1133 mhz) and measured the power usage with an external device and it stayed with or without patch at 27W. (HZ was at about 28)

Does your machine enter C3 state? Check the usage
in /proc/acpi/processor/CPU0/power

If usage is 0, unplug all USB peripherals (at least those connected to uhci controllers). Also shut down sound servers.

Without C3, there won't be any power savings from lower HZ. Desktop CPUs often don't support C3 at all.

Marc

Daniel Petrini

2005-08-04 21:17 UTC #28 lore raw

PATCH 1/1 Timer Top was: i386 No-Idle-Hz aka Dynamic-Ticks 3

Hi,

Here we have some support to have more tests on Dynamic Tick. We have some functions that exports timers information to a proc entry (/proc/top_info), in a kernel patch and a script that handles this info and give some output to analyse. We tried to make it less intrusive as possible.

It is based in suggestions from Tony Lindgren.

It is experimental and should evolve.

Must be applied after 2.6.13-rc5-dtck-3.patch and 2.6.13-rc5.

Usage: with kernel compiled with attached patch: "perl timer_top.pl 5", to have refresh time of 5s.

Regards,

Daniel Petrini

Signature · 3 lines

10LE - Linux Lab
Instituto Nokia de Tecnologia - INdT
Manaus - Brazil

kernel/Makefile +1 -1

diff -uprN linux-2.6.12-orig/kernel/Makefile linux-dyn-tick/kernel/Makefile
@@ -7,7 +7,7 @@ obj-y     = sched.o fork.o exec_domain.o
 	    sysctl.o capability.o ptrace.o timer.o user.o \
 	    signal.o sys.o kmod.o workqueue.o pid.o \
 	    rcupdate.o intermodule.o extable.o params.o posix-timers.o \
-	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o
+	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o timer_top.o
 
 obj-$(CONFIG_FUTEX) += futex.o
 obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o

kernel/timer.c +3 -0

diff -uprN linux-2.6.12-orig/kernel/timer.c linux-dyn-tick/kernel/timer.c
@@ -508,6 +508,8 @@ static inline void __run_timers(tvec_bas
 }
 
 #ifdef CONFIG_NO_IDLE_HZ
+extern struct timer_top_info top_info;
+extern int account_timer(unsigned int function, struct timer_top_info * top_info);
 /*
  * Find out when the next timer event is due to happen. This
  * is used on S/390 to stop all activity when a cpus is idle.

@@ -571,6 +573,7 @@ found:
 				expires = nte->expires;
 		}
 	}
+	account_timer((unsigned int)nte->function, &top_info);
 	spin_unlock(&base->t_base.lock);
 	return expires;
 }

kernel/timer_top.c +101 -0

101 lines not on this page · raw message

Adrian Bunk

2005-08-04 21:47 UTC #33 lore raw

arch/i386/kernel/time.c EXPORT_SYMBOL(profile_pc); · line 257

Quoted 8 lines · Con Kolivas

...

arch/i386/kernel/time.c +1 -1

...
-static inline void do_timer_interrupt(int irq, void *dev_id,
+inline void do_timer_interrupt(int irq, void *dev_id,
 					struct pt_regs *regs)
...

A global inline functions implies an increase of the size of the binary. Can you drop the "inline"?

cu
Adrian

Signature · 4 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Marc Ballarin

2005-08-04 22:14 UTC #34 lore raw

Quoted 3 lines · On Wed, 3 Aug 2005 15:59:24 +1000 Con Kolivas <[email protected]> wrote:

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5

One issue (tested the -rc4 Version on -mm):
- on interrupt flood (ping -f) HZ goes down to 0-4 HZ. This matches "ticks to skip" below. Coincidence?

- ping -f complains:
.Warning: time of day goes back (-304us), taking countermeasures.
...
.Warning: time of day goes back (-33us), taking countermeasures.

Yet, system time _seems_ to be kept correctly.

CPU is Pentium M.

dmesg:
Using pmtmr for high-res timesource
dyn-tick: Found suitable timer: pmtmr

dyn-tick: Maximum ticks to skip limited to 54
dyn-tick: Timer not enabled during boot

sysfs:

suitable:       1
enabled:        1
using APIC:     0

Regards

Con Kolivas

author

2005-08-05 00:35 UTC #35 lore raw

Quoted 11 lines · Marc Ballarin

On Wed, 3 Aug 2005 15:59:24 +1000
Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
One issue (tested the -rc4 Version on -mm):
- on interrupt flood (ping -f) HZ goes down to 0-4 HZ. This matches "ticks to skip" below. Coincidence?

Quoted 6 lines

- ping -f complains:
.Warning: time of day goes back (-304us), taking countermeasures.
...
.Warning: time of day goes back (-33us), taking countermeasures.
Yet, system time _seems_ to be kept correctly.

Interesting... It almost seems like if you throw enough interrupts at it the next_timer_interrupt function gets confused. I wonder if S390 and ARM are seeing this at all since (it seems to me) they use that function?

Con

Paul

2005-08-05 01:30 UTC #36 lore raw

In reply to Daniel Petrini

Quoted 7 lines · Con Kolivas <[email protected]>, on Wed Aug 03, 2005 [03:59:24 PM] said:

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.

Quoted 8 lines

The sysfs interface was moved to its own directory
in /sys/devices/system/dyn_tick and split into separate files to enable/disable dynamic ticks and usage of apic on the fly. It makes sense to enable dynamic ticks and usage of apic by default if they're actually built into the kernel so that is now done.
Cheers,
Con

	Hi;

	Ive been running this all afternoon on a pIIx2 @400mhz desktop

machine, SMP enabled Preempt kernel.
Initially, I tried it using APIC, got X up and tried to play
some music, but mplayer just hung and prompts in other terminals were super slugish or blocked until I killed mplayer. (there were no interrupts from the sound card during this time.) Without using APIC, it seems to work just great.
With my regular desktop running and me not doing much, 'vmstat 5'
shows interrupts at 100-200 per entry. Loading the box completely with 'make -j4' kernel build their are just over 1000 interrupts. Various other loads show numbers in between. (eg. it seems to work as one would expect, presuming the timer interupts dominate...)
Time seems fine. I havent noticed any interactivity problems.
Performance on 'kernbench' was within a percentage point of 2.6.12
It does not seem to make any difference as far as the heat of the
cpus or mb, though.

Hope this datapoint is useful;
Paul
[email protected]

Con Kolivas

author

2005-08-05 03:30 UTC #37 lore raw

Quoted 5 lines · Paul

Quoted 3 lines · Con Kolivas <[email protected]>, on Wed Aug 03, 2005 [03:59:24 PM] said:
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5

Quoted 16 lines

Ive been running this all afternoon on a pIIx2 @400mhz desktop
machine, SMP enabled Preempt kernel.
Initially, I tried it using APIC, got X up and tried to play
some music, but mplayer just hung and prompts in other terminals were super slugish or blocked until I killed mplayer. (there were no interrupts from the sound card during this time.) Without using APIC, it seems to work just great.
With my regular desktop running and me not doing much, 'vmstat 5'
shows interrupts at 100-200 per entry. Loading the box completely with 'make -j4' kernel build their are just over 1000 interrupts. Various other loads show numbers in between. (eg. it seems to work as one would expect, presuming the timer interupts dominate...)
Time seems fine. I havent noticed any interactivity problems.
Performance on 'kernbench' was within a percentage point of 2.6.12
It does not seem to make any difference as far as the heat of the
cpus or mb, though.

Thanks for the feedback. As Tony mentioned, it simply allows for better power management to be done rather than saving power on its own. At idle it allows the processor to enter a lower power state for example, which I doubt the pII does.

Cheers,
Con

Con Kolivas

author

2005-08-05 04:11 UTC #29 lore raw

PATCH 1/1 Timer Top tweaks

Quoted 8 lines · Daniel Petrini

Hi,
Here we have some support to have more tests on Dynamic Tick. We have some functions that exports timers information to a proc entry (/proc/top_info), in a kernel patch and a script that handles this info and give some output to analyse. We tried to make it less intrusive as possible.

Quoted 8 lines

It is based in suggestions from Tony Lindgren.
It is experimental and should evolve.
Must be applied after 2.6.13-rc5-dtck-3.patch and 2.6.13-rc5.
Usage: with kernel compiled with attached patch: "perl timer_top.pl 5", to have refresh time of 5s.

Yes that's very nice.

It's probably premature but here's a small patch to your timer_top patch.

Cheers,
Con
---

kernel/Makefile +2 -2

Index: linux-2.6.13-rc5-ck2/kernel/Makefile
===================================================================
@@ -7,7 +7,7 @@ obj-y     = sched.o fork.o exec_domain.o
 	    sysctl.o capability.o ptrace.o timer.o user.o \
 	    signal.o sys.o kmod.o workqueue.o pid.o \
 	    rcupdate.o intermodule.o extable.o params.o posix-timers.o \
-	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o timer_top.o
+	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o
 
 obj-$(CONFIG_FUTEX) += futex.o
 obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o

@@ -30,7 +30,7 @@ obj-$(CONFIG_SYSFS) += ksysfs.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
 obj-$(CONFIG_SECCOMP) += seccomp.o
-obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o
+obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o timer_top.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

kernel/timer.c +3 -1

Index: linux-2.6.13-rc5-ck2/kernel/timer.c
===================================================================
@@ -509,7 +509,9 @@ static inline void __run_timers(tvec_bas
 
 #ifdef CONFIG_NO_IDLE_HZ
 extern struct timer_top_info top_info;
-extern int account_timer(unsigned int function, struct timer_top_info * top_info);
+extern int account_timer(unsigned int function,
+			struct timer_top_info *top_info);
+
 /*
  * Find out when the next timer event is due to happen. This
  * is used on S/390 to stop all activity when a cpus is idle.

kernel/timer_top.c +14 ~~-19~~

Index: linux-2.6.13-rc5-ck2/kernel/timer_top.c
===================================================================
@@ -11,7 +11,6 @@
  * published by the Free Software Foundation.
  */
 
-
 #include <linux/list.h>
 #include <linux/proc_fs.h>
 #include <linux/module.h>

@@ -20,8 +19,8 @@ static LIST_HEAD(timer_list);
 
 struct timer_top_info {
 	unsigned int		func_pointer;
-	unsigned int long	counter;
-	struct list_head 	list;      	
+	unsigned long		counter;
+	struct list_head 	list;
 };
 
 struct timer_top_info top_info;

@@ -30,37 +29,38 @@ int account_timer(unsigned int function,
 {
 	struct timer_top_info *top;
 
-	list_for_each_entry (top, &timer_list, list) {
+	list_for_each_entry(top, &timer_list, list) {
 		/* if it is in the list increment its count */
 		if (top->func_pointer == function) {
-			top->counter += 1;
-			return 0;
+			top->counter++;
+			goto out;
 		}
 	}
-	
+
 	/* if you are here then it didnt find so inserts in the list */
 
 	top = kmalloc(sizeof(struct timer_top_info), GFP_KERNEL);
-	if (!top) 
+	if (!top)
 		return -ENOMEM;
 	top->func_pointer = function;
 	top->counter = 1;
 	list_add(&top->list, &timer_list);
 
+out:
 	return 0;
 }
 
 EXPORT_SYMBOL(account_timer);
 
 struct top_info_poll {
-  char value[18];
+	char value[18];
 };
 
 struct top_info_poll top_info_poll_dt;
 struct proc_dir_entry *top_info_file;
 
 static int proc_read_top_info(char *page, char **start, off_t off,
-                                int count, int *eof, void *data)
+				int count, int *eof, void *data)
 {
 	char aux[18];
 	struct timer_top_info *top;

@@ -69,21 +69,19 @@ static int proc_read_top_info(char *page
 
 	sprintf(page, "Function counter - %s\n", info_poll_data->value);
 
-	list_for_each_entry (top, &timer_list, list) {
+	list_for_each_entry(top, &timer_list, list) {
 		sprintf(aux, "%x %lu\n", top->func_pointer, top->counter);
 		strcat(page, aux);
 	}
 
 	return strlen(page);
- 
-} 
+}
 
 static int init_top_info(void)
 {
 	top_info_file = create_proc_entry("top_info", 0666, NULL);
-	if(top_info_file == NULL) {
-	  return -ENOMEM;
-	}
+	if (top_info_file == NULL)
+		return -ENOMEM;
 
 	strcpy(top_info_poll_dt.value, "Timer Top v0.9.1");

@@ -96,6 +94,3 @@ static int init_top_info(void)
 
 module_init(init_top_info);
 //module_exit();
-
-
-

Con Kolivas

author

2005-08-05 05:16 UTC #5 lore raw

In reply to Vojtech Pavlik

Quoted 19 lines · Vojtech Pavlik

Quoted 18 lines · Jan De Luyck
Quoted 6 lines · Con Kolivas
Quoted 5 lines · Jan De Luyck
Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
Quoted 6 lines
Quoted 4 lines
Compiles and runs ok here.
Is there actually any timer frequency that's advisable to set as maximum? (in the kernel config)
I'd recommend 1000.
Thanks. Currently the system - under X, KDE, no artsd, bottoms at around 300HZ. In total single mode with every module unloaded that I can unload it stops around 22HZ.

Quoted 6 lines

Quoted 2 lines
I guess I'll have to go hunting whatever thing is causing the pollings. no timertop yet, I guess? :P
i8042 runs a steady periodic 20Hz timer. You can make it slower to get even the total low lower, and it will not affect performance under normal (sane hardware) circumstances.

Indeed and this patch (safely tried at home but serves no useful purpose really) confirms a reasonable drop without problems. After this, only fbcon polls at a similar rate (HZ/5).

Of interest to those using an ondemand scaling governor, now that we have timertop, I have found that the default ondemand settings lead to delayed_work_timer_fn at about 140Hz and this can be dropped substantially by slowing the rate of polling (and subsequently slowing the speed with which the ondemand governor responds) down to <25 by

echo 100000 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/sampling_rate

(the default is 10000 and the value is confusing as the rate goes down as you increase this value).

Cheers,
Con

drivers/input/serio/i8042.h +1 -1

Index: linux-2.6.13-rc5-ck2/drivers/input/serio/i8042.h
===================================================================
@@ -44,7 +44,7 @@
  * polling.
  */
 
-#define I8042_POLL_PERIOD	HZ/20
+#define I8042_POLL_PERIOD	HZ/5
 
 /*
  * Status register bits.

Jens Axboe

2005-08-05 06:46 UTC #30 lore raw

Re: [ck] [PATCH] Timer Top was: i386 No-Idle-Hz aka Dynamic-Ticks 3

In reply to Daniel Petrini

Quoted 11 lines · Daniel Petrini

kernel/timer_top.c +11 -0

+static LIST_HEAD(timer_list);
+
+struct timer_top_info {
+	unsigned int		func_pointer;
+	unsigned int long	counter;
+	struct list_head 	list;      	
+};
+
+struct timer_top_info top_info;
+
+int account_timer(unsigned int function, struct timer_top_info * top_info)

kernel/timer_top.c line 39

Quoted 10 lines

kernel/timer_top.c +10 -0

+{
+	struct timer_top_info *top;
+
+	list_for_each_entry (top, &timer_list, list) {
+		/* if it is in the list increment its count */
+		if (top->func_pointer == function) {
+			top->counter += 1;
+			return 0;
+		}
+	}

What protects this list?

kernel/timer_top.c line 45

Quoted 6 lines

kernel/timer_top.c +6 -0

+	
+	/* if you are here then it didnt find so inserts in the list */
+
+	top = kmalloc(sizeof(struct timer_top_info), GFP_KERNEL);
+	if (!top) 
+		return -ENOMEM;

You can't use GFP_KERNEL here, you are inside the timer base lock.

Signature · 1 line

Jens Axboe

Srivatsa Vaddagiri

2005-08-05 12:37 UTC #38 lore raw

Quoted 6 lines · Con Kolivas

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.

Con,
I have been looking at some of the requirement of tickless idle CPUs in
core kernel areas like scheduler and RCU. Basically, both power management and virtualization benefit if idle CPUs can cut off useless timer ticks. Especially from a virtualization standpoint, I think it makes sense that we enable this feature on a per-CPU basis i.e let individual CPUs cut off their ticks as and when they become idle. The benefit of this is more visible in platforms that host lot of (SMP) VMs on the same machine. Most of the time, these VMs may be partially idle (some CPUs in it are idle, some not) and it is good that we quiesce the timer ticks on the partial set of idle CPUs. Both S390 and Xen ports of Linux kernel have this ability today (S390 has it in mainline already and Xen has it out of tree).

Quoted 1 line

From this viewpoint, I think the current implementation of dynamic tick

falls short of this requirement. It cuts of the timer ticks only when all CPUs go idle.

Apart from this observation, I have some others about the current dynamic tick patch:

- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't this wrong, considering that the timer list is per-CPU? This will cause some timers to be serviced much later than usual.

- The fact that dyn_tick_state is global and accessed from all CPUs is probably a scalability concern, especially if we allow the ticks to be cut off on per-CPU basis.

- Again, when we allow this on a per-CPU basis, subsystems like RCU need to know the partial set of idle CPUs. RCU already does that thr' nohz_cpu_mask (which will need to replace dyn_cpu_map).

- Looking at dyn_tick_timer_interrupt, would it be nice if we avoid calling do_timer_interrupt so many times and instead update jiffies to (skipped_ticks - 1) and then call do_timer_interrupt once? I think VST does it that way.

- dyn_tick->max_skip = 0xffffff / apic_timer_val;
From my reading of Intel docs, APIC_TMICT is 32-bit. So why does the
above calculation take only 24-bits into account? What am I missing here?

I can take a shot at addressing these concerns in dynamic_tick patch, but it seems to me that VST has already addressed all these to a big extent. Had you considered VST before? The biggest bottleneck I see in VST going mainline is its dependency on HRT patch but IMO it should be possible to write a small patch to support VST w/o HRT.

George, what do you think?

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Daniel Petrini

2005-08-05 12:39 UTC #31 lore raw

Re: [ck] [PATCH] Timer Top was: i386 No-Idle-Hz aka Dynamic-Ticks 3

In reply to Jens Axboe

Hi,

Quoted 3 lines

--
Jens Axboe

Thanks for your corrections. Here we have a new version.

Daniel Petrini

Signature · 2 lines

10LE - Linux
INdT - Manaus - Brazil

kernel/Makefile +1 -1

diff -uprN linux-2.6.12-orig/kernel/Makefile linux-dyn-tick/kernel/Makefile
@@ -7,7 +7,7 @@ obj-y     = sched.o fork.o exec_domain.o
 	    sysctl.o capability.o ptrace.o timer.o user.o \
 	    signal.o sys.o kmod.o workqueue.o pid.o \
 	    rcupdate.o intermodule.o extable.o params.o posix-timers.o \
-	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o
+	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o timer_top.o
 
 obj-$(CONFIG_FUTEX) += futex.o
 obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o

kernel/timer.c +3 -0

diff -uprN linux-2.6.12-orig/kernel/timer.c linux-dyn-tick/kernel/timer.c
@@ -508,6 +508,8 @@ static inline void __run_timers(tvec_bas
 }
 
 #ifdef CONFIG_NO_IDLE_HZ
+extern struct timer_top_info top_info;
+extern int account_timer(unsigned int function, struct timer_top_info * top_info);
 /*
  * Find out when the next timer event is due to happen. This
  * is used on S/390 to stop all activity when a cpus is idle.

@@ -571,6 +573,7 @@ found:
 				expires = nte->expires;
 		}
 	}
+	account_timer((unsigned int)nte->function, &top_info);
 	spin_unlock(&base->t_base.lock);
 	return expires;
 }

kernel/timer_top.c +111 -0

111 lines not on this page · raw message

Con Kolivas

author

2005-08-05 13:17 UTC #39 lore raw

In reply to Daniel Petrini

Quoted 9 lines · Srivatsa Vaddagiri

Quoted 7 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.

Quoted 12 lines

Con,
I have been looking at some of the requirement of tickless idle CPUs in
core kernel areas like scheduler and RCU. Basically, both power management and virtualization benefit if idle CPUs can cut off useless timer ticks. Especially from a virtualization standpoint, I think it makes sense that we enable this feature on a per-CPU basis i.e let individual CPUs cut off their ticks as and when they become idle. The benefit of this is more visible in platforms that host lot of (SMP) VMs on the same machine. Most of the time, these VMs may be partially idle (some CPUs in it are idle, some not) and it is good that we quiesce the timer ticks on the partial set of idle CPUs. Both S390 and Xen ports of Linux kernel have this ability today (S390 has it in mainline already and Xen has it out of tree).

Hi Srivatsa.

Thanks very much for your comments. The actual codebase for dynticks I've never staked any claim to as it isn't mine. The demand for such a feature in the kernel on multiple architectures appears to have reached critical mass with mainline now considering another Hz change. This is why I decided to champion this patch at this time by polishing and tweaking it (in purely cosmetic ways). I mostly agree with your comments about limitations of this patch and would like to see changes like those you describe done in an arch-neutral fashion as well. I have not the hardware resources to develop such code and would be more than happy if have generated enough interest in dynticks for you to develop it further via whatever means. Knowing arm and S390 already have dynticks makes it more important that cross-arch code is consolidated sooner rather than later.

Cheers,
Con

Daniel Petrini

2005-08-05 13:55 UTC #32 lore raw

Re: [ck] [PATCH] Timer Top was: i386 No-Idle-Hz aka Dynamic-Ticks 3

Hi,

Here we have a new version that includes Jens Axboe's corrections and Con Kolivas tweaks.

Thanks,

Daniel

Signature · 2 lines

10LE - Linux
INdT - Manaus - Brazil

kernel/Makefile +1 -1

diff -uprN linux-2.6.12-orig/kernel/Makefile linux-dyn-tick/kernel/Makefile
@@ -30,7 +30,7 @@ obj-$(CONFIG_SYSFS) += ksysfs.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
 obj-$(CONFIG_SECCOMP) += seccomp.o
-obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o
+obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o timer_top.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

kernel/timer.c +4 -0

diff -uprN linux-2.6.12-orig/kernel/timer.c linux-dyn-tick/kernel/timer.c
@@ -508,6 +508,9 @@ static inline void __run_timers(tvec_bas
 }
 
 #ifdef CONFIG_NO_IDLE_HZ
+extern struct timer_top_info top_info;
+extern int account_timer(unsigned int function,
+	       		struct timer_top_info * top_info);
 /*
  * Find out when the next timer event is due to happen. This
  * is used on S/390 to stop all activity when a cpus is idle.

@@ -571,6 +574,7 @@ found:
 				expires = nte->expires;
 		}
 	}
+	account_timer((unsigned int)nte->function, &top_info);
 	spin_unlock(&base->t_base.lock);
 	return expires;
 }

kernel/timer_top.c +108 -0

108 lines not on this page · raw message

Con Kolivas

author

2005-08-05 16:51 UTC #40 lore raw

PATCH 1/1 i386 No-Idle-Hz aka Dynamic-Ticks 4

Here's my most current version of the dynamic ticks patch for i386 with some more minor cleanups already discussed and cosmetic changes ( also available at http://ck.kolivas.org/patches/dyn-ticks/ ).

Cheers,
Con
---

arch/i386/Kconfig +35 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/Kconfig
===================================================================
@@ -457,6 +457,41 @@ config HPET_EMULATE_RTC
 	bool "Provide RTC interrupt"
 	depends on HPET_TIMER && RTC=y
 
+config NO_IDLE_HZ
+	bool "Dynamic Tick Timer - Skip timer ticks during idle"
+	depends on EXPERIMENTAL
+	help
+	  This option enables support for skipping timer ticks when the
+	  processor is idle. During system load, timer is continuous.
+	  This option saves power, as it allows the system to stay in
+	  idle mode longer. Currently supported timers are ACPI PM
+	  timer, local APIC timer, and TSC timer. HPET timer is currently
+	  not supported.
+
+	  Note that you can disable dynamic tick timer either by
+	  passing dyntick=disable command line option, or via sysfs:
+
+	  # echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/enable
+
+config DYN_TICK_USE_APIC
+	bool "Use APIC timer instead of PIT timer"
+	depends on NO_IDLE_HZ
+	help
+	  This option enables using APIC timer interrupt if your hardware
+	  supports it. APIC timer allows longer sleep periods compared
+	  to PIT timer, however on MOST recent hardware disabling the PIT
+	  timer also disables APIC timer interrupts, and the system won't
+	  run properly. Symptoms include slow system boot, and time running 
+	  slow.
+
+	  If unsure, do NOT enable this option.
+
+	  Note that you can disable apic usage by dynamic tick timer
+	  either by passing dyntick=noapic command line option, or via 
+	  sysfs:
+
+	  # echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/useapic
+
 config SMP
 	bool "Symmetric multi-processing support"
 	---help---

arch/i386/kernel/apic.c +18 -1

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/apic.c
===================================================================
@@ -27,6 +27,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/sysdev.h>
 #include <linux/cpu.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/atomic.h>
 #include <asm/smp.h>

@@ -931,6 +932,8 @@ void (*wait_timer_tick)(void) __devinitd
 
 #define APIC_DIVISOR 16
 
+u32 apic_timer_val;
+
 static void __setup_APIC_LVTT(unsigned int clocks)
 {
 	unsigned int lvtt_value, tmp_value, ver;

@@ -949,7 +952,12 @@ static void __setup_APIC_LVTT(unsigned i
 				& ~(APIC_TDR_DIV_1 | APIC_TDR_DIV_TMBASE))
 				| APIC_TDR_DIV_16);
 
-	apic_write_around(APIC_TMICT, clocks/APIC_DIVISOR);
+	apic_timer_val = clocks / APIC_DIVISOR;
+
+	if (apic_timer_val)
+		set_dyn_tick_max_skip(apic_timer_val);
+
+	apic_write_around(APIC_TMICT, apic_timer_val);
 }
 
 static void __devinit setup_APIC_timer(unsigned int clocks)

@@ -1062,6 +1070,8 @@ void __init setup_boot_APIC_clock(void)
 	 */
 	setup_APIC_timer(calibration_result);
 
+	setup_dyn_tick_use_apic(calibration_result);
+
 	local_irq_enable();
 }

@@ -1200,6 +1210,13 @@ fastcall void smp_apic_timer_interrupt(s
 	 * interrupt lock, which is the WrongThing (tm) to do.
 	 */
 	irq_enter();
+
+	/*
+	 * Check if we need to wake up PIT interrupt handler.
+	 * Otherwise just wake up local APIC timer.
+	 */
+	wakeup_pit_or_apic(cpu, regs);
+
 	smp_local_timer_interrupt(regs);
 	irq_exit();
 }

arch/i386/kernel/dyn-tick.c +150 -0

150 lines not on this page · raw message

arch/i386/kernel/irq.c +3 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/irq.c
===================================================================
@@ -18,6 +18,7 @@
 #include <linux/notifier.h>
 #include <linux/cpu.h>
 #include <linux/delay.h>
+#include <linux/dyn-tick.h>
 
 DEFINE_PER_CPU(irq_cpustat_t, irq_stat) ____cacheline_maxaligned_in_smp;
 EXPORT_PER_CPU_SYMBOL(irq_stat);

@@ -76,6 +77,8 @@ fastcall unsigned int do_IRQ(struct pt_r
 	}
 #endif
 
+	dyn_tick_interrupt(irq, regs);
+
 #ifdef CONFIG_4KSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();

arch/i386/kernel/Makefile +1 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/Makefile
===================================================================
@@ -32,6 +32,7 @@ obj-$(CONFIG_MODULES)		+= module.o
 obj-y				+= sysenter.o vsyscall.o
 obj-$(CONFIG_ACPI_SRAT) 	+= srat.o
 obj-$(CONFIG_HPET_TIMER) 	+= time_hpet.o
+obj-$(CONFIG_NO_IDLE_HZ) 	+= dyn-tick.o
 obj-$(CONFIG_EFI) 		+= efi.o efi_stub.o
 obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o

arch/i386/kernel/process.c +3 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/process.c
===================================================================
@@ -39,6 +39,7 @@
 #include <linux/ptrace.h>
 #include <linux/random.h>
 #include <linux/kprobes.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>

@@ -200,6 +201,8 @@ void cpu_idle(void)
 			if (cpu_is_offline(cpu))
 				play_dead();
 
+			dyn_tick_reprogram_timer();
+
 			__get_cpu_var(irq_stat).idle_timestamp = jiffies;
 			idle();
 		}

arch/i386/kernel/time.c +5 -3

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/time.c
===================================================================
@@ -46,6 +46,7 @@
 #include <linux/bcd.h>
 #include <linux/efi.h>
 #include <linux/mca.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/io.h>
 #include <asm/smp.h>

@@ -252,8 +253,7 @@ EXPORT_SYMBOL(profile_pc);
  * timer_interrupt() needs to keep up the real-time clock,
  * as well as call the "do_timer()" routine every clocktick
  */
-static inline void do_timer_interrupt(int irq, void *dev_id,
-					struct pt_regs *regs)
+void do_timer_interrupt(int irq, void *dev_id, struct pt_regs *regs)
 {
 #ifdef CONFIG_X86_IO_APIC
 	if (timer_ack) {

@@ -423,7 +423,7 @@ static struct sysdev_class timer_sysclas
 
 
 /* XXX this driverfs stuff should probably go elsewhere later -john */
-static struct sys_device device_timer = {
+struct sys_device device_timer = {
 	.id	= 0,
 	.cls	= &timer_sysclass,
 };

@@ -479,5 +479,7 @@ void __init time_init(void)
 	cur_timer = select_timer();
 	printk(KERN_INFO "Using %s for high-res timesource\n",cur_timer->name);
 
+	dyn_tick_time_init(cur_timer);
+
 	time_init_hook();
 }

arch/i386/kernel/timers/timer_pit.c +38 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_pit.c
===================================================================
@@ -148,6 +148,44 @@ static unsigned long get_offset_pit(void
 	return count;
 }
 
+/*
+ * REVISIT: Looks like on P3 APIC timer keeps running if PIT mode
+ *	    is changed. On P4, changing PIT mode seems to kill
+ *	    APIC timer interrupts. Same thing with disabling PIT
+ *	    interrupt.
+ */
+void disable_pit_timer(void)
+{
+	extern spinlock_t i8253_lock;
+	unsigned long flags;
+
+	spin_lock_irqsave(&i8253_lock, flags);
+	outb_p(0x32, PIT_MODE);		/* binary, mode 1, LSB/MSB, ch 0 */
+	spin_unlock_irqrestore(&i8253_lock, flags);
+}
+
+/*
+ * Reprograms the next timer interrupt
+ * PIT timer reprogramming code taken from APM code.
+ * Note that PIT timer is a 16-bit timer, which allows max
+ * skip of only few seconds.
+ */
+void reprogram_pit_timer(int jiffies_to_skip)
+{
+	int skip;
+	extern spinlock_t i8253_lock;
+	unsigned long flags;
+
+	skip = jiffies_to_skip * LATCH;
+	if (skip > 0xffff)
+		skip = 0xffff;
+
+	spin_lock_irqsave(&i8253_lock, flags);
+	outb_p(0x34, PIT_MODE);		/* binary, mode 2, LSB/MSB, ch 0 */
+	outb_p(skip & 0xff, PIT_CH0);	/* LSB */
+	outb(skip >> 8, PIT_CH0);	/* MSB */
+	spin_unlock_irqrestore(&i8253_lock, flags);
+}
 
 /* tsc timer_opts struct */
 struct timer_opts timer_pit = {

arch/i386/kernel/timers/timer_pm.c +4 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_pm.c
===================================================================
@@ -15,6 +15,7 @@
 #include <linux/module.h>
 #include <linux/device.h>
 #include <linux/init.h>
+#include <linux/dyn-tick.h>
 #include <asm/types.h>
 #include <asm/timer.h>
 #include <asm/smp.h>

@@ -168,6 +169,9 @@ static void mark_offset_pmtmr(void)
 	monotonic_base += delta * NSEC_PER_USEC;
 	write_sequnlock(&monotonic_lock);
 
+	if (dyn_tick_enabled())
+		return;
+
 	/* convert to ticks */
 	delta += offset_delay;
 	lost = delta / (USEC_PER_SEC / HZ);

arch/i386/kernel/timers/timer_tsc.c +21 -8

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_tsc.c
===================================================================
@@ -14,6 +14,7 @@
 #include <linux/cpufreq.h>
 #include <linux/string.h>
 #include <linux/jiffies.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/timer.h>
 #include <asm/io.h>

@@ -166,10 +167,19 @@ static void delay_tsc(unsigned long loop
 	} while ((now-bclock) < loops);
 }
 
+/* update the monotonic base value */
+static inline void update_monotonic_base(unsigned long long last_offset)
+{
+	unsigned long long this_offset;
+
+	this_offset = ((unsigned long long)last_tsc_high << 32) | last_tsc_low;
+	monotonic_base += cycles_2_ns(this_offset - last_offset);
+}
+
 #ifdef CONFIG_HPET_TIMER
 static void mark_offset_tsc_hpet(void)
 {
-	unsigned long long this_offset, last_offset;
+	unsigned long long last_offset;
  	unsigned long offset, temp, hpet_current;
 
 	write_seqlock(&monotonic_lock);

@@ -197,9 +207,7 @@ static void mark_offset_tsc_hpet(void)
 	}
 	hpet_last = hpet_current;
 
-	/* update the monotonic base value */
-	this_offset = ((unsigned long long)last_tsc_high<<32)|last_tsc_low;
-	monotonic_base += cycles_2_ns(this_offset - last_offset);
+	update_monotonic_base(last_offset);
 	write_sequnlock(&monotonic_lock);
 
 	/* calculate delay_at_last_interrupt */

@@ -346,7 +354,7 @@ static void mark_offset_tsc(void)
 	int count;
 	int countmp;
 	static int count1 = 0;
-	unsigned long long this_offset, last_offset;
+	unsigned long long last_offset;
 	static int lost_count = 0;
 
 	write_seqlock(&monotonic_lock);

@@ -367,6 +375,12 @@ static void mark_offset_tsc(void)
 
 	rdtsc(last_tsc_low, last_tsc_high);
 
+	if (dyn_tick_enabled()) {
+		update_monotonic_base(last_offset);
+		write_sequnlock(&monotonic_lock);
+		return;
+	}
+
 	spin_lock(&i8253_lock);
 	outb_p(0x00, PIT_MODE);     /* latch the count ASAP */

@@ -434,9 +448,8 @@ static void mark_offset_tsc(void)
 			cpufreq_delayed_get();
 	} else
 		lost_count = 0;
-	/* update the monotonic base value */
-	this_offset = ((unsigned long long)last_tsc_high<<32)|last_tsc_low;
-	monotonic_base += cycles_2_ns(this_offset - last_offset);
+
+	update_monotonic_base(last_offset);
 	write_sequnlock(&monotonic_lock);
 
 	/* calculate delay_at_last_interrupt */

arch/i386/mach-default/setup.c +16 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/mach-default/setup.c
===================================================================
@@ -93,6 +93,22 @@ void __init time_init_hook(void)
 	setup_irq(0, &irq0);
 }
 
+/**
+ * replace_timer_interrupt - allow replacing timer interrupt handler
+ *
+ * Description:
+ *	Can be used to replace timer interrupt handler with a more optimized
+ *	handler. Used for enabling and disabling of CONFIG_NO_IDLE_HZ.
+ */
+void replace_timer_interrupt(void *new_handler)
+{
+	unsigned long flags;
+
+	write_seqlock_irqsave(&xtime_lock, flags);
+	irq0.handler = new_handler;
+	write_sequnlock_irqrestore(&xtime_lock, flags);
+}
+
 #ifdef CONFIG_MCA
 /**
  * mca_nmi_hook - hook into MCA specific NMI chain

include/asm-i386/dyn-tick.h +85 -0

85 lines not on this page · raw message

include/linux/dyn-tick.h +61 -0

61 lines not on this page · raw message

kernel/dyn-tick.c +273 -0

273 lines not on this page · raw message

kernel/Makefile +1 -0

Index: linux-2.6.13-rc5-ck2/kernel/Makefile
===================================================================
@@ -30,6 +30,7 @@ obj-$(CONFIG_SYSFS) += ksysfs.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
 obj-$(CONFIG_SECCOMP) += seccomp.o
+obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

Adrian Bunk

2005-08-06 17:47 UTC #41 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 4

arch/i386/kernel/dyn-tick.c

Quoted 12 lines · Con Kolivas

Here's my most current version of the dynamic ticks patch for i386 with some more minor cleanups already discussed and cosmetic changes ( also available at http://ck.kolivas.org/patches/dyn-ticks/ ).
Cheers,
Con
...
arch/i386/kernel/dyn-tick.c +2 -0
...
+void arch_reprogram_timer(void)
+{

static?

arch/i386/kernel/dyn-tick.c

Quoted 3 lines

...
arch/i386/kernel/dyn-tick.c +2 -0
+int __init dyn_tick_init(void)
+{

static?

arch/i386/kernel/dyn-tick.c line 59

Quoted 4 lines

...

arch/i386/kernel/dyn-tick.c +3 -0

+irqreturn_t dyn_tick_timer_interrupt(int irq, void *dev_id,
+				     struct pt_regs *regs)
+{

static?

arch/i386/kernel/dyn-tick.c

Quoted 3 lines

...
arch/i386/kernel/dyn-tick.c +2 -0
+int __init dyn_tick_arch_init(void)
+{

static?

Quoted 4 lines

...

arch/i386/kernel/dyn-tick.c +3 -0

+/* Functions that need blank prototypes for !CONFIG_NO_IDLE_HZ below here */
+inline void set_dyn_tick_max_skip(u32 apic_timer_val)
+{

arch/i386/kernel/dyn-tick.c line 116

Quoted 10 lines

arch/i386/kernel/dyn-tick.c +10 -0

+	dyn_tick->max_skip = 0xffffff / apic_timer_val;
+}
+
+inline void setup_dyn_tick_use_apic(unsigned int calibration_result)
+{
+	if (calibration_result)
+		dyn_tick->state |= DYN_TICK_USE_APIC;
+	else
+		printk(KERN_INFO "dyn-tick: Cannot use local APIC\n");
+}

You can drop the "inline" on both functions (it doesn't make any practical difference since they are only called from other files).

Quoted 6 lines

...

include/asm-i386/dyn-tick.h +2 -0

...
+#if defined(CONFIG_DYN_TICK_USE_APIC) && \
+	(defined(CONFIG_SMP) || defined(CONFIG_X86_UP_APIC))

include/asm-i386/dyn-tick.h line 45

Quoted 10 lines

include/asm-i386/dyn-tick.h +10 -0

+#define cpu_has_local_apic()	(dyn_tick->state & DYN_TICK_USE_APIC)
+#else
+#define cpu_has_local_apic()	0
+#endif
+
+#if defined(CONFIG_DYN_TICK_USE_APIC)
+#define dyntick_apicable()	1
+#else
+#define dyntick_apicable()	0
+#endif

Constants that mimic functions aren't good coding style.

Please make them either uppercase #define's without function braces or "static inline" functions.

kernel/dyn-tick.c line 32

Quoted 5 lines

...
kernel/dyn-tick.c +1 -0
...
+struct dyn_tick_state dyn_tick_state;

static?

Quoted 3 lines

...
+struct dyn_tick_timer *dyn_tick_cfg;
...

static?

cu
Adrian

Signature · 4 lines

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Con Kolivas

author

2005-08-07 05:14 UTC #42 lore raw

PATCH 1/1 i386 No-Idle-Hz aka Dynamic-Ticks 5

Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?

Cheers,
Con
---

This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>, and modified by Con Kolivas <[email protected]>.

Signed-off-by Con Kolivas <[email protected]>

 arch/i386/Kconfig                   |   35 ++++
 arch/i386/kernel/Makefile           |    1
 arch/i386/kernel/apic.c             |   19 ++
 arch/i386/kernel/dyn-tick.c         |  150 +++++++++++++++++++
 arch/i386/kernel/irq.c              |    3
 arch/i386/kernel/process.c          |    3
 arch/i386/kernel/time.c             |    8 -
 arch/i386/kernel/timers/timer_pit.c |   38 +++++
 arch/i386/kernel/timers/timer_pm.c  |    4
 arch/i386/kernel/timers/timer_tsc.c |   29 ++-
 arch/i386/mach-default/setup.c      |   16 ++
 include/asm-i386/dyn-tick.h         |   93 ++++++++++++
 include/linux/dyn-tick.h            |   75 +++++++++
 kernel/Makefile                     |    1
 kernel/dyn-tick.c                   |  273 ++++++++++++++++++++++++++++++++++++
 15 files changed, 736 insertions(+), 12 deletions(-)

arch/i386/Kconfig +35 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/Kconfig
===================================================================
@@ -457,6 +457,41 @@ config HPET_EMULATE_RTC
 	bool "Provide RTC interrupt"
 	depends on HPET_TIMER && RTC=y
 
+config NO_IDLE_HZ
+	bool "Dynamic Tick Timer - Skip timer ticks during idle"
+	depends on EXPERIMENTAL
+	help
+	  This option enables support for skipping timer ticks when the
+	  processor is idle. During system load, timer is continuous.
+	  This option saves power, as it allows the system to stay in
+	  idle mode longer. Currently supported timers are ACPI PM
+	  timer, local APIC timer, and TSC timer. HPET timer is currently
+	  not supported.
+
+	  Note that you can disable dynamic tick timer either by
+	  passing dyntick=disable command line option, or via sysfs:
+
+	  # echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/enable
+
+config DYN_TICK_USE_APIC
+	bool "Use APIC timer instead of PIT timer"
+	depends on NO_IDLE_HZ
+	help
+	  This option enables using APIC timer interrupt if your hardware
+	  supports it. APIC timer allows longer sleep periods compared
+	  to PIT timer, however on MOST recent hardware disabling the PIT
+	  timer also disables APIC timer interrupts, and the system won't
+	  run properly. Symptoms include slow system boot, and time running 
+	  slow.
+
+	  If unsure, do NOT enable this option.
+
+	  Note that you can disable apic usage by dynamic tick timer
+	  either by passing dyntick=noapic command line option, or via 
+	  sysfs:
+
+	  # echo 0 > /sys/devices/system/dyn_tick/dyn_tick0/useapic
+
 config SMP
 	bool "Symmetric multi-processing support"
 	---help---

arch/i386/kernel/apic.c +18 -1

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/apic.c
===================================================================
@@ -27,6 +27,7 @@
 #include <linux/kernel_stat.h>
 #include <linux/sysdev.h>
 #include <linux/cpu.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/atomic.h>
 #include <asm/smp.h>

@@ -931,6 +932,8 @@ void (*wait_timer_tick)(void) __devinitd
 
 #define APIC_DIVISOR 16
 
+u32 apic_timer_val;
+
 static void __setup_APIC_LVTT(unsigned int clocks)
 {
 	unsigned int lvtt_value, tmp_value, ver;

@@ -949,7 +952,12 @@ static void __setup_APIC_LVTT(unsigned i
 				& ~(APIC_TDR_DIV_1 | APIC_TDR_DIV_TMBASE))
 				| APIC_TDR_DIV_16);
 
-	apic_write_around(APIC_TMICT, clocks/APIC_DIVISOR);
+	apic_timer_val = clocks / APIC_DIVISOR;
+
+	if (apic_timer_val)
+		set_dyn_tick_max_skip(apic_timer_val);
+
+	apic_write_around(APIC_TMICT, apic_timer_val);
 }
 
 static void __devinit setup_APIC_timer(unsigned int clocks)

@@ -1062,6 +1070,8 @@ void __init setup_boot_APIC_clock(void)
 	 */
 	setup_APIC_timer(calibration_result);
 
+	setup_dyn_tick_use_apic(calibration_result);
+
 	local_irq_enable();
 }

@@ -1200,6 +1210,13 @@ fastcall void smp_apic_timer_interrupt(s
 	 * interrupt lock, which is the WrongThing (tm) to do.
 	 */
 	irq_enter();
+
+	/*
+	 * Check if we need to wake up PIT interrupt handler.
+	 * Otherwise just wake up local APIC timer.
+	 */
+	wakeup_pit_or_apic(cpu, regs);
+
 	smp_local_timer_interrupt(regs);
 	irq_exit();
 }

arch/i386/kernel/dyn-tick.c +150 -0

150 lines not on this page · raw message

arch/i386/kernel/irq.c +3 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/irq.c
===================================================================
@@ -18,6 +18,7 @@
 #include <linux/notifier.h>
 #include <linux/cpu.h>
 #include <linux/delay.h>
+#include <linux/dyn-tick.h>
 
 DEFINE_PER_CPU(irq_cpustat_t, irq_stat) ____cacheline_maxaligned_in_smp;
 EXPORT_PER_CPU_SYMBOL(irq_stat);

@@ -76,6 +77,8 @@ fastcall unsigned int do_IRQ(struct pt_r
 	}
 #endif
 
+	dyn_tick_interrupt(irq, regs);
+
 #ifdef CONFIG_4KSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();

arch/i386/kernel/Makefile +1 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/Makefile
===================================================================
@@ -32,6 +32,7 @@ obj-$(CONFIG_MODULES)		+= module.o
 obj-y				+= sysenter.o vsyscall.o
 obj-$(CONFIG_ACPI_SRAT) 	+= srat.o
 obj-$(CONFIG_HPET_TIMER) 	+= time_hpet.o
+obj-$(CONFIG_NO_IDLE_HZ) 	+= dyn-tick.o
 obj-$(CONFIG_EFI) 		+= efi.o efi_stub.o
 obj-$(CONFIG_EARLY_PRINTK)	+= early_printk.o

arch/i386/kernel/process.c +3 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/process.c
===================================================================
@@ -39,6 +39,7 @@
 #include <linux/ptrace.h>
 #include <linux/random.h>
 #include <linux/kprobes.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>

@@ -200,6 +201,8 @@ void cpu_idle(void)
 			if (cpu_is_offline(cpu))
 				play_dead();
 
+			dyn_tick_reprogram_timer();
+
 			__get_cpu_var(irq_stat).idle_timestamp = jiffies;
 			idle();
 		}

arch/i386/kernel/time.c +5 -3

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/time.c
===================================================================
@@ -46,6 +46,7 @@
 #include <linux/bcd.h>
 #include <linux/efi.h>
 #include <linux/mca.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/io.h>
 #include <asm/smp.h>

@@ -252,8 +253,7 @@ EXPORT_SYMBOL(profile_pc);
  * timer_interrupt() needs to keep up the real-time clock,
  * as well as call the "do_timer()" routine every clocktick
  */
-static inline void do_timer_interrupt(int irq, void *dev_id,
-					struct pt_regs *regs)
+void do_timer_interrupt(int irq, void *dev_id, struct pt_regs *regs)
 {
 #ifdef CONFIG_X86_IO_APIC
 	if (timer_ack) {

@@ -423,7 +423,7 @@ static struct sysdev_class timer_sysclas
 
 
 /* XXX this driverfs stuff should probably go elsewhere later -john */
-static struct sys_device device_timer = {
+struct sys_device device_timer = {
 	.id	= 0,
 	.cls	= &timer_sysclass,
 };

@@ -479,5 +479,7 @@ void __init time_init(void)
 	cur_timer = select_timer();
 	printk(KERN_INFO "Using %s for high-res timesource\n",cur_timer->name);
 
+	dyn_tick_time_init(cur_timer);
+
 	time_init_hook();
 }

arch/i386/kernel/timers/timer_pit.c +38 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_pit.c
===================================================================
@@ -148,6 +148,44 @@ static unsigned long get_offset_pit(void
 	return count;
 }
 
+/*
+ * REVISIT: Looks like on P3 APIC timer keeps running if PIT mode
+ *	    is changed. On P4, changing PIT mode seems to kill
+ *	    APIC timer interrupts. Same thing with disabling PIT
+ *	    interrupt.
+ */
+void disable_pit_timer(void)
+{
+	extern spinlock_t i8253_lock;
+	unsigned long flags;
+
+	spin_lock_irqsave(&i8253_lock, flags);
+	outb_p(0x32, PIT_MODE);		/* binary, mode 1, LSB/MSB, ch 0 */
+	spin_unlock_irqrestore(&i8253_lock, flags);
+}
+
+/*
+ * Reprograms the next timer interrupt
+ * PIT timer reprogramming code taken from APM code.
+ * Note that PIT timer is a 16-bit timer, which allows max
+ * skip of only few seconds.
+ */
+void reprogram_pit_timer(int jiffies_to_skip)
+{
+	int skip;
+	extern spinlock_t i8253_lock;
+	unsigned long flags;
+
+	skip = jiffies_to_skip * LATCH;
+	if (skip > 0xffff)
+		skip = 0xffff;
+
+	spin_lock_irqsave(&i8253_lock, flags);
+	outb_p(0x34, PIT_MODE);		/* binary, mode 2, LSB/MSB, ch 0 */
+	outb_p(skip & 0xff, PIT_CH0);	/* LSB */
+	outb(skip >> 8, PIT_CH0);	/* MSB */
+	spin_unlock_irqrestore(&i8253_lock, flags);
+}
 
 /* tsc timer_opts struct */
 struct timer_opts timer_pit = {

arch/i386/kernel/timers/timer_pm.c +4 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_pm.c
===================================================================
@@ -15,6 +15,7 @@
 #include <linux/module.h>
 #include <linux/device.h>
 #include <linux/init.h>
+#include <linux/dyn-tick.h>
 #include <asm/types.h>
 #include <asm/timer.h>
 #include <asm/smp.h>

@@ -168,6 +169,9 @@ static void mark_offset_pmtmr(void)
 	monotonic_base += delta * NSEC_PER_USEC;
 	write_sequnlock(&monotonic_lock);
 
+	if (dyn_tick_enabled())
+		return;
+
 	/* convert to ticks */
 	delta += offset_delay;
 	lost = delta / (USEC_PER_SEC / HZ);

arch/i386/kernel/timers/timer_tsc.c +21 -8

Index: linux-2.6.13-rc5-ck2/arch/i386/kernel/timers/timer_tsc.c
===================================================================
@@ -14,6 +14,7 @@
 #include <linux/cpufreq.h>
 #include <linux/string.h>
 #include <linux/jiffies.h>
+#include <linux/dyn-tick.h>
 
 #include <asm/timer.h>
 #include <asm/io.h>

@@ -166,10 +167,19 @@ static void delay_tsc(unsigned long loop
 	} while ((now-bclock) < loops);
 }
 
+/* update the monotonic base value */
+static inline void update_monotonic_base(unsigned long long last_offset)
+{
+	unsigned long long this_offset;
+
+	this_offset = ((unsigned long long)last_tsc_high << 32) | last_tsc_low;
+	monotonic_base += cycles_2_ns(this_offset - last_offset);
+}
+
 #ifdef CONFIG_HPET_TIMER
 static void mark_offset_tsc_hpet(void)
 {
-	unsigned long long this_offset, last_offset;
+	unsigned long long last_offset;
  	unsigned long offset, temp, hpet_current;
 
 	write_seqlock(&monotonic_lock);

@@ -197,9 +207,7 @@ static void mark_offset_tsc_hpet(void)
 	}
 	hpet_last = hpet_current;
 
-	/* update the monotonic base value */
-	this_offset = ((unsigned long long)last_tsc_high<<32)|last_tsc_low;
-	monotonic_base += cycles_2_ns(this_offset - last_offset);
+	update_monotonic_base(last_offset);
 	write_sequnlock(&monotonic_lock);
 
 	/* calculate delay_at_last_interrupt */

@@ -346,7 +354,7 @@ static void mark_offset_tsc(void)
 	int count;
 	int countmp;
 	static int count1 = 0;
-	unsigned long long this_offset, last_offset;
+	unsigned long long last_offset;
 	static int lost_count = 0;
 
 	write_seqlock(&monotonic_lock);

@@ -367,6 +375,12 @@ static void mark_offset_tsc(void)
 
 	rdtsc(last_tsc_low, last_tsc_high);
 
+	if (dyn_tick_enabled()) {
+		update_monotonic_base(last_offset);
+		write_sequnlock(&monotonic_lock);
+		return;
+	}
+
 	spin_lock(&i8253_lock);
 	outb_p(0x00, PIT_MODE);     /* latch the count ASAP */

@@ -434,9 +448,8 @@ static void mark_offset_tsc(void)
 			cpufreq_delayed_get();
 	} else
 		lost_count = 0;
-	/* update the monotonic base value */
-	this_offset = ((unsigned long long)last_tsc_high<<32)|last_tsc_low;
-	monotonic_base += cycles_2_ns(this_offset - last_offset);
+
+	update_monotonic_base(last_offset);
 	write_sequnlock(&monotonic_lock);
 
 	/* calculate delay_at_last_interrupt */

arch/i386/mach-default/setup.c +16 -0

Index: linux-2.6.13-rc5-ck2/arch/i386/mach-default/setup.c
===================================================================
@@ -93,6 +93,22 @@ void __init time_init_hook(void)
 	setup_irq(0, &irq0);
 }
 
+/**
+ * replace_timer_interrupt - allow replacing timer interrupt handler
+ *
+ * Description:
+ *	Can be used to replace timer interrupt handler with a more optimized
+ *	handler. Used for enabling and disabling of CONFIG_NO_IDLE_HZ.
+ */
+void replace_timer_interrupt(void *new_handler)
+{
+	unsigned long flags;
+
+	write_seqlock_irqsave(&xtime_lock, flags);
+	irq0.handler = new_handler;
+	write_sequnlock_irqrestore(&xtime_lock, flags);
+}
+
 #ifdef CONFIG_MCA
 /**
  * mca_nmi_hook - hook into MCA specific NMI chain

include/asm-i386/dyn-tick.h +93 -0

93 lines not on this page · raw message

include/linux/dyn-tick.h +75 -0

75 lines not on this page · raw message

kernel/dyn-tick.c +273 -0

273 lines not on this page · raw message

kernel/Makefile +1 -0

Index: linux-2.6.13-rc5-ck2/kernel/Makefile
===================================================================
@@ -30,6 +30,7 @@ obj-$(CONFIG_SYSFS) += ksysfs.o
 obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_CRASH_DUMP) += crash_dump.o
 obj-$(CONFIG_SECCOMP) += seccomp.o
+obj-$(CONFIG_NO_IDLE_HZ) += dyn-tick.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

Srivatsa Vaddagiri

2005-08-07 16:57 UTC #43 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 2 lines · Con Kolivas

Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?

Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.

Another observation I have made regarding dynamic tick patch is that PIT is being reprogrammed whenever the CPUs are coming out of sleep state (because of an interrupt say). This can happen at any arbitary time, not necessarily on jiffy boundaries. As a result, there will be an offset between when jiffy interrupts will now occur vs when they would have originally occured had PIT never been stopped. Not sure if having this offset is good, but atleast one necessary change that I foresee if zeroing delay_at_last_interrupt when disabling dynamic tick. For that matter, it may be easier to disable the PIT timer by just masking PIT interrupts (instead of changing its mode).

Will keep you posted of my progress with dynamic tick patch.

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Con Kolivas

author

2005-08-07 23:53 UTC #44 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 11 lines · Srivatsa Vaddagiri

Quoted 2 lines · Con Kolivas
Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.
Will keep you posted of my progress with dynamic tick patch.

Great! I wasn't sure what time frame you meant when you last posted. I won't do anything more, leaving this patch as it is, and pass the baton to you.

Cheers,
Con

Kyle Moffett

2005-08-08 01:20 UTC #45 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 9 lines · Con Kolivas

Quoted 5 lines · Srivatsa Vaddagiri
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.
Great! I wasn't sure what time frame you meant when you last posted. I won't do anything more, leaving this patch as it is, and pass the baton to you.

I'm curious what has happened to the PPC side of the patch. IIRC, someone was working on such a port, but it seems to have been lost along the way at some point. Is there any additional information on that patch?

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things, because that would also stop them from doing clever things.
-- Doug Gwyn

Con Kolivas

author

2005-08-08 01:36 UTC #46 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 14 lines · Kyle Moffett

Quoted 8 lines · Con Kolivas
Quoted 6 lines · Srivatsa Vaddagiri
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which
I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.
Quoted 4 lines
Great! I wasn't sure what time frame you meant when you last posted. I won't
do anything more, leaving this patch as it is, and pass the baton to you.

Quoted 4 lines

I'm curious what has happened to the PPC side of the patch. IIRC, someone
was working on such a port, but it seems to have been lost along the way at some point. Is there any additional information on that patch?

Tony said he had it lying around somewhere and needed to find time to dust it off and get it up to speed.

Cheers,
Con

Gabriel Devenyi

2005-08-08 01:47 UTC #47 lore raw

Re: [ck] Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Along the same lines, is there an x86_64 patch around?

Quoted 30 lines · Con Kolivas

Quoted 14 lines · Kyle Moffett
Quoted 8 lines · Con Kolivas
Quoted 6 lines · Srivatsa Vaddagiri
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which
I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.
Quoted 4 lines
Great! I wasn't sure what time frame you meant when you last posted. I won't
do anything more, leaving this patch as it is, and pass the baton to you.
Quoted 4 lines
I'm curious what has happened to the PPC side of the patch. IIRC, someone
was working on such a port, but it seems to have been lost along the way at some point. Is there any additional information on that patch?
Tony said he had it lying around somewhere and needed to find time to dust it off and get it up to speed.
Cheers,
Con
_______________________________________________
[email protected]
ck mailing list. Please reply-to-all when posting.
If replying to an email please reply below the original message.
http://bhhdoa.org.au/mailman/listinfo/ck

Signature · 2 lines

Gabriel Devenyi
[email protected]

Srivatsa Vaddagiri

2005-08-08 02:43 UTC #48 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 2 lines · Con Kolivas

Tony said he had it lying around somewhere and needed to find time to dust it off and get it up to speed.

PPC64 is on my ToDo list as well. Will take it up after the dyn-tick is cleaned up for SMP.

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Nigel Cunningham

2005-08-08 07:05 UTC #49 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Hi.

Quoted 6 lines · Srivatsa Vaddagiri

Quoted 2 lines · Con Kolivas
Tony said he had it lying around somewhere and needed to find time to dust it off and get it up to speed.
PPC64 is on my ToDo list as well. Will take it up after the dyn-tick is cleaned up for SMP.

An SMP related area you might want to look at (if you haven't already) is hotplug cpu support. There seems to be some interaction between dynticks and hotplug that prevents a CPU from getting to set its state to CPU_DEAD (despite being in the idle function), while the thread seeking to down it sits in __cpu_die waiting. I observed the issue while testing Suspend2 - about 1 in 5 resumes would lock solid or have a big delay at resume time (never when suspending, interestingly). Turning off dynticks allowed me to do 15 successful resumes on the trot.

Regards.

Nigel

Signature · 4 lines

Evolution.
Enumerate the requirements.
Consider the interdependencies.
Calculate the probabilities.

Tony Lindgren

2005-08-08 07:27 UTC #59 lore raw

Quoted 8 lines · Srivatsa Vaddagiri

Quoted 6 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.

Quoted 12 lines

Con,
I have been looking at some of the requirement of tickless idle CPUs in
core kernel areas like scheduler and RCU. Basically, both power management and virtualization benefit if idle CPUs can cut off useless timer ticks. Especially from a virtualization standpoint, I think it makes sense that we enable this feature on a per-CPU basis i.e let individual CPUs cut off their ticks as and when they become idle. The benefit of this is more visible in platforms that host lot of (SMP) VMs on the same machine. Most of the time, these VMs may be partially idle (some CPUs in it are idle, some not) and it is good that we quiesce the timer ticks on the partial set of idle CPUs. Both S390 and Xen ports of Linux kernel have this ability today (S390 has it in mainline already and Xen has it out of tree).

Good point, and it would be nice to have it resolved for systems that support idling individual CPUs. The current setup was done because when I was tinkering with the amd76x_pm patch a while a back, I noticed that idling the cpu disconnects all cpus from the bus. (As far as I remember)

So this may need to be configured depending on the system.

Quoted 10 lines

From this viewpoint, I think the current implementation of dynamic tick falls short of this requirement. It cuts of the timer ticks only when all CPUs go idle.
Apart from this observation, I have some others about the current dynamic tick patch:
- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't this wrong, considering that the timer list is per-CPU? This will cause some timers to be serviced much later than usual.

Yes if it's done on per-CPU basis. In the current setup the first interrupt will kick the system off the dyn-tick state and the timers get checked again.

Quoted 5 lines

- The fact that dyn_tick_state is global and accessed from all CPUs is probably a scalability concern, especially if we allow the ticks to be cut off on per-CPU basis.
From idling devices point of view, we still need some global variable I

believe. How else would you be able to tell all devices that the whole system does not have any timers for next 2 seconds?

Quoted 3 lines

- Again, when we allow this on a per-CPU basis, subsystems like RCU need to know the partial set of idle CPUs. RCU already does that thr' nohz_cpu_mask (which will need to replace dyn_cpu_map).

Sounds like that could work for dyn-tick too.

Quoted 4 lines

- Looking at dyn_tick_timer_interrupt, would it be nice if we avoid calling do_timer_interrupt so many times and instead update jiffies to (skipped_ticks - 1) and then call do_timer_interrupt once? I think VST does it that way.

In the long run we would do the calculations in usecs and just emulate jiffies from the hw timer. But yes, optimizing updating the time would be great.

Quoted 3 lines

- dyn_tick->max_skip = 0xffffff / apic_timer_val;
From my reading of Intel docs, APIC_TMICT is 32-bit. So why does the
above calculation take only 24-bits into account? What am I missing here?

Hmm, could be a bug here, needs to be checked. Maybe 32-bit APIC timer is optional support, or maybe I accidentally pulled the optional 24-bit support from the ACPI PM timer.

But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.

Quoted 7 lines

I can take a shot at addressing these concerns in dynamic_tick patch, but it seems to me that VST has already addressed all these to a big extent. Had you considered VST before? The biggest bottleneck I see in VST going mainline is its dependency on HRT patch but IMO it should be possible to write a small patch to support VST w/o HRT.
George, what do you think?

HRT + VST depend on APIC only, and does not use next_timer_interrupt(). CONFIG_NO_IDLE_HZ is already integrated in the mainline kernel for s390 and ARM.

You may also want to check out the ARM implementation as it does not have the issues listed above, which are mostly x86 specific issues.

Regards,

Tony

Tony Lindgren

2005-08-08 07:38 UTC #50 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 9 lines · Srivatsa Vaddagiri

Quoted 2 lines · Con Kolivas
Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.

Hopefully you get something worked out that works for all. Please see my amd76x_pm related comments earlier too.

Quoted 9 lines

Another observation I have made regarding dynamic tick patch is that PIT is being reprogrammed whenever the CPUs are coming out of sleep state (because of an interrupt say). This can happen at any arbitary time, not necessarily on jiffy boundaries. As a result, there will be an offset between when jiffy interrupts will now occur vs when they would have originally occured had PIT never been stopped. Not sure if having this offset is good, but atleast one necessary change that I foresee if zeroing delay_at_last_interrupt when disabling dynamic tick. For that matter, it may be easier to disable the PIT timer by just masking PIT interrupts (instead of changing its mode).

It should not matter when the PIT gets reprogrammed, as the system time is not updated from PIT timer. Jiffies are calculated from a second timer, such as ACPI PM timer or TSC.

Tony

Srivatsa Vaddagiri

2005-08-08 14:53 UTC #60 lore raw

Quoted 5 lines · Tony Lindgren

Good point, and it would be nice to have it resolved for systems that support idling individual CPUs. The current setup was done because when I was tinkering with the amd76x_pm patch a while a back, I noticed that idling the cpu disconnects all cpus from the bus. (As far as I remember)
So this may need to be configured depending on the system.

What do you mean by "idling" a cpu? Is it reprogramming the timer source to skip ticks? Or putting the cpu actually in some low-power state? And what do you mean by "disconnect all CPUs from the bus"? Is it the system bus? And what happens if the CPUs are disconnected from the bus like this while it is executing?

Both requirements (idling all CPUs together vs individually) I think will make the patch more complex. Are such systems (which require having to idle all CPUs together) pretty common that we have to care about?!

Quoted 6 lines

Quoted 3 lines
- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't this wrong, considering that the timer list is per-CPU? This will cause some timers to be serviced much later than usual.
Yes if it's done on per-CPU basis. In the current setup the first interrupt will kick the system off the dyn-tick state and the timers get checked again.

But that may be too late on some CPUs. If dyn_tick->skip = 100, all CPUs skip 100 ticks. However some CPUs may have timers that need to be service much before that.

Quoted 7 lines

Quoted 5 lines
- The fact that dyn_tick_state is global and accessed from all CPUs is probably a scalability concern, especially if we allow the ticks to be cut off on per-CPU basis.
From idling devices point of view, we still need some global variable I
believe. How else would you be able to tell all devices that the whole system does not have any timers for next 2 seconds?

Why would anyone want to know that? Moreover, when we enable this per-CPU, the no-timer-interval is also distributed across CPUs.

Quoted 3 lines

But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.

On these systems, do you disabled APIC (dyntick=noapic)?

Quoted 1 line

HRT + VST depend on APIC only, and does not use next_timer_interrupt().

AFAIK, it supports pm timer too and yes it has its own implementation of next_timer_interrupt, which makes it more work for merging. At this point, looks like dyn-tick is gaining more momentum and is more likely a candidate for merging. I will send out my SMP-support changes to dynamic tick soon.

Quoted 2 lines

You may also want to check out the ARM implementation as it does not have the issues listed above, which are mostly x86 specific issues.

Thanks for the pointer. Will look at it.

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Srivatsa Vaddagiri

2005-08-08 15:06 UTC #51 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 3 lines · Tony Lindgren

It should not matter when the PIT gets reprogrammed, as the system time is not updated from PIT timer. Jiffies are calculated from a second timer, such as ACPI PM timer or TSC.

I am more worried about timer latencies and its effects. Because of this skew between when jiffy interrupt was originally supposed to happen and when it will now happen, it can lead to, for example, scheduling latencies upto 1 jiffy (for HZ=100, that could hurt some apps?) Not to mention that this will not play well with high-res-timers, but that is probably not a concern now!

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Folkert van Heusden

2005-08-08 15:08 UTC #57 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Are there any tools for checking the current number of ticks? I currently do:
#!/bin/sh

S1=cat /proc/interrupts | grep "^ 0: " | awk '{ print $2; }' sleep 1
S2=cat /proc/interrupts | grep "^ 0: " | awk '{ print $2; }' echo $((S2-S1))

But that gives me output like this on a quiet system with only firefox, a couple of gnometerms and a bittorrent downloader values like:

folkert@ehm:~$ for i in `seq 1 10` ; do ./ticks ; done
566
511
630
501
522
533
503
516
518
515
which I find quiet high, aren't they?

Quoted 8 lines · Con Kolivas

Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?
Cheers,
Con
---

Folkert van Heusden

Signature · 5 lines

Auto te koop, zie: http://www.vanheusden.com/daihatsu.php
--------------------------------------------------------------------
Get your PGP/GPG key signed at www.biglumber.com!
--------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE

Daniel Petrini

2005-08-08 15:16 UTC #58 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Hi Folkert,

Yes, there are some utilities. You can use pmstats-0.2 (look for the link in past messages) or timertop (sent some minutes ago in LKML). Regards,
Daniel

Quoted 50 lines · Folkert van Heusden

Are there any tools for checking the current number of ticks? I currently do:
#!/bin/sh
S1=cat /proc/interrupts | grep "^ 0: " | awk '{ print $2; }' sleep 1
S2=cat /proc/interrupts | grep "^ 0: " | awk '{ print $2; }' echo $((S2-S1))
But that gives me output like this on a quiet system with only firefox, a couple of gnometerms and a bittorrent downloader values like:
folkert@ehm:~$ for i in `seq 1 10` ; do ./ticks ; done
566
511
630
501
522
533
503
516
518
515
which I find quiet high, aren't they?
Quoted 8 lines · Con Kolivas
Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?
Cheers,
Con
---
Folkert van Heusden
--
Auto te koop, zie: http://www.vanheusden.com/daihatsu.php
--------------------------------------------------------------------
Get your PGP/GPG key signed at www.biglumber.com!
--------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines

10LE - Linux
INdT - Manaus - Brazil

Tony Lindgren

2005-08-08 15:20 UTC #61 lore raw

Quoted 7 lines · Srivatsa Vaddagiri

Quoted 5 lines · Tony Lindgren
Good point, and it would be nice to have it resolved for systems that support idling individual CPUs. The current setup was done because when I was tinkering with the amd76x_pm patch a while a back, I noticed that idling the cpu disconnects all cpus from the bus. (As far as I remember)
So this may need to be configured depending on the system.

Quoted 5 lines

What do you mean by "idling" a cpu? Is it reprogramming the timer source to skip ticks? Or putting the cpu actually in some low-power state? And what do you mean by "disconnect all CPUs from the bus"? Is it the system bus? And what happens if the CPUs are disconnected from the bus like this while it is executing?

As far as I remember enabling AMD stop grant disconnects all cpus. This means the system won't be able to do any work until the dyntick timer interrupt wakes up the system.

Quoted 3 lines

Both requirements (idling all CPUs together vs individually) I think will make the patch more complex. Are such systems (which require having to idle all CPUs together) pretty common that we have to care about?!

Probably all AMD SMP based systems? Somebody with better knowledge should verify this.

Quoted 10 lines

Quoted 6 lines
Quoted 3 lines
- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't this wrong, considering that the timer list is per-CPU? This will cause some timers to be serviced much later than usual.
Yes if it's done on per-CPU basis. In the current setup the first interrupt will kick the system off the dyn-tick state and the timers get checked again.
But that may be too late on some CPUs. If dyn_tick->skip = 100, all CPUs skip 100 ticks. However some CPUs may have timers that need to be service much before that.

Not in the current case, as the system is completely idle until some interrupt wakes up the system. Of course it would be different if you make it per-CPU.

Quoted 10 lines

Quoted 7 lines
Quoted 5 lines
- The fact that dyn_tick_state is global and accessed from all CPUs is probably a scalability concern, especially if we allow the ticks to be cut off on per-CPU basis.
From idling devices point of view, we still need some global variable I
believe. How else would you be able to tell all devices that the whole system does not have any timers for next 2 seconds?
Why would anyone want to know that? Moreover, when we enable this per-CPU, the no-timer-interval is also distributed across CPUs.

Well we need to be able to do various things in the idle loop depending on the length of the estimated sleep. For example, if next_timer_interrupt is 2 jiffies away, we cannot do much. But if next_timer_interrupt is 2 seconds away, we can idle pretty much all devices.

Quoted 5 lines

Quoted 3 lines
But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.
On these systems, do you disabled APIC (dyntick=noapic)?

Yeah. It only seems to work on P3 systems.

Quoted 7 lines

Quoted 1 line
HRT + VST depend on APIC only, and does not use next_timer_interrupt().
AFAIK, it supports pm timer too and yes it has its own implementation of next_timer_interrupt, which makes it more work for merging. At this point, looks like dyn-tick is gaining more momentum and is more likely a candidate for merging. I will send out my SMP-support changes to dynamic tick soon.

Cool :) Sounds like SMP and HPET are about the only things that still need work. Or can somebody think of anything else?

Quoted 4 lines

Quoted 2 lines
You may also want to check out the ARM implementation as it does not have the issues listed above, which are mostly x86 specific issues.
Thanks for the pointer. Will look at it.

Tony

Zwane Mwaikambo

2005-08-09 14:18 UTC #62 lore raw

Quoted 10 lines · Tony Lindgren

As far as I remember enabling AMD stop grant disconnects all cpus. This means the system won't be able to do any work until the dyntick timer interrupt wakes up the system.
Quoted 3 lines
Both requirements (idling all CPUs together vs individually) I think will make the patch more complex. Are such systems (which require having to idle all CPUs together) pretty common that we have to care about?!
Probably all AMD SMP based systems? Somebody with better knowledge should verify this.

It would be the K7 only.

Quoted 7 lines

Quoted 3 lines
But that may be too late on some CPUs. If dyn_tick->skip = 100, all CPUs skip 100 ticks. However some CPUs may have timers that need to be service much before that.
Not in the current case, as the system is completely idle until some interrupt wakes up the system. Of course it would be different if you make it per-CPU.

I once did a weekend version of this with SMP support and for the PIT, i had the last cpu to enter idle turn reprogram the PIT. Unfortunately this means waiting for all processors and isn't as effective as a result.

Quoted 5 lines

Well we need to be able to do various things in the idle loop depending on the length of the estimated sleep. For example, if next_timer_interrupt is 2 jiffies away, we cannot do much. But if next_timer_interrupt is 2 seconds away, we can idle pretty much all devices.

Quoted 7 lines

Quoted 5 lines
Quoted 3 lines
But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.
On these systems, do you disabled APIC (dyntick=noapic)?
Yeah. It only seems to work on P3 systems.

Odd, does reprogramming the APIC at that point get it going again?

Zwane

George Anzinger

2005-08-09 19:39 UTC #52 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 11 lines · Srivatsa Vaddagiri

Quoted 1 line · Con Kolivas
Respin of the dynamic ticks patch for i386 by Tony Lindgen and Tuukka Tikkanen with further code cleanups. Are were there yet?
Con,
I am afraid until SMP correctness is resolved, then this is not
in a position to go in (unless you want to enable it only for UP, which
I think should not be our target). I am working on making this work correctly on SMP systems. Hopefully I will post a patch soon.
Another observation I have made regarding dynamic tick patch is that PIT is being reprogrammed whenever the CPUs are coming out of sleep state (because of an interrupt say). This can happen at any arbitary time, not necessarily on jiffy boundaries. As a result, there will be an offset between when jiffy interrupts will now occur vs when they would have originally occured had PIT never been stopped. Not sure if having this offset is good, but atleast one necessary change that I foresee if zeroing delay_at_last_interrupt when disabling dynamic tick. For that matter, it may be easier to disable the PIT timer by just masking PIT interrupts (instead of changing its mode).

IMNOHO, this is the ONLY way to keep proper time. As soon as you reprogram the PIT you have lost track of the time.

My VST patch just turns masks the interrupt.

Signature · 2 lines

George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

George Anzinger

2005-08-09 20:07 UTC #64 lore raw

In reply to Zwane Mwaikambo

Quoted 80 lines · Tony Lindgren

Quoted 10 lines · Srivatsa Vaddagiri
Quoted 3 lines · Con Kolivas
This is the dynamic ticks patch for i386 as written by Tony Lindgen <[email protected]> and Tuukka Tikkanen <[email protected]>. Patch for 2.6.13-rc5
There were a couple of things that I wanted to change so here is an updated version. This code should have stabilised enough for general testing now.
Con,
I have been looking at some of the requirement of tickless idle CPUs in
core kernel areas like scheduler and RCU. Basically, both power management and virtualization benefit if idle CPUs can cut off useless timer ticks. Especially from a virtualization standpoint, I think it makes sense that we enable this feature on a per-CPU basis i.e let individual CPUs cut off their ticks as and when they become idle. The benefit of this is more visible in platforms that host lot of (SMP) VMs on the same machine. Most of the time, these VMs may be partially idle (some CPUs in it are idle, some not) and it is good that we quiesce the timer ticks on the partial set of idle CPUs. Both S390 and Xen ports of Linux kernel have this ability today (S390 has it in mainline already and Xen has it out of tree).
Good point, and it would be nice to have it resolved for systems that support idling individual CPUs. The current setup was done because when I was tinkering with the amd76x_pm patch a while a back, I noticed that idling the cpu disconnects all cpus from the bus. (As far as I remember)
So this may need to be configured depending on the system.
Quoted 9 lines
From this viewpoint, I think the current implementation of dynamic tick falls short of this requirement. It cuts of the timer ticks only when all CPUs go idle.
Apart from this observation, I have some others about the current dynamic tick patch:
- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't this wrong, considering that the timer list is per-CPU? This will cause some timers to be serviced much later than usual.
Yes if it's done on per-CPU basis. In the current setup the first interrupt will kick the system off the dyn-tick state and the timers get checked again.
Quoted 6 lines
- The fact that dyn_tick_state is global and accessed from all CPUs is probably a scalability concern, especially if we allow the ticks to be cut off on per-CPU basis.
From idling devices point of view, we still need some global variable I
believe. How else would you be able to tell all devices that the whole system does not have any timers for next 2 seconds?
Quoted 3 lines
- Again, when we allow this on a per-CPU basis, subsystems like RCU need to know the partial set of idle CPUs. RCU already does that thr' nohz_cpu_mask (which will need to replace dyn_cpu_map).
Sounds like that could work for dyn-tick too.
Quoted 4 lines
- Looking at dyn_tick_timer_interrupt, would it be nice if we avoid calling do_timer_interrupt so many times and instead update jiffies to (skipped_ticks - 1) and then call do_timer_interrupt once? I think VST does it that way.
In the long run we would do the calculations in usecs and just emulate jiffies from the hw timer. But yes, optimizing updating the time would be great.
Quoted 3 lines
- dyn_tick->max_skip = 0xffffff / apic_timer_val;
From my reading of Intel docs, APIC_TMICT is 32-bit. So why does the
above calculation take only 24-bits into account? What am I missing here?
Hmm, could be a bug here, needs to be checked. Maybe 32-bit APIC timer is optional support, or maybe I accidentally pulled the optional 24-bit support from the ACPI PM timer.
But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.

Quoted 7 lines

Quoted 4 lines
I can take a shot at addressing these concerns in dynamic_tick patch, but it seems to me that VST has already addressed all these to a big extent. Had you considered VST before? The biggest bottleneck I see in VST going mainline is its dependency on HRT patch but IMO it should be possible to write a small patch to support VST w/o HRT.
George, what do you think?
HRT + VST depend on APIC only, and does not use next_timer_interrupt().

I convinced my self that the next_timer... code in timer.c misses timers (i.e. gives the wrong answer). I did this (after wondering due to performance) by scanning the whole timer list after I had the next_timer... answer and finding a better answer, not always, but some times. That code does not address the cascade list correctly.

Signature · 2 lines

George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

Daniel Petrini

2005-08-09 20:22 UTC #65 lore raw

Quoted 5 lines

I convinced my self that the next_timer... code in timer.c misses timers (i.e. gives the wrong answer). I did this (after wondering due to performance) by scanning the whole timer list after I had the next_timer... answer and finding a better answer, not always, but some times. That code does not address the cascade list correctly.

The timertop kernel patch accounts for all the scheduled timers. Try to take a look at its output at /proc/top_info. Maybe it can help to detect it.

Daniel Petrini

Signature · 2 lines

10LE - Linux
INdT - Manaus - Brazil

Tony Lindgren

2005-08-10 07:46 UTC #63 lore raw

Quoted 6 lines · Zwane Mwaikambo

Quoted 4 lines · Tony Lindgren
As far as I remember enabling AMD stop grant disconnects all cpus. This means the system won't be able to do any work until the dyntick timer interrupt wakes up the system.

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
Both requirements (idling all CPUs together vs individually) I think will make the patch more complex. Are such systems (which require having to idle all CPUs together) pretty common that we have to care about?!
Probably all AMD SMP based systems? Somebody with better knowledge should verify this.
It would be the K7 only.

OK, still quite a few systems.

Quoted 21 lines

Quoted 7 lines
Quoted 3 lines
But that may be too late on some CPUs. If dyn_tick->skip = 100, all CPUs skip 100 ticks. However some CPUs may have timers that need to be service much before that.
Not in the current case, as the system is completely idle until some interrupt wakes up the system. Of course it would be different if you make it per-CPU.
I once did a weekend version of this with SMP support and for the PIT, i had the last cpu to enter idle turn reprogram the PIT. Unfortunately this means waiting for all processors and isn't as effective as a result.
Quoted 9 lines
Well we need to be able to do various things in the idle loop depending on the length of the estimated sleep. For example, if next_timer_interrupt is 2 jiffies away, we cannot do much. But if next_timer_interrupt is 2 seconds away, we can idle pretty much all devices.
Quoted 4 lines
Quoted 3 lines
But in any case on P4 systems the APIC timer is not the bottleneck as stopping or reprogramming PIT also kills APIC. (This does not happen on P3 systems). So the bottleneck most likely is the length of PIT.

Quoted 5 lines

Quoted 3 lines
Quoted 1 line
On these systems, do you disabled APIC (dyntick=noapic)?
Yeah. It only seems to work on P3 systems.
Odd, does reprogramming the APIC at that point get it going again?

Hmmm, might be worth trying.

Tony

Tony Lindgren

2005-08-10 08:03 UTC #66 lore raw

In reply to George Anzinger

Quoted 11 lines · George Anzinger

Quoted 11 lines
Quoted 9 lines
I can take a shot at addressing these concerns in dynamic_tick patch, but it seems to me that VST has already addressed all these to a big extent. Had you considered VST before? The biggest bottleneck I see in VST going mainline is its dependency on HRT patch but IMO it should be possible to write a small patch
to support VST w/o HRT.
George, what do you think?

Quoted 7 lines

Quoted 1 line
HRT + VST depend on APIC only, and does not use next_timer_interrupt().
I convinced my self that the next_timer... code in timer.c misses timers (i.e. gives the wrong answer). I did this (after wondering due to performance) by scanning the whole timer list after I had the next_timer... answer and finding a better answer, not always, but some times. That code does not address the cascade list correctly.

Do you have a patch around for improving next_timer_interrupt()?

Tony

Srivatsa Vaddagiri

2005-08-10 14:04 UTC #53 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

In reply to George Anzinger

Quoted 2 lines · George Anzinger

IMNOHO, this is the ONLY way to keep proper time. As soon as you reprogram the PIT you have lost track of the time.

George,
Can't TSC (or equivalent) serve as a backup while PIT is disabled,
especially considering that we disable PIT only for short duration in practice (few seconds maybe) _and_ that we don't have HRT support yet?

Signature · 5 lines

Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Bill Davidsen

2005-08-10 20:00 UTC #15 lore raw

In reply to Jim MacBaine

Quoted 9 lines · Jim MacBaine

I just borrowed a power meter to see (or not to see) real effects of dyntick. The difference between static 1000 HZ and dynamic HZ is much less than I expected, only a very little about noise. With dyntick disabled at 1000 HZ my laptop needs 31,3 W. With dyntick enabled I get 29.8 W, the pmstats-0.2 script shows me that the system is at 35-45 HZ when it is idle.
The power consumption difference between 250 HZ static and dyntick is below the noise, so maybe hardly worth all the struggle.

I think it's the other way round, we have the lower power without the higher latency. At least as I can measure...

Bravo to all concerned to get this to the testing stage!

Signature · 3 lines

   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me

George Anzinger

2005-08-10 22:40 UTC #54 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 9 lines · Srivatsa Vaddagiri

Quoted 1 line · George Anzinger
IMNOHO, this is the ONLY way to keep proper time. As soon as you reprogram the PIT you have lost track of the time.
George,
Can't TSC (or equivalent) serve as a backup while PIT is disabled,
especially considering that we disable PIT only for short duration in practice (few seconds maybe) _and_ that we don't have HRT support yet?

I think it really depends on what you want. If you really want to keep good time, the only rock in town is the one connected to the PIT (and the pmtimer). The problem is, if you want the jiffie edge to be stable, there is just now way to reprogram the PIT to get it back to where it was.

In an old version of HRT I did a trick of loading a short count (based on reading the TSC or pmtimer) and then put the LATCH count on top of it. In a correctly performing PIT, it should count down the short count, interrupt, load the long count and continue from there. Aside from the machines that had BAD PITs (they reset on the load instead of the expiry of the current count) there were other problems that, in the end, cause loss of time (too fast, too slow, take your pick). I also found PITs that signaled that they had loaded the count (they set a status bit) prior to actually loading it. All in all, I find the PIT is just an ugly beast to try to program. On the other hand, if you want regular interrupts at some fixed period, it will do this forever (give or take a epoch or two;) with out touching anything after the initial program set up.

In the end, I concluded that, for the community kernel, it is really best to just interrupt the irq line and leave the PIT run. Then you use the TSC or pmtimer to figure the gross loss of interrupts and leave the PIT interrupt again to define the jiffie edge. If you have other, more pressing, concerns I suppose you can program the PIT, but don't expect your wall clock to be as stable as it is now.

Signature · 2 lines

George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

George Anzinger

2005-08-10 22:43 UTC #67 lore raw

In reply to George Anzinger

Tony Lindgren wrote:
~

Quoted 2 lines

Do you have a patch around for improving next_timer_interrupt()?

Well, sort of. The code in the VST patch does the right thing. Problem is it does a bit more than the timer.c code. You can find that code on the HRT site CVS.

Signature · 2 lines

George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

Bill Davidsen

2005-08-11 21:31 UTC #55 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 12 lines · George Anzinger

Quoted 10 lines · Srivatsa Vaddagiri
Quoted 1 line · George Anzinger
IMNOHO, this is the ONLY way to keep proper time. As soon as you reprogram the PIT you have lost track of the time.
George,
Can't TSC (or equivalent) serve as a backup while PIT is disabled,
especially considering that we disable PIT only for short duration in practice (few seconds maybe) _and_ that we don't have HRT support yet?

Quoted 6 lines

I think it really depends on what you want. If you really want to keep good time, the only rock in town is the one connected to the PIT (and the pmtimer). The problem is, if you want the jiffie edge to be stable, there is just now way to reprogram the PIT to get it back to where it was.
In an old version of HRT I did a trick of loading a short count (based on reading the TSC or pmtimer) and then put the LATCH count on top of it. In a correctly performing PIT, it should count down the short count, interrupt, load the long count and continue from there. Aside from the machines that had BAD PITs (they reset on the load instead of the expiry of the current count) there were other problems that, in the end, cause loss of time (too fast, too slow, take your pick). I also found PITs that signaled that they had loaded the count (they set a status bit) prior to actually loading it. All in all, I find the PIT is just an ugly beast to try to program. On the other hand, if you want regular interrupts at some fixed period, it will do this forever (give or take a epoch or two;) with out touching anything after the initial program set up.
In the end, I concluded that, for the community kernel, it is really best to just interrupt the irq line and leave the PIT run. Then you use the TSC or pmtimer to figure the gross loss of interrupts and leave the PIT interrupt again to define the jiffie edge. If you have other, more pressing, concerns I suppose you can program the PIT, but don't expect your wall clock to be as stable as it is now.

What are the portability and scaling issues if it were done this way? It clearly looks practical on x86 uni, but if we want per-CPU non-tick, I'm less sure how it would work.

But when you go to non-x86 hardware, is there always going to be another source of wakeup available if the PIT is blocked instead of reset? I have to go back and look at how SPARC hardware works, I don't remember enough to be useful.

Signature · 3 lines

   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me

George Anzinger

2005-08-12 15:16 UTC #56 lore raw

Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 5

Quoted 15 lines · Bill Davidsen

Quoted 13 lines · George Anzinger
Quoted 11 lines · Srivatsa Vaddagiri
Quoted 1 line · George Anzinger
IMNOHO, this is the ONLY way to keep proper time. As soon as you reprogram the PIT you have lost track of the time.
George,
Can't TSC (or equivalent) serve as a backup while PIT is disabled,
especially considering that we disable PIT only for short duration in practice (few seconds maybe) _and_ that we don't have HRT support yet?

Quoted 7 lines

Quoted 6 lines
I think it really depends on what you want. If you really want to keep good time, the only rock in town is the one connected to the PIT (and the pmtimer). The problem is, if you want the jiffie edge to be stable, there is just now way to reprogram the PIT to get it back to where it was.
In an old version of HRT I did a trick of loading a short count (based on reading the TSC or pmtimer) and then put the LATCH count on top of it. In a correctly performing PIT, it should count down the short count, interrupt, load the long count and continue from there. Aside from the machines that had BAD PITs (they reset on the load instead of the expiry of the current count) there were other problems that, in the end, cause loss of time (too fast, too slow, take your pick). I also found PITs that signaled that they had loaded the count (they set a status bit) prior to actually loading it. All in all, I find the PIT is just an ugly beast to try to program. On the other hand, if you want regular interrupts at some fixed period, it will do this forever (give or take a epoch or two;) with out touching anything after the initial program set up.
In the end, I concluded that, for the community kernel, it is really best to just interrupt the irq line and leave the PIT run. Then you use the TSC or pmtimer to figure the gross loss of interrupts and leave the PIT interrupt again to define the jiffie edge. If you have other, more pressing, concerns I suppose you can program the PIT, but don't expect your wall clock to be as stable as it is now.
What are the portability and scaling issues if it were done this way? It clearly looks practical on x86 uni, but if we want per-CPU non-tick, I'm less sure how it would work.

I am not sure how much is involved. For VST I disabled the tick generated NMI watchdog interrupt on a per cpu basis but stopped the PIT tick only when all cpus were idle. The next step would be to mess with the interrupt steering logic to keep the tick away from idle cpus. I did not get into this level in my work, being mainly interested in embedded systems.

Quoted 2 lines

But when you go to non-x86 hardware, is there always going to be another source of wakeup available if the PIT is blocked instead of reset? I have to go back and look at how SPARC hardware works, I don't remember enough to be useful.

Most (all) other archs don't have PITs. The x86 sucks big time when it comes to time keeping hardware. The most common hardware is a counter that runs forever (much as the TSC but FIXED in frequency). Interrupts are generated either by comparing a register to this or using companion counters that just count down to zero. In either case you don't loose time because you can always precisely set up an interrupt. To sleep, then, you just set your sleep time in the normal time base interrupt counter. At the end, you know exactly what to set to get back to the regular tick.

These other platforms make VST and High Res Timers so easy...

Signature · 2 lines

George Anzinger   [email protected]
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/

Pavel Machek

2005-08-14 20:15 UTC #16 lore raw