Periodic clock tick considered harmful (was: Re: HZ, preferably as small as possible)
Quoted 5 lines · Mark Mielke
Quoted 2 lines · Cort Dougan
Yes, please do make it a config option. 10x interrupt overhead makes me worry. It lets users tailor the kernel to their expected load.
Quoted 8 lines
All this talk is getting to me.
I thought we recently (1 month ago? 2 months ago?) concluded that increases in interrupt frequency only affects performance by a very small amount, but generates an increase in responsiveness. The only real argument against that I have seen, is the 'power conservation' argument. The idea was, that the scheduler itself did not execute on most interrupts. The clock is updated, and that is about all.
On UML and mainframe Linux, *any* periodic clock tick is heavy overhead when you have a large number of (mostly idle) instances of Linux running, isn't it? I think I once heard those architectures went to great lengths to avoid periodic clock ticks. (My memory is rusty, though.)
How about this: let's apply the high-resolution timer patch, which adds explicit timer events inbetween the normal 100 Hz events when needed to satisfy precise sleep requests. Then let's increase the interval between the normal periodic clock events from 10ms to infinity. Everything will keep working, as the high-resolution timer patch code will schedule timer events as needed -- but suddenly we'll have power consumption as low as possible, snappier performance, and the thousands-of-instances case will no longer have this huge drain on performance from periodic timer events that do nothing but update jiffiers.
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
- Dan
Hi,
Quoted 7 lines · On Thu, 11 Jul 2002 [email protected] wrote:
Then let's increase the interval between the normal periodic clock events from 10ms to infinity. Everything will keep working, as the high-resolution timer patch code will schedule timer events as needed -- but suddenly we'll have power consumption as low as possible, snappier performance, and the thousands-of-instances case will no longer have this huge drain on performance from periodic timer events that do nothing but update jiffiers.
Well, that's the aim.
Quoted 3 lines
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
The only thing that was mentioned yet was the amount of stuff that depends on periodic ticks. If we just tick unperiodically, we'd fail for sure, but if we make these instances depend on another timer - we won.
I think a good scheduler can handle this and should also be able to determine a halfaway optimal tick rate for the current load.
That's a _real_ challenge, guys!
Regards, Thunder
Signature · 7 lines
(Use http://www.ebb.org/ungeek if you can't decode) ------BEGIN GEEK CODE BLOCK------ Version: 3.12 GCS/E/G/S/AT d- s++:-- a? C++$ ULAVHI++++$ P++$ L++++(+++++)$ E W-$ N--- o? K? w-- O- M V$ PS+ PE- Y- PGP+ t+ 5+ X+ R- !tv b++ DI? !D G e++++ h* r--- y- ------END GEEK CODE BLOCK------
Quoted 11 lines · Thunder from the hill
Quoted 3 lines · On Thu, 11 Jul 2002 [email protected] wrote:
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
The only thing that was mentioned yet was the amount of stuff that depends on periodic ticks. If we just tick unperiodically, we'd fail for sure, but if we make these instances depend on another timer - we won.
I think a good scheduler can handle this and should also be able to determine a halfaway optimal tick rate for the current load.
The current approach on s390 is stop the timer tick only for idle cpus, because that's where it hurts. A busy system can just keep on using 100 (or 1000) Hz timers.
The jiffies value then gets updated from the time stamp counter when an interrupt happens on an idle CPU.
See Martin Schwidefsky's recent post for code:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102578746520177&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=102578746420174&w=2
Arnd <><
Quoted 24 lines · Arnd Bergmann
Quoted 11 lines · Thunder from the hill
Quoted 3 lines · On Thu, 11 Jul 2002 [email protected] wrote:
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
The only thing that was mentioned yet was the amount of stuff that depends on periodic ticks. If we just tick unperiodically, we'd fail for sure, but if we make these instances depend on another timer - we won.
I think a good scheduler can handle this and should also be able to determine a halfaway optimal tick rate for the current load.
The current approach on s390 is stop the timer tick only for idle cpus, because that's where it hurts. A busy system can just keep on using 100 (or 1000) Hz timers.
The jiffies value then gets updated from the time stamp counter when an interrupt happens on an idle CPU.See Martin Schwidefsky's recent post for code:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102578746520177&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=102578746420174&w=2
Quoted 6 lines
Arnd <><-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
There has also been some work done by Michael Olsen ([email protected]) regarding noperiodic jiffie updates. This was in the context of embedded handheld devices, in particular IBM's LinuxWatch. Similar problem arise in such devices where the timer tick can be a significant source of battery drainage.
There are/were some settled differences between Michael and Martin approach, which I won't be able to adequately describe.
I believe Michael has a publication on this, I let him respond.
Signature · 1 line
-- Hubertus Franke ([email protected])
Quoted 22 lines · [email protected] wrote:
Quoted 5 lines · Mark Mielke
Quoted 2 lines · Cort Dougan
Yes, please do make it a config option. 10x interrupt overhead makes me worry. It lets users tailor the kernel to their expected load.
Quoted 8 lines
All this talk is getting to me.
I thought we recently (1 month ago? 2 months ago?) concluded that increases in interrupt frequency only affects performance by a very small amount, but generates an increase in responsiveness. The only real argument against that I have seen, is the 'power conservation' argument. The idea was, that the scheduler itself did not execute on most interrupts. The clock is updated, and that is about all.
On UML and mainframe Linux, *any* periodic clock tick is heavy overhead when you have a large number of (mostly idle) instances of Linux running, isn't it? I think I once heard those architectures went to great lengths to avoid periodic clock ticks. (My memory is rusty, though.)
Quoted 10 lines
How about this: let's apply the high-resolution timer patch, which adds explicit timer events inbetween the normal 100 Hz events when needed to satisfy precise sleep requests. Then let's increase the interval between the normal periodic clock events from 10ms to infinity. Everything will keep working, as the high-resolution timer patch code will schedule timer events as needed -- but suddenly we'll have power consumption as low as possible, snappier performance, and the thousands-of-instances case will no longer have this huge drain on performance from periodic timer events that do nothing but update jiffiers.
Ah, but you haven't looked at all that happens on a 1/HZ tick. The high-res-timers patch does NOT eliminate the 1/HZ tick. That tick is used to do a LOT of accounting activity which IMHO is best done by a periodic tick. In particular, the time slice and execution time management depend on the periodic tick. As a test we put together a tickless system, much as suggested above, and put enough stuff in it to see what the overhead was and how it changed. The conclusion was that the timer over head increased far beyond the current overhead as soon as the system load (actually the number of context switches per second) increased beyond what a moderately busy system experiences. In other words, the system was overload prone. The current accounting activity is flat WRT to context switching which is IMHO just what it should be. For those who want to know, a patch to put that test system together is still on the HRT sourceforge site.
-g
Quoted 6 lines
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
- Dan
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
george,
with the HRT is there any reason to have 10 timer interrupts per process
quantum anymore? (10 ms ticks, 100 ms proc quantum)
Quoted 24 lines · george anzinger
Ah, but you haven't looked at all that happens on a 1/HZ tick. The high-res-timers patch does NOT eliminate the 1/HZ tick. That tick is used to do a LOT of accounting activity which IMHO is best done by a periodic tick. In particular, the time slice and execution time management depend on the periodic tick. As a test we put together a tickless system, much as suggested above, and put enough stuff in it to see what the overhead was and how it changed. The conclusion was that the timer over head increased far beyond the current overhead as soon as the system load (actually the number of context switches per second) increased beyond what a moderately busy system experiences. In other words, the system was overload prone. The current accounting activity is flat WRT to context switching which is IMHO just what it should be. For those who want to know, a patch to put that test system together is still on the HRT sourceforge site.
-g
Quoted 5 lines
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
- Dan
Signature · 8 lines
/************************************************** ** Mark Salisbury || [email protected] ** ** If you would like to sponsor me for the ** ** Mass Getaway, a 150 mile bicycle ride to for ** ** MS, contact me to donate by cash or check or ** ** click the link below to donate by credit card ** **************************************************/ https://www.nationalmssociety.org/pledge/pledge.asp?participantid=86736
Quoted 5 lines · mbs
george,
with the HRT is there any reason to have 10 timer interrupts per process
quantum anymore? (10 ms ticks, 100 ms proc quantum)
First blush is HELL YES! The issue is accounting. When you ask how long a program ran, you are looking at the accounting that happens on a tick. This is where one of two counters gets bumped (one for system, the other for user, depending on what was interrupted). This information could, of course, be gathered every system call/ exit and every context switch, BUT, there are FAR more system calls and context switches than 1/HZ ticks. Thus collecting accounting info this way adds overhead as the system load increases, a VERY BAD thing IMHO.
A second point is that tasks only run for a quantum IF they never block or get preempted. Most tasks will have some fraction of a quantum remaining when they are scheduled.
-g
Quoted 36 lines · george anzinger
Quoted 17 lines
Ah, but you haven't looked at all that happens on a 1/HZ tick. The high-res-timers patch does NOT eliminate the 1/HZ tick. That tick is used to do a LOT of accounting activity which IMHO is best done by a periodic tick. In particular, the time slice and execution time management depend on the periodic tick. As a test we put together a tickless system, much as suggested above, and put enough stuff in it to see what the overhead was and how it changed. The conclusion was that the timer over head increased far beyond the current overhead as soon as the system load (actually the number of context switches per second) increased beyond what a moderately busy system experiences. In other words, the system was overload prone. The current accounting activity is flat WRT to context switching which is IMHO just what it should be. For those who want to know, a patch to put that test system together is still on the HRT sourceforge site.
Quoted 7 lines
-g
Quoted 5 lines
OK, so I'm just an ignorant member of the peanut gallery, but I'd like to hear a real kernel hacker explain why this isn't the way to go.
- Dan
-- /**************************************************
** Mark Salisbury || [email protected] ** ** If you would like to sponsor me for the ** ** Mass Getaway, a 150 mile bicycle ride to for ** ** MS, contact me to donate by cash or check or **** click the link below to donate by credit card ** **************************************************/
https://www.nationalmssociety.org/pledge/pledge.asp?participantid=86736
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Re: Periodic clock tick considered harmful (was: Re: HZ, preferably as
Quoted 3 lines
First blush is HELL YES! The issue is accounting. When you ask how long a program ran, you are looking at the accounting that happens on a tick. This is where one of two
Thats also an implementation issue. Note that the current code is also wildly inaccurate. Mr Shannon says we are good to at best 50 run/sleep changes a second. I've got "100% busy" workloads that are 99% asleep.
Tracking cpu usage at task switch works a lot better for newer processors which as well as having rdtsc also have performance counters. In fact you can do much more interesting things on modern PC class platforms like scheduling using pre-emption interrupts based on instructions executed, memory accesses and more.
Alan
Re: Periodic clock tick considered harmful (was: Re: HZ, preferably as
Quoted 9 lines · Alan Cox
Quoted 4 lines
First blush is HELL YES! The issue is accounting. When you ask how long a program ran, you are looking at the accounting that happens on a tick. This is where one of two
Thats also an implementation issue. Note that the current code is also wildly inaccurate. Mr Shannon says we are good to at best 50 run/sleep changes a second. I've got "100% busy" workloads that are 99% asleep.
Quoted 6 lines
Tracking cpu usage at task switch works a lot better for newer processors which as well as having rdtsc also have performance counters. In fact you can do much more interesting things on modern PC class platforms like scheduling using pre-emption interrupts based on instructions executed, memory accesses and more.
Oh, I agree. Hardware could make all this a lot easier.
Signature · 6 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Real time sched: http://sourceforge.net/projects/rtsched/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Quoted 10 lines · george anzinger
First blush is HELL YES! The issue is accounting. When you ask how long a program ran, you are looking at the accounting that happens on a tick. This is where one of two counters gets bumped (one for system, the other for user, depending on what was interrupted). This information could, of course, be gathered every system call/ exit and every context switch, BUT, there are FAR more system calls and context switches than 1/HZ ticks. Thus collecting accounting info this way adds overhead as the system load increases, a VERY BAD thing IMHO.
In addition to syscall entries/exits and sched changes, you then also need to keep track of interrupt and trap entries/exits in order to have the complete picture. Even then, determining exactly when you're going to return to user-space can be tricky. Your statement is indeed accurate, exact accounting's cost increases linearly with the number of events that occur.
Having exact accounting all the time is certainly not necessary, but it is indeed sometimes useful. *shameless self-promotion*, That's yet another reason why I think LTT's inclusion in the mailing kernel would be helpful. In addition to the rest of the capabilities it provides, it provides exact accounting for whoever really needs it on the spot.
If you are looking for better accounting using clock ticks, then have a look at McCanne and Torek's paper "A Randomized Sampling Clock for CPU Utilization Estimation and Code Profiling" presented at Usenix in '93. As the title implies, they vary the clock to obtain random samples and therefore obtain very accurate results about system accounting. The use of hardware counters to obtain samples is also viable, although it's much more arch dependent than random clock ticks.
Cheers,
Karim
===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================
Quoted 4 lines · In article <[email protected]>, <[email protected]> wrote:
How about this: let's apply the high-resolution timer patch, which adds explicit timer events inbetween the normal 100 Hz events when needed to satisfy precise sleep requests.
The thing is, I think that's the wrong way around.
What we should have is a notion of a reasonably high frequency timer, and then _slow_it_down_ to something else if not needed.
Speeding the timer up is bad, because:
- you do need to limit the speedup to _something_ anyway (and it might as well be HZ)
- you get "partial jiffies", which means that only stuff that knows about the finer granularity gets it.
In contrast, if you slow things down in integer increments of "n", the only thing you need to do is to add in "n" instead of "1" in the timer tick handler. Nobody else needs to really care - there is no such thing as a "fractional jiffy".
Linus
In article <[email protected]>,
Quoted 5 lines · Linus Torvalds
In contrast, if you slow things down in integer increments of "n", the only thing you need to do is to add in "n" instead of "1" in the timer tick handler. Nobody else needs to really care - there is no such thing as a "fractional jiffy".
An added issue: if you slow the tick down, you can easily do it in _one_
place: the idle loop. When you exit the idle loop you speed it up again, and nobody is ever any wiser (sure, you need to know enough about timers in idle to know when you can do it, but that's still fairly localized).
In constrast, if you speed the timer up, you have to make this quite fundamental in timer handling, and have architecture-specific issues on how to speed the timer up and down etc. Big ugh, compared to just havin git in one place inside code that is already architecture-specific for other reasons.
Linus