Skip to contentSkip to conversation

x86: Reduce the default HZ value

PATCH In review Alok Kataria 9 participants 61 messages May 4, 2009 – May 8, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: Reduce the default HZ value

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

Thanks,
Alok

Signature · 55 lines
With HRT support in the kernel we shouldn't actually be needing a high interrupt
frequency. This patch reduces the HZ value to 100 for x86 defconfig.

A high HZ value may affect the performance of the system if its nonidle.
I ran a simple experiment with 2.6.29 kernel running on VMware..
A simple tight loop took about 264s to complete with a HZ value of 1000.
The system serviced a total of 264405 timer interrupts during that time.
The same loop with HZ=100 took only about 255sec to complete.
Total timer interrupts were 25593.
More information here - http://lkml.org/lkml/2009/4/28/401

With highres timers most of the important timers are not tied down with the
how often the jiffy value is updated so this shouldn't have any adverse
effects on the latency of these timers either.

Signed-off-by: Alok N Kataria <[email protected]>

Index: linux-tip-master/arch/x86/configs/i386_defconfig
===================================================================
--- linux-tip-master.orig/arch/x86/configs/i386_defconfig	2009-05-01 16:47:43.000000000 -0700
+++ linux-tip-master/arch/x86/configs/i386_defconfig	2009-05-01 16:49:48.000000000 -0700
@@ -312,11 +312,11 @@
 CONFIG_X86_PAT=y
 CONFIG_EFI=y
 CONFIG_SECCOMP=y
-# CONFIG_HZ_100 is not set
+CONFIG_HZ_100=y
 # CONFIG_HZ_250 is not set
 # CONFIG_HZ_300 is not set
-CONFIG_HZ_1000=y
-CONFIG_HZ=1000
+# CONFIG_HZ_1000 is not set
+CONFIG_HZ=100
 CONFIG_SCHED_HRTICK=y
 CONFIG_KEXEC=y
 CONFIG_CRASH_DUMP=y
Index: linux-tip-master/arch/x86/configs/x86_64_defconfig
===================================================================
--- linux-tip-master.orig/arch/x86/configs/x86_64_defconfig	2009-05-01 16:37:53.000000000 -0700
+++ linux-tip-master/arch/x86/configs/x86_64_defconfig	2009-05-01 16:50:22.000000000 -0700
@@ -316,11 +316,11 @@
 CONFIG_X86_PAT=y
 CONFIG_EFI=y
 CONFIG_SECCOMP=y
-# CONFIG_HZ_100 is not set
+CONFIG_HZ_100=y
 # CONFIG_HZ_250 is not set
 # CONFIG_HZ_300 is not set
-CONFIG_HZ_1000=y
-CONFIG_HZ=1000
+# CONFIG_HZ_1000 is not set
+CONFIG_HZ=100
 CONFIG_SCHED_HRTICK=y
 CONFIG_KEXEC=y
 CONFIG_CRASH_DUMP=y
#1 lore raw
Quoted 7 lines · Alok Kataria

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

What is the benefit of this?

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

-hpa

#2 lore raw
Quoted 1 line

What is the benefit of this?

I believe the "benefit" is that a certain posters proprietary virtualisation product works better.

Quoted 3 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

And HZ=100 actually causes real problems for some video work (not in Europe where its just peachy). We switched to 1000Hz a very long time ago because it improved desktop feel and responsiveness. We switched to tickless to keep that behaviour with good power and idle behaviour.

If we still have problems that give virtualisers hiccups then they will be giving real processor power management the same grief so I don't think meddling with HZ to harm the desktop is the right move - but if there are improvements to tickless (eg timer granularity and bunching under virtualisation) that would be far better to explore.

Alan

#8 lore raw
In reply to H. Peter Anvin
Quoted 10 lines · H. Peter Anvin
Quoted 7 lines · Alok Kataria

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

What is the benefit of this?

I did some experiments on linux 2.6.29 guests running on VMware and noticed that the number of timer interrupts could have some slowdown on the total throughput on the system.
A simple tight loop experiment showed that with HZ=1000 we took about 264sec to complete the loop and that same loop took about 255sec with HZ=100.
You can find more information here http://lkml.org/lkml/2009/4/28/401

And with HRT i don't see any downsides in terms of increased latencies for device timer's or anything of that sought.

Quoted 4 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

If that at all is such a big problem shouldn't we think about moving to using schedule_hrtimeout for such cases rather than relying on jiffy based timeouts.
The hrtimer explanation over here http://www.tglx.de/hrtimers.html
also talks about where these HZ (timer wheel) based timeouts be used and shouldn't really be dependent on accurate timing.

Also the default HZ value was 250 before this commit

commit 5cb04df8d3f03e37a19f2502591a84156be71772
x86: defconfig updates

And it was 250 for a very long time before that too. The commit log doesn't explain why the value was bumped up either.

Thanks,
Alok

#3 lore raw
In reply to Alan Cox
Quoted 4 lines · Alan Cox
Quoted 1 line

What is the benefit of this?

I believe the "benefit" is that a certain posters proprietary virtualisation product works better.

Hi Alan,

I posted numbers that I had, and I don't think that the problem is limited just to virtualization or our platform.

Quoted 8 lines
Quoted 3 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

And HZ=100 actually causes real problems for some video work (not in Europe where its just peachy). We switched to 1000Hz a very long time ago because it improved desktop feel and responsiveness. We switched to tickless to keep that behaviour with good power and idle behaviour.

IMO, one of the main motives of HRT implementation apart from getting higher precision timers was that we now don't necessarily need to rely on a high timer frequency. If you see problems with Desktop feel and responsiveness don't you think there would be other problem which might be causing that ? Your argument about the "desktop feel and responsiveness" doesn't explain what actual problem did you see. Also there are lots of distribution kernels which ship with a lower HZ value anyway, so I don't see why is HZ=1000 such a big requirement for your desktop use case.

Alok

#4 lore raw
Quoted 2 lines

IMO, one of the main motives of HRT implementation apart from getting higher precision timers was that we now don't necessarily need to rely

Timer frequency and HZ are two entirely different things nowdyas

Quoted 4 lines

on a high timer frequency. If you see problems with Desktop feel and responsiveness don't you think there would be other problem which might be causing that ? Your argument about the "desktop feel and responsiveness" doesn't explain what actual problem did you see.

People spent months poking at the differences before HZ=1000 became the default. It wasn't due for amusement values - but this is irrelevant anyway on a modern kernel as HZ=1000 is simply a precision setting that affects things like poll()

HZ on a tickless system has no meaningful relationship to wakup rates - which are what I assume you actually care about.

So do you want to change the precision of poll() and other functionality ? or do you want to change the wakeup rates and corresponding virtualisation overhead ?

If the latter then HZ is not the thing to touch.

What are you *actually* trying to achieve ?
What measurements have you done that make you think HZ is relevant in a tickless kernel ?

Alan

#5 lore raw
Quoted 4 lines · Alan Cox
Quoted 2 lines

IMO, one of the main motives of HRT implementation apart from getting higher precision timers was that we now don't necessarily need to rely

Timer frequency and HZ are two entirely different things nowdyas

Huh ? maybe I am reading this code incorrectly, but this is what I understand, the APIC is still being programmed to wake HZ time every second if the system is nonidle (periodic mode).
Only if the system is idle does the kernel program the APIC in one shot mode as a result the tickless kernel gives us a lot less pain when the guest is idle.

Here are the numbers with a HZ=100 kernel which proves this hypothesis.

[root@alok-vm-rhel64 ~]# cat /proc/interrupts | grep "timer" ; time sleep 30 ; cat /proc/interrupts | grep "timer"
0: 36 0 IO-APIC-edge timer
LOC: 7549 7176 Local timer interrupts

real    0m30.006s
user    0m0.000s
sys     0m0.000s

0: 36 0 IO-APIC-edge timer
LOC: 7616 7209 Local timer interrupts

So in this case when the system is (pretty much) "idle" the total number of wakeup's are far less just about 65 in the total 30sec on cpu0.

If I run a simple program which does a tight loop, this to check the behavior when the system is non-idle,

[root@alok-vm-rhel64 ~]# cat /proc/interrupts | grep "timer" ; time ./tightloop_short ; cat /proc/interrupts | grep "timer"
0: 36 0 IO-APIC-edge timer
LOC: 8008 7453 Local timer interrupts

real    0m30.377s
user    0m30.370s
sys     0m0.000s

0: 36 0 IO-APIC-edge timer
LOC: 11049 10493 Local timer interrupts

Here we see that we had a total of ~3000 interrupts. In this case the system was non-idle and hence the APIC was programmed in periodic mode.

The tightloop program only does this
int main()

{
        unsigned long long count;
        while(count++ < 5999999999UL);
        return 0;
}

If I do the same experiments on a HZ=1000 kernel I see that the number of interrupts would rise to 30000 in the second case.

I did check that the "apic_timer_irqs" counter - that is read from the proc file - is updated only from smp_apic_timer_interrupt code path, so this can't be a interrupt accounting bug.

In short, I don't believe that HZ and timer frequency are not related nowadays, please correct me if I am missing anything here.

Quoted 6 lines
Quoted 5 lines

on a high timer frequency. If you see problems with Desktop feel and responsiveness don't you think there would be other problem which might be causing that ? Your argument about the "desktop feel and responsiveness" doesn't explain what actual problem did you see.

Quoted 7 lines

People spent months poking at the differences before HZ=1000 became the default. It wasn't due for amusement values - but this is irrelevant anyway on a modern kernel as HZ=1000 is simply a precision setting that affects things like poll()

HZ on a tickless system has no meaningful relationship to wakup rates - which are what I assume you actually care about.

Yes I care about the wakeup rates and as explained above HZ does affect that.

Quoted 11 lines

So do you want to change the precision of poll() and other functionality ? or do you want to change the wakeup rates and corresponding virtualisation overhead ?

If the latter then HZ is not the thing to touch.

What are you *actually* trying to achieve ?
What measurements have you done that make you think HZ is relevant in a tickless kernel ?

I hope all these questions are answered above.

Thanks,
Alok

Quoted 2 lines

Alan

#6 lore raw
In reply to Alan Cox
Quoted 2 lines · Alan Cox

HZ on a tickless system has no meaningful relationship to wakup rates - which are what I assume you actually care about.

Linux is not tickless. It only switches off ticks if a processor is idle.

Quoted 3 lines

So do you want to change the precision of poll() and other functionality ? or do you want to change the wakeup rates and corresponding virtualisation overhead ?

select and poll use timeouts based on high resolution timers.

Quoted 2 lines

What measurements have you done that make you think HZ is relevant in a tickless kernel ?

Just reading the code gets you there.

#9 lore raw
In reply to Alok Kataria

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

#7 lore raw
In reply to Christoph Lameter
Quoted 6 lines · On Thu, 7 May 2009 10:09:56 -0400 (EDT) Christoph Lameter <[email protected]> wrote:
Quoted 2 lines · Alan Cox

HZ on a tickless system has no meaningful relationship to wakup rates - which are what I assume you actually care about.

Linux is not tickless. It only switches off ticks if a processor is idle.

Hooray - finally someone admits the *real* problem here, and for power management too. Otherwise known as "referencing jiffies as a variable must die"

Alan

#10 lore raw
In reply to Christoph Lameter
Quoted 6 lines · On Thu, 7 May 2009 10:13:52 -0400 (EDT) Christoph Lameter <[email protected]> wrote:

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

50 works for various european video apps, 60 breaks, 60 works for various US video apps, 50 breaks. Now that may have changed with all the select stuff being hrtimer based (which I'd missed).

The tick also still appears to be involved in ntp and in cpu stats where a 50Hz tick would mean only 25Hz accuracy on CPU usage etc

#11 lore raw
Quoted 2 lines · Alan Cox

The tick also still appears to be involved in ntp and in cpu stats where a 50Hz tick would mean only 25Hz accuracy on CPU usage etc

That could be fixed. If a thread is running then its using the processor and that does not change as long as it runs. That alone is enough to calculate cpu usage.

Only if we want to examine the cpu state periodically while its running (profiling) then we would have a justification for interrupting the thread.

#12 lore raw
In reply to Alan Cox
Quoted 7 lines · Alan Cox
Quoted 4 lines · On Thu, 7 May 2009 10:13:52 -0400 (EDT) Christoph Lameter <[email protected]> wrote:

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Quoted 6 lines
Quoted 2 lines

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

50 works for various european video apps, 60 breaks, 60 works for various US video apps, 50 breaks. Now that may have changed with all the select stuff being hrtimer based (which I'd missed).

Hence 300 Hz. ;)

Quoted 3 lines

Hooray - finally someone admits the *real* problem here, and for power management too. Otherwise known as "referencing jiffies as a variable must die"

Amen. Also, "using HZ as a unit of measurement must die, too."

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#13 lore raw
Quoted 1 line · H. Peter Anvin

Amen. Also, "using HZ as a unit of measurement must die, too."

Dont think that is a problem if we had a convention of seeing 1 HZ as an interval of 1 ms or so.

#54 lore raw
In reply to Alok Kataria
Quoted 13 lines · Alok Kataria
Quoted 10 lines · H. Peter Anvin
Quoted 7 lines · Alok Kataria

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

What is the benefit of this?

Quoted 7 lines

I did some experiments on linux 2.6.29 guests running on VMware and noticed that the number of timer interrupts could have some slowdown on the total throughput on the system.
A simple tight loop experiment showed that with HZ=1000 we took about 264sec to complete the loop and that same loop took about 255sec with HZ=100.
You can find more information here http://lkml.org/lkml/2009/4/28/401

This is why certain niches, such as HPC users, often prefer HZ=100 kernels. For the rest of us, sacrificing a few percent CPU throughput for significant latency gains is well worth it.

Quoted 8 lines

And with HRT i don't see any downsides in terms of increased latencies for device timer's or anything of that sought.

Quoted 4 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

Quoted 6 lines

If that at all is such a big problem shouldn't we think about moving to using schedule_hrtimeout for such cases rather than relying on jiffy based timeouts.
The hrtimer explanation over here http://www.tglx.de/hrtimers.html
also talks about where these HZ (timer wheel) based timeouts be used and shouldn't really be dependent on accurate timing.

But your patch doesn't do this. If you want us to merge a patch that makes VMware systems faster, we're a lot more likely to take it if it make everyone else's systems faster, or at least not slower.

Quoted 7 lines

Also the default HZ value was 250 before this commit

commit 5cb04df8d3f03e37a19f2502591a84156be71772
 x86: defconfig updates

And it was 250 for a very long time before that too. The commit log doesn't explain why the value was bumped up either.

250 was considered a compromise between 100 and 1000, but almost everyone who cared just ended up using one or the other, and most of them preferred 1000.

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway. If you can figure out a way to do that without harming latency, they'll be thrilled.

-- Chris

#20 lore raw
In reply to Alan Cox
Quoted 7 lines · Alan Cox
Quoted 4 lines · On Thu, 7 May 2009 10:13:52 -0400 (EDT) Christoph Lameter <[email protected]> wrote:

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Quoted 6 lines
Quoted 2 lines

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

50 works for various european video apps, 60 breaks, 60 works for various US video apps, 50 breaks. Now that may have changed with all the select stuff being hrtimer based (which I'd missed).

I would have assumed whatever timeout mechanism the video apps use should have already been converted to hrtimers ? Or are you saying that they use select stuff which is already hrtimer based and so there shouldn't be any problem now for video apps ?

Quoted 4 lines

The tick also still appears to be involved in ntp and in cpu stats where a 50Hz tick would mean only 25Hz accuracy on CPU usage etc

#55 lore raw
In reply to Chris Snook
Quoted 34 lines · Chris Snook
Quoted 13 lines · Alok Kataria
Quoted 10 lines · H. Peter Anvin
Quoted 7 lines · Alok Kataria

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

What is the benefit of this?

Quoted 7 lines

I did some experiments on linux 2.6.29 guests running on VMware and noticed that the number of timer interrupts could have some slowdown on the total throughput on the system.
A simple tight loop experiment showed that with HZ=1000 we took about 264sec to complete the loop and that same loop took about 255sec with HZ=100.
You can find more information here http://lkml.org/lkml/2009/4/28/401

This is why certain niches, such as HPC users, often prefer HZ=100 kernels. For the rest of us, sacrificing a few percent CPU throughput for significant latency gains is well worth it.

Quoted 8 lines

And with HRT i don't see any downsides in terms of increased latencies for device timer's or anything of that sought.

Quoted 4 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

Quoted 8 lines
Quoted 6 lines

If that at all is such a big problem shouldn't we think about moving to using schedule_hrtimeout for such cases rather than relying on jiffy based timeouts.
The hrtimer explanation over here http://www.tglx.de/hrtimers.html
also talks about where these HZ (timer wheel) based timeouts be used and shouldn't really be dependent on accurate timing.

But your patch doesn't do this.

The reason it doesn't do it is because poll and select already use hrtimer. So IMO no important subsystem relies on jiffies for wakeups. Thus the latency problem is not actually present in the kernel.

Quoted 3 lines

If you want us to merge a patch that
makes VMware systems faster, we're a lot more likely to take it if it make everyone else's systems faster, or at least not slower.

I doubt it would make any system slower, running these simple experiments is not hard at all and one could run these on native system too to check.

Quoted 9 lines
Quoted 8 lines

Also the default HZ value was 250 before this commit

commit 5cb04df8d3f03e37a19f2502591a84156be71772
x86: defconfig updates

And it was 250 for a very long time before that too. The commit log doesn't explain why the value was bumped up either.

Quoted 7 lines

250 was considered a compromise between 100 and 1000, but almost everyone who cared just ended up using one or the other, and most of them preferred 1000.

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway.

Yeah but I don't think there is any better platform other than LKML to figure out if at all this is a problem anymore. Once we are assured that a low HZ is no more a problem I don't see why would the various distros not consider reducing it.

Quoted 3 lines

If you can
figure out a way to do that without harming latency, they'll be thrilled.

Why do you think it would harm latency ?
The sched_tick too is driven by hrtimers, if there is any specific subsystem which you think still relies on jiffy we could think about using hrtimer's for them too, right ?
I did a quick scan and the only things that rely on jiffy are the device timeout's where latency is not a issue.
So please let me know in what cases do you think it could affect system latency.

Thanks,
Alok

Quoted 2 lines

-- Chris

#14 lore raw
In reply to H. Peter Anvin
Quoted 6 lines · H. Peter Anvin
Quoted 3 lines · Alan Cox

Hooray - finally someone admits the *real* problem here, and for power management too. Otherwise known as "referencing jiffies as a variable must die"

Amen. Also, "using HZ as a unit of measurement must die, too."

Love to -- now, what will it be replaced with?

grep for 'deadline' in drivers/ata/libata* to find an example not so easily converted away from jiffies.

Jeff

#21 lore raw
In reply to Christoph Lameter
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Another random user that relies on the jiffy tick is CLOCK_THREAD_CPUTIME_ID posix timers, although I'm planning to convert that to hrtimers some time in the future.

We also use the scheduler tick to generate a somewhat coupled time source from flaky TSCs -- reducing HZ decreases the accuracy. This is something only fixable in hardware by providing a proper (and cheap) high resolution clock source -- nehalem class machines have such a thing, provided you stick to one (maybe two) sockets [s390, ppc64 and sparc64 also rule].

#15 lore raw
In reply to Jeff Garzik
Quoted 6 lines · On Thu, 07 May 2009 12:55:05 -0400 Jeff Garzik <[email protected]> wrote:
Quoted 5 lines · H. Peter Anvin
Quoted 3 lines · Alan Cox

Hooray - finally someone admits the *real* problem here, and for power management too. Otherwise known as "referencing jiffies as a variable must die"

Quoted 6 lines
Quoted 1 line

Amen. Also, "using HZ as a unit of measurement must die, too."

Love to -- now, what will it be replaced with?

grep for 'deadline' in drivers/ata/libata* to find an example not so easily converted away from jiffies.

I don't see any.

I do see a complicated interface that appears to actually really want to implement

		add_timer(&foo->expiry_timer);

and checks against the timer completing. In fact it looks as if all the stuff in there is really down to

add a timer
check if it expired
check how long until it expires
delete it

And you might as well measure that in HZ=1000's better known as "milliseconds"

#22 lore raw
In reply to Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

Quoted 6 lines

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

#23 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

Quoted 6 lines
Quoted 2 lines

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

#43 lore raw
In reply to Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

But the schedule is activated when a process is being woken up. The main use is for load balancing.

#24 lore raw
In reply to Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Quoted 7 lines
Quoted 3 lines

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One approach would be to enter nohz mode when running a CPU-bound application on a CPU that had nothing else (other than the idle task) on its runqueue and for which rcu_needs_cpu() returns zero. In this mode, RCU would need to be informed on each system call, perhaps with an rcu_kernel_enter() and rcu_kernel_exit() that work like rcu_irq_enter() and rcu_irq_exit() -- and that perhaps replace rcu_irq_enter() and rcu_irq_exit().

Then RCU would ignore any CPU that was executing a CPU-bound application, allowing the HZ to be dialed down as low as you like, or perhaps really entering something like nohz mode.

Thanx, Paul

#44 lore raw
In reply to Christoph Lameter
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

But the schedule is activated when a process is being woken up. The main use is for load balancing.

Somehow people like involuntary preemption to happen at slightly more than glacial pace as well.

#25 lore raw
In reply to Paul E. McKenney
Quoted 38 lines · Paul E. McKenney
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Quoted 7 lines
Quoted 3 lines

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One approach would be to enter nohz mode when running a CPU-bound application on a CPU that had nothing else (other than the idle task) on its runqueue and for which rcu_needs_cpu() returns zero. In this mode, RCU would need to be informed on each system call, perhaps with an rcu_kernel_enter() and rcu_kernel_exit() that work like rcu_irq_enter() and rcu_irq_exit() -- and that perhaps replace rcu_irq_enter() and rcu_irq_exit().

Quoted 3 lines

Then RCU would ignore any CPU that was executing a CPU-bound application, allowing the HZ to be dialed down as low as you like, or perhaps really entering something like nohz mode.

Which would make syscall more expensive, not something you'd want to do :-)

#38 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One could also schedule RCU via hrtimers with a large fuzz period?

No, that's not the point, the longer these period are, the more callbacks you can accumulate in a period. You need a cap on the callback list, we already have seen DoS scenarios in this space.

#49 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Peter Zijlstra

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

If you aggregate events into a single hrtimer event then it may work. Wasnt that recently added?

No it won't, you want fairly decent involuntary preemption rate to keep the full service latency at a usable figure.

The problem with scheduling a hrtimer along with tasks is that at high context switch rates the timer will never fire but you do pay the overhead of programming the hardware each time, something that can be about as expensive as the whole context switch itself.

Although, I guess we could amortize that by not re-programming the timer when the existing timer is within a reasonable period (say 1ms) of the requested on.

#39 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

Quoted 5 lines
Quoted 1 line

One could also schedule RCU via hrtimers with a large fuzz period?

No, that's not the point, the longer these period are, the more callbacks you can accumulate in a period. You need a cap on the callback list, we already have seen DoS scenarios in this space.

At some point the RCU must run and if the callback list gets too long the callbacks must be processed. Thats unavoidable.

#40 lore raw
In reply to Christoph Lameter
Quoted 7 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One could also schedule RCU via hrtimers with a large fuzz period?

You could, but then you would still have a periodic interrupt introducing jitter into your HPC workload. The approach I suggested allows RCU to be happy with no periodic interrupts on any CPU that has only one runnable task that is a CPU-bound user-level task (in addition to the idle task, of course).

Thanx, Paul

#46 lore raw
In reply to Christoph Lameter
Quoted 9 lines · Christoph Lameter
Quoted 2 lines · Peter Zijlstra

Somehow people like involuntary preemption to happen at slightly more than glacial pace as well.

These intervals could be dynamically established by the scheduler.

If there is a single process running on a processor and no other process is contending then there is no need to do involuntary preemption.

Sure, and CONFIG_SCHED_HRTICK does that, the trick is getting it to not hurt others.

#16 lore raw
In reply to Alan Cox
Quoted 20 lines · Alan Cox
Quoted 9 lines · On Thu, 07 May 2009 12:55:05 -0400 Jeff Garzik <[email protected]> wrote:
Quoted 5 lines · H. Peter Anvin
Quoted 3 lines · Alan Cox

Hooray - finally someone admits the *real* problem here, and for power management too. Otherwise known as "referencing jiffies as a variable must die"

Amen. Also, "using HZ as a unit of measurement must die, too."

Love to -- now, what will it be replaced with?

grep for 'deadline' in drivers/ata/libata* to find an example not so easily converted away from jiffies.

I don't see any.

I do see a complicated interface that appears to actually really want to implement

		add_timer(&foo->expiry_timer);
Quoted 7 lines

and checks against the timer completing. In fact it looks as if all the stuff in there is really down to

add a timer
check if it expired
check how long until it expires
delete it

This is why I mentioned this example... because it's not as easy as you seem to think it is :)

We care only about a decreasing time interval. This interval is passed to register polling functions (bitbang no longer than <this> amount of time), as well as _cumulatively_ affecting the entire EH [sub-]process.

A timer-based solution, in addition to being an ugly hack, would imply replacing a simple variable with _at least_ two spinlocks, plus a timer callback function that simply says "I expired". With loops such as

	max_msecs = calc_deadline(overall_deadline, ...)
	while (!(register & bit))
		msleep(1)
		max_msecs--
		register = readl(...)

must be converted to the more-complex timer-based solution.

libata would be happy to use milliseconds rather than jiffies; the unit does not matter. What matters is calculating our progress versus the clock tick, as spread across multiple functions, multiple contexts, and register polling loops.

The current code is a -lot- more simple than checking "is timer expired?" all over the code, given that any sort of timer-based function implies dealing with additional concurrency issues -- a complication the libata EH does not need.

Jeff

#26 lore raw
In reply to Peter Zijlstra
Quoted 39 lines · Peter Zijlstra
Quoted 38 lines · Paul E. McKenney
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Christoph Lameter

I think we need to reduce the general tick frequency to be as low as possible. With high resolution timers the tick frequency is just the frequency with which the timer interrupt disturbs a running application.

Are there any benefits remaining from frequent timer interrupts? I would think that 60 HZ would be sufficient.

It would be good if the kernel would be truly tickless. Scheduler events would be driven by the scheduling intervals and not the invokations of the scheduler softirq.

The only thing that's driven by the softirq is load-balancing, there's way more to the scheduler-tick than kicking that thing awake every so often.

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

Quoted 7 lines
Quoted 3 lines

Therefore, decreasing the HZ value to say 50, we'd get a minimum involuntary preemption granularity of 20ms, something on the high end of barely usable.

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One approach would be to enter nohz mode when running a CPU-bound application on a CPU that had nothing else (other than the idle task) on its runqueue and for which rcu_needs_cpu() returns zero. In this mode, RCU would need to be informed on each system call, perhaps with an rcu_kernel_enter() and rcu_kernel_exit() that work like rcu_irq_enter() and rcu_irq_exit() -- and that perhaps replace rcu_irq_enter() and rcu_irq_exit().

Quoted 6 lines
Quoted 3 lines

Then RCU would ignore any CPU that was executing a CPU-bound application, allowing the HZ to be dialed down as low as you like, or perhaps really entering something like nohz mode.

Which would make syscall more expensive, not something you'd want to do :-)

In general, I agree. However, in the case where you have a single CPU-bound task running in user mode, you don't care that much about syscall performance. So, yes, this would mean having yet another config variable that users running big CPU-bound scientific applications would need to worry about, which is not perfect either.

For whatever it is worth, the added overhead on entry would be something like the following:

void rcu_irq_enter(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (rdtp->dynticks_nesting++)
		return;
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(!(rdtp->dynticks & 0x1), &rcu_rs);
	smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */
}

On exit, a bit more:

void rcu_irq_exit(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (--rdtp->dynticks_nesting)
		return;
	smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(rdtp->dynticks & 0x1, &rcu_rs);

	/* If the interrupt queued a callback, get out of dyntick mode. */
	if (__get_cpu_var(rcu_data).nxtlist ||
	    __get_cpu_var(rcu_bh_data).nxtlist)
		set_need_resched();
}

But I could move the callback check into call_rcu(), which would get the overhead of rcu_irq_exit() down to about that of rcu_irq_enter().

Thanx, Paul

#48 lore raw
In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

The problem is that running the scheduler of off hrtimers is too expensive. We have the code, we tried it, people complained.

If you aggregate events into a single hrtimer event then it may work. Wasnt that recently added?

#37 lore raw
In reply to Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

One could also schedule RCU via hrtimers with a large fuzz period?

#27 lore raw
In reply to Paul E. McKenney

To get back to the main point: I agree with Alok that the default HZ needs to be as low as possible. The remaining justification is the load balancing and possible context switching between multiple tasks contenting for a processor.

Is it enough if this occurs 100 times per second?

If so then we should change the default in kernel/Kconfig.hz.

#45 lore raw
In reply to Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

Somehow people like involuntary preemption to happen at slightly more than glacial pace as well.

These intervals could be dynamically established by the scheduler.

If there is a single process running on a processor and no other process is contending then there is no need to do involuntary preemption.

#47 lore raw
In reply to Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines

If there is a single process running on a processor and no other process is contending then there is no need to do involuntary preemption.

Sure, and CONFIG_SCHED_HRTICK does that, the trick is getting it to not hurt others.

If other processors are scheduled on the processor then an IPI could notify the scheduler over there to start timer interrupts for preemption again.

I still see timer interrupts ongoing even with CONFIG_SCHED_HRTICK.

#50 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra

No it won't, you want fairly decent involuntary preemption rate to keep the full service latency at a usable figure.

The problem with scheduling a hrtimer along with tasks is that at high context switch rates the timer will never fire but you do pay the overhead of programming the hardware each time, something that can be about as expensive as the whole context switch itself.

What are high context switch rates? 1000 HZ? Generally it seems that context switches are bad for cpu caches and thus to be avoided.

#41 lore raw
In reply to Paul E. McKenney
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

Quoted 7 lines
Quoted 1 line

One could also schedule RCU via hrtimers with a large fuzz period?

You could, but then you would still have a periodic interrupt introducing jitter into your HPC workload. The approach I suggested allows RCU to be happy with no periodic interrupts on any CPU that has only one runnable task that is a CPU-bound user-level task (in addition to the idle task, of course).

Sounds good.

An HPC workload typically has minimal kernel interaction. RCU would only need to run once and then the system would be quiet.

#28 lore raw
In reply to Christoph Lameter
Quoted 6 lines · Christoph Lameter

To get back to the main point: I agree with Alok that the default HZ needs to be as low as possible. The remaining justification is the load balancing and possible context switching between multiple tasks contenting for a processor.

Is it enough if this occurs 100 times per second?

I am sorry, I got into solution mode too quickly in my earlier posts.

As long as you have enough memory for the callbacks, and as long as you don't mind things like netfilter changes taking longer, RCU doesn't much care what value HZ has.

Quoted 1 line

If so then we should change the default in kernel/Kconfig.hz.

But some workloads and users do care deeply about memory size and netfilter-change speed, of course. So perhaps HPC workloads need a different default HZ value.

Thanx, Paul

#51 lore raw
In reply to Christoph Lameter
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

No it won't, you want fairly decent involuntary preemption rate to keep the full service latency at a usable figure.

Quoted 7 lines
Quoted 4 lines

The problem with scheduling a hrtimer along with tasks is that at high context switch rates the timer will never fire but you do pay the overhead of programming the hardware each time, something that can be about as expensive as the whole context switch itself.

What are high context switch rates? 1000 HZ? Generally it seems that context switches are bad for cpu caches and thus to be avoided.

We're talking about in excess of 250k switches a second.

#52 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines

What are high context switch rates? 1000 HZ? Generally it seems that context switches are bad for cpu caches and thus to be avoided.

We're talking about in excess of 250k switches a second.

Timer resources are not processor specific? Why such a high rate?

#42 lore raw
In reply to Christoph Lameter
Quoted 17 lines · Christoph Lameter
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Peter Zijlstra

Another user is RCU, the grace period is tick driven, growing these ticks by a factor 50 or so might require some tinkering with forced grace periods when we notice our batch queues getting too long.

Quoted 7 lines
Quoted 1 line

One could also schedule RCU via hrtimers with a large fuzz period?

You could, but then you would still have a periodic interrupt introducing jitter into your HPC workload. The approach I suggested allows RCU to be happy with no periodic interrupts on any CPU that has only one runnable task that is a CPU-bound user-level task (in addition to the idle task, of course).

Quoted 4 lines

Sounds good.

An HPC workload typically has minimal kernel interaction. RCU would only need to run once and then the system would be quiet.

Peter Z's post leads me to believe that there might be dragons in this approach that I am blissfully unaware of. However, here is what would have to happen from an RCU perspective, in case it helps:

o This new mode needs to imply CONFIG_NO_HZ.

o When a given CPU is transitioning into tickless mode, invoke
rcu_enter_nohz(). This already happens for dynticks-idle, this would be a dynticks-CPU-bound-usermode-task. Note that CONFIG_NO_HZ kernels already invokes rcu_enter_nohz() from tick_nohz_stop_sched_tick(), and many of the things in tick_nohz_stop_sched_tick() would need to be done in this case as well.

o When a given CPU is transitioning out of tickless mode, invoke
rcu_exit_nohz(). Again, this already happens for dynticks-idle. Note that CONFIG_NO_HZ kernels already invoke rcu_exit_nohz() from tick_nohz_restart_sched_tick(), which does other stuff that would be required in your case as well.

o When a given CPU in tickless mode transitions into the kernel
via a system call or trap, invoke rcu_irq_enter(). Note that rcu_irq_enter() is already invoked on irq entry if CONFIG_NO_HZ. NMIs are also already handled via rcu_nmi_enter().

o When a given CPU in tickless mode transitions out of the kernel
from a system call or trap, invoke rcu_irq_exit(). Note that rcu_irq_exit() is already invoked on irq exit if CONFIG_NO_HZ. NMIs are also already handled via rcu_nmi_exit().

Then RCU would know that any CPU running a CPU-bound user-mode task need not be consulted when working out when a grace period ends, since user-mode code cannot contain kernel-mode RCU read-side critical sections.

Thanx, Paul

#17 lore raw
In reply to Jeff Garzik

O> We care only about a decreasing time interval. This interval is passed

Quoted 2 lines

to register polling functions (bitbang no longer than <this> amount of time), as well as _cumulatively_ affecting the entire EH [sub-]process.

Yes so its a simple timer.

Quoted 3 lines

A timer-based solution, in addition to being an ugly hack, would imply replacing a simple variable with _at least_ two spinlocks, plus a timer callback function that simply says "I expired". With loops such as

That simple variable is incredibly incredibly expensive in power, in CPU use, in cache poisoning and more - its very misleading to think of it as "free".

Quoted 4 lines

The current code is a -lot- more simple than checking "is timer expired?" all over the code, given that any sort of timer-based function implies dealing with additional concurrency issues -- a complication the libata EH does not need.

I disagree - your implementation seems very very ugly. But even then its a case of swapping jiffies for jiffies() for unconverted code and using that function read the timer value from somewhere. The important thing is that asking the time is active and we don't burn processor time and wakeups and power going "lets wake up, lets turn the cache on, lets load some cache lines, lets increment a variable, and poke some lines on the bus waking up bits of other logic, now lets write the cache back out, and go back to sleep for a tiny amount of time"

jiffies is *really* expensive...

Even a

	jiffies = time_count_begin();

	blah blah with jiffies;

	time_count_end();

would help. But that seems a good way to add bugs.

However you don't need HZ - you are using it to implement state changes and you can do that really easily by making your timer switch a single value to the correct state for the time and in your loops just checking

	while ( ap->reset_state == WIBBLING_RANDOMLY) {
		wobble = readl(ap->ioaddr.wibble);
		mdelay(1);
		if (wobble & WIBBLED) {
			...
		}
	}

Alan

#29 lore raw
In reply to Christoph Lameter
Quoted 6 lines · On Thu, 7 May 2009 14:12:15 -0400 (EDT) Christoph Lameter <[email protected]> wrote:

To get back to the main point: I agree with Alok that the default HZ needs to be as low as possible. The remaining justification is the load balancing and possible context switching between multiple tasks contenting for a processor.

Is it enough if this occurs 100 times per second?

I would like to see evidence that it is, and by evidence I don't mean "our virtual machine manager runs better" but good evidence that it isn't affecting performance of any games, applications, video suites etc - because we know it did in the past.

Even then its pointlessly jumping up and down and changing stuff rather than fixing the real problem - which is jiffies.

Alan

#18 lore raw
In reply to Alan Cox
Quoted 1 line · Alan Cox

jiffies is *really* expensive...

Certainly.

Quoted 1 line

However you don't need HZ

Thus I said "libata would be happy to use milliseconds rather than jiffies; the unit does not matter."

All crucial constants and variables are already in useful units, and we are forced to convert them to jiffies, and test against 'jiffies' global var, because the API does not provide other reasonable alternatives.

Pretty much all users of time -- including users of timers -- are forced to convert from a useful unit of time to jiffies, because that's what the API requires.

Quoted 10 lines

and you can do that really easily by making your timer switch a single value to the correct state for the time and in your loops just checking

	while ( ap->reset_state == WIBBLING_RANDOMLY) {
		wobble = readl(ap->ioaddr.wibble);
		mdelay(1);
		if (wobble & WIBBLED) {
			...
		}
	}

And when the deadline changes, you need mod_timer rather than a simple non-global-variable increment.

And when you consider all the bits of state, spread across not only libata core __but drivers as well__, a timer-based solution gets even uglier.

But you're still missing the point. jiffies are just a unit, a unit forced upon us by all the time-based kernel API functions that take jiffies rather than usec or msec.

I would rather fix the API, than investigate hacky solutions that complicate the code even more.

Jeff

#30 lore raw
In reply to Alan Cox
Quoted 2 lines · Alan Cox

Even then its pointlessly jumping up and down and changing stuff rather than fixing the real problem - which is jiffies.

Reducing the number of application interruptions by a factor of 10 is not pointless. The lower HZ becomes the more useless it will become for the users of jiffies because they will have to create timers and use time intervals if the HZ intervals become too large.

#31 lore raw
Quoted 4 lines

Reducing the number of application interruptions by a factor of 10 is not pointless. The lower HZ becomes the more useless it will become for the users of jiffies because they will have to create timers and use time intervals if the HZ intervals become too large.

More like "the more your performance goes down the toilet" - as everything is rounded up.

If you want to follow that argument set the default HZ to 1. Then see if it has any side effects.

Given HZ=1 doesn't work, its fairly clear that all you are doing is frobbing pointlessly with defaults that have been evolved over about 15 years because they work.

Whereas if you make jiffies jiffies() and it reads a timer it puts the pain in the right place and you can begin to make actual progress to making things work properly.

#56 lore raw
In reply to Alok Kataria
Quoted 43 lines · Alok Kataria
Quoted 41 lines · Chris Snook
Quoted 13 lines · Alok Kataria
Quoted 10 lines · H. Peter Anvin
Quoted 7 lines · Alok Kataria

Hi,

Given that there were no major objections that came up regarding reducing the HZ value in http://lkml.org/lkml/2009/4/27/499.

Below is the patch which actually reduces it, please consider for tip.

What is the benefit of this?

Quoted 7 lines

I did some experiments on linux 2.6.29 guests running on VMware and noticed that the number of timer interrupts could have some slowdown on the total throughput on the system.
A simple tight loop experiment showed that with HZ=1000 we took about 264sec to complete the loop and that same loop took about 255sec with HZ=100.
You can find more information here http://lkml.org/lkml/2009/4/28/401

This is why certain niches, such as HPC users, often prefer HZ=100 kernels.  For the rest of us, sacrificing a few percent CPU throughput for significant latency gains is well worth it.

Quoted 14 lines

And with HRT i don't see any downsides in terms of increased latencies for device timer's or anything of that sought.

Quoted 4 lines

I can see at least one immediate downside: some timeout values in the kernel are still maintained in units of HZ (like poll, I believe), and so with a lower HZ value we'll have higher roundoff errors.

If that at all is such a big problem shouldn't we think about moving to using schedule_hrtimeout for such cases rather than relying on jiffy based timeouts.
The hrtimer explanation over here http://www.tglx.de/hrtimers.html
also talks about where these HZ (timer wheel) based timeouts be used and shouldn't really be dependent on accurate timing.

Quoted 5 lines
Quoted 1 line

But your patch doesn't do this.

The reason it doesn't do it is because poll and select already use hrtimer. So IMO no important subsystem relies on jiffies for wakeups. Thus the latency problem is not actually present in the kernel.

TCP/IP still uses jiffies. There's been talk of changing that, but it hasn't been done yet, and it's definitely a latency-critical subsystem.

Quoted 7 lines
Quoted 3 lines

 If you want us to merge a patch that
makes VMware systems faster, we're a lot more likely to take it if it make everyone else's systems faster, or at least not slower.

I doubt it would make any system slower, running these simple experiments is not hard at all and one could run these on native system too to check.

If this patch improves performance for both simple loops and transaction processing by changing a non-idiotic tuning parameter, it would be a first. Can you at least run some sort of database benchmark to back this up?

Quoted 26 lines
Quoted 9 lines
Quoted 8 lines

Also the default HZ value was 250 before this commit

commit 5cb04df8d3f03e37a19f2502591a84156be71772
 x86: defconfig updates

And it was 250 for a very long time before that too. The commit log doesn't explain why the value was bumped up either.

Quoted 7 lines

250 was considered a compromise between 100 and 1000, but almost everyone who cared just ended up using one or the other, and most of them preferred 1000.

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway.

Yeah but I don't think there is any better platform other than LKML to figure out if at all this is a problem anymore. Once we are assured that a low HZ is no more a problem I don't see why would the various distros not consider reducing it.

Quoted 3 lines

  If you can
figure out a way to do that without harming latency, they'll be thrilled.

Quoted 8 lines

Why do you think it would harm latency ?
The sched_tick too is driven by hrtimers, if there is any specific subsystem which you think still relies on jiffy we could think about using hrtimer's for them too, right ?
I did a quick scan and the only things that rely on jiffy are the device timeout's where latency is not a issue.
So please let me know in what cases do you think it could affect system latency.

If you can get TCP/IP converted, or convince me that this won't hurt transaction processing, I'm sold.

-- Chris

#32 lore raw
In reply to Alan Cox
Quoted 3 lines · Alan Cox

Given HZ=1 doesn't work, its fairly clear that all you are doing is frobbing pointlessly with defaults that have been evolved over about 15 years because they work.

Are you sure that 1000 HZ default was not developed due to the need to have sufficient time granulaity for poll and select? In most of the 15 years we did not have high resolution timers.

Quoted 3 lines

Whereas if you make jiffies jiffies() and it reads a timer it puts the pain in the right place and you can begin to make actual progress to making things work properly.

Good idea. A patch would be appreciated.

#19 lore raw
In reply to Jeff Garzik
Quoted 2 lines

I would rather fix the API, than investigate hacky solutions that complicate the code even more.

The fundamental problem is that the API that is wrong is "jiffies". The false notion that there is some magical free timer variable.

#57 lore raw
In reply to Chris Snook
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway.

As a side note Red Hat ships runtime configurable tick behaviour in RHEL these days. HZ is fixed but the ticks can be bunched up. That was done as a quick fix to keep stuff portable but its a lot more sensible than randomly messing with the HZ value and its not much code either.

#53 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

The problem with scheduling a hrtimer along with tasks is that at high context switch rates the timer will never fire but you do pay the overhead of programming the hardware each time, something that can be about as expensive as the whole context switch itself.

Although, I guess we could amortize that by not re-programming the timer when the existing timer is within a reasonable period (say 1ms) of the requested on.

That seems like a reasonable optimization since we're talking about multi-kHz context switch rates, here.

-hpa

#58 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway.

As a side note Red Hat ships runtime configurable tick behavior in RHEL these days. HZ is fixed but the ticks can be bunched up. That was done as a quick fix to keep stuff portable but its a lot more sensible than randomly messing with the HZ value and its not much code either.

That's interesting!
Could you please point us to the patch if you can?

(As a HPC + virtualization shop, we set HZ to 100 all the time, and a patch like the one you mention above sounds great)

Thanks,
Kiran

#59 lore raw
In reply to Alan Cox
Quoted 9 lines · Alan Cox
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Given your use case, what you really need to do is get Red Hat, Novell, et al. on the phone and ask them to ship kernels with HZ=100, because the distributions do their own thing anyway.

As a side note Red Hat ships runtime configurable tick behaviour in RHEL these days. HZ is fixed but the ticks can be bunched up. That was done as a quick fix to keep stuff portable but its a lot more sensible than randomly messing with the HZ value and its not much code either.

Hi Alan,

I guess you are talking about the tick_divider patch ?
And that's still same as reducing the HZ value only that it can be done dynamically (boot time), right ?

Thanks,
Alok

#60 lore raw
Quoted 10 lines
Quoted 5 lines

As a side note Red Hat ships runtime configurable tick behaviour in RHEL these days. HZ is fixed but the ticks can be bunched up. That was done as a quick fix to keep stuff portable but its a lot more sensible than randomly messing with the HZ value and its not much code either.

Hi Alan,

I guess you are talking about the tick_divider patch ?
And that's still same as reducing the HZ value only that it can be done dynamically (boot time), right ?

Yes - which has the advantage that you can select different behaviours rather than distributions having to build with HZ=1000 either for compatibility or responsiveness can still allow users to drop to a lower HZ value if doing stuff like HPC.

Basically it removes the need to argue about it at build time and lets the user decide.

#33 lore raw
In reply to Paul E. McKenney
Quoted 38 lines · Paul E. McKenney

In general, I agree. However, in the case where you have a single CPU-bound task running in user mode, you don't care that much about syscall performance. So, yes, this would mean having yet another config variable that users running big CPU-bound scientific applications would need to worry about, which is not perfect either.

For whatever it is worth, the added overhead on entry would be something like the following:

void rcu_irq_enter(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (rdtp->dynticks_nesting++)
		return;
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(!(rdtp->dynticks & 0x1), &rcu_rs);
	smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */
}

On exit, a bit more:

void rcu_irq_exit(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (--rdtp->dynticks_nesting)
		return;
	smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(rdtp->dynticks & 0x1, &rcu_rs);

	/* If the interrupt queued a callback, get out of dyntick mode. */
	if (__get_cpu_var(rcu_data).nxtlist ||
	    __get_cpu_var(rcu_bh_data).nxtlist)
		set_need_resched();
}
Quoted 2 lines

But I could move the callback check into call_rcu(), which would get the overhead of rcu_irq_exit() down to about that of rcu_irq_enter().

Can't you simply enter idle state after a grace period completes and finds no pending callbacks for the next period. And leave idle state at the next call_rcu()?

#34 lore raw
Quoted 40 lines · Peter Zijlstra
Quoted 38 lines · Paul E. McKenney

In general, I agree. However, in the case where you have a single CPU-bound task running in user mode, you don't care that much about syscall performance. So, yes, this would mean having yet another config variable that users running big CPU-bound scientific applications would need to worry about, which is not perfect either.

For whatever it is worth, the added overhead on entry would be something like the following:

void rcu_irq_enter(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (rdtp->dynticks_nesting++)
		return;
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(!(rdtp->dynticks & 0x1), &rcu_rs);
	smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */
}

On exit, a bit more:

void rcu_irq_exit(void)

{
	struct rcu_dynticks *rdtp = &__get_cpu_var(rcu_dynticks);

	if (--rdtp->dynticks_nesting)
		return;
	smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */
	rdtp->dynticks++;
	WARN_ON_RATELIMIT(rdtp->dynticks & 0x1, &rcu_rs);

	/* If the interrupt queued a callback, get out of dyntick mode. */
	if (__get_cpu_var(rcu_data).nxtlist ||
	    __get_cpu_var(rcu_bh_data).nxtlist)
		set_need_resched();
}
Quoted 6 lines
Quoted 2 lines

But I could move the callback check into call_rcu(), which would get the overhead of rcu_irq_exit() down to about that of rcu_irq_enter().

Can't you simply enter idle state after a grace period completes and finds no pending callbacks for the next period. And leave idle state at the next call_rcu()?

If there were no RCU callbacks -globally- across all CPUs, yes. But the check at the end of rcu_irq_exit() is testing only on the current CPU. Checking across all CPUs is expensive and racy.

So what happens instead is that there is rcu_needs_cpu(), which gates entry into dynticks-idle mode. This function returns 1 if there are callbacks on the current CPU. So, if no CPU has an RCU callback, then all CPUs can enter dynticks-idle mode so that the entire system is quiescent from an RCU viewpoint -- no RCU processing at all.

Or am I missing what you are getting at with your question?

Thanx, Paul

#35 lore raw
Quoted 8 lines · Paul E. McKenney
Quoted 3 lines

Can't you simply enter idle state after a grace period completes and finds no pending callbacks for the next period. And leave idle state at the next call_rcu()?

If there were no RCU callbacks -globally- across all CPUs, yes. But the check at the end of rcu_irq_exit() is testing only on the current CPU. Checking across all CPUs is expensive and racy.

Quoted 5 lines

So what happens instead is that there is rcu_needs_cpu(), which gates entry into dynticks-idle mode. This function returns 1 if there are callbacks on the current CPU. So, if no CPU has an RCU callback, then all CPUs can enter dynticks-idle mode so that the entire system is quiescent from an RCU viewpoint -- no RCU processing at all.

Did not follow RCU developments. But wasnt there a time when RCU periods were processor specific and a global RCU period was done when all the processors went through their rcu periods?

Cpu cache hotness may not be relevant to RCU since rcu involves long time periods in which cachelines cool down. Can the RCU callbacks all be done on processor 0 (or a so designated processor)? That would avoiding disturbances of the other processors.

#36 lore raw
Quoted 16 lines · Christoph Lameter
Quoted 8 lines · Paul E. McKenney
Quoted 3 lines

Can't you simply enter idle state after a grace period completes and finds no pending callbacks for the next period. And leave idle state at the next call_rcu()?

If there were no RCU callbacks -globally- across all CPUs, yes. But the check at the end of rcu_irq_exit() is testing only on the current CPU. Checking across all CPUs is expensive and racy.

Quoted 5 lines

So what happens instead is that there is rcu_needs_cpu(), which gates entry into dynticks-idle mode. This function returns 1 if there are callbacks on the current CPU. So, if no CPU has an RCU callback, then all CPUs can enter dynticks-idle mode so that the entire system is quiescent from an RCU viewpoint -- no RCU processing at all.

Quoted 3 lines

Did not follow RCU developments. But wasnt there a time when RCU periods were processor specific and a global RCU period was done when all the processors went through their rcu periods?

For non-realtime RCU implementations, after a given grace period starts, once each CPU goes through a "quiescent state", then that grace period can end. For realtime (AKA "preemptable") RCU, the focus is on tasks rather than CPUs, but the same general principle applies, give or take some implementation details: after a given grace period starts, once each task goes through a quiescent state, then that grace period can end.

Quoted 4 lines

Cpu cache hotness may not be relevant to RCU since rcu involves long time periods in which cachelines cool down. Can the RCU callbacks all be done on processor 0 (or a so designated processor)? That would avoiding disturbances of the other processors.

This approach -might- be OK for a specially configured and protected HPC machine. But it is a non-starter for general-purpose machines. For an example of why, consider a denial-of-service attack that continually change routing tables could saturate CPU 0 and start falling behind, eventually OOMing the machine.

But if you would like to experiment with this, make call_rcu() be a wrapper that causes an underlying call_rcu_cpu_0() to be executed on CPU 0. That won't get exactly the cache-warmth effects that you are after, but it will let you see whether the machine would gracefully handle various events that might dump large numbers of callbacks.

Thanx, Paul