3.8-rc6-nohz4
Hi,
The support for printk and cputime accounting to work on full dynticks CPUs have been merged in -tip tree and is likely deemed for the 3.9 merge window. So this new release is a rebase against the relevant branches in -tip and v3.8-rc6.
The remaining amount of patches has thus quite schrinked.
You can pull this branch from:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
3.8-rc6-nohz4
Remember it doesn't yet support everything the tick does, namely it lacks full support for:
- Posix CPU timers
- Perf events
- sched_class::task_tick()
- various other scheduler details
- ...
So use it with caution!
Thanks.
PS: next upstream focus will probably be on posix cpu timers. May be we can try to make it work using timer_list or hrtimers, I don't know yet. I need to experiment.
---
Changes since 3.8-rc4-nohz3 (including those of cputime):
* Rebase against v3.8-rc6 and latest tip:/sched/core and tip:/irq/core
* Fix cputime build error with kvm modules (Thanks Sedat Dilek and Wu Fenguang)
* Fix cputime mistyped header inclusion in ia64
* Fix more missing symbols for kvm
* Removal of profiling's timer hook also applied in -tip
---
Frederic Weisbecker (26):
nohz: Basic full dynticks interface
nohz: Assign timekeeping duty to a non-full-nohz CPU
nohz: Trace timekeeping update
nohz: Wake up full dynticks CPUs when a timer gets enqueued
rcu: Restart the tick on non-responding full dynticks CPUs
sched: Comment on rq->clock correctness in ttwu_do_wakeup() in nohz
sched: Update rq clock on nohz CPU before migrating tasks
sched: Update rq clock on nohz CPU before setting fair group shares
sched: Update rq clock on tickless CPUs before calling
check_preempt_curr()
sched: Update rq clock earlier in unthrottle_cfs_rq
sched: Update clock of nohz busiest rq before balancing
sched: Update rq clock before idle balancing
sched: Update nohz rq clock before searching busiest group on load
balancing
nohz: Move nohz load balancer selection into idle logic
nohz: Full dynticks mode
nohz: Only stop the tick on RCU nocb CPUs
nohz: Don't turn off the tick if rcu needs it
nohz: Don't stop the tick if posix cpu timers are running
nohz: Add some tracing
rcu: Don't keep the tick for RCU while in userspace
timer: Don't run non-pinned timer to full dynticks CPUs
sched: Use an accessor to read rq clock
sched: Debug nohz rq clock
sched: Remove broken check for skip clock update
sched: Update rq clock before rt sched average scale
sched: Disable lb_bias feature for full dynticks
include/linux/posix-timers.h | 1 + include/linux/rcupdate.h | 8 +++ include/linux/sched.h | 10 +++- include/linux/tick.h | 9 +++ kernel/hrtimer.c | 3 +- kernel/posix-cpu-timers.c | 11 ++++ kernel/rcutree.c | 19 +++++-- kernel/rcutree.h | 1 - kernel/rcutree_plugin.h | 13 +--- kernel/sched/core.c | 104 ++++++++++++++++++++++++++++++--- kernel/sched/fair.c | 96 ++++++++++++++++++++++-------- kernel/sched/features.h | 3 + kernel/sched/rt.c | 8 +- kernel/sched/sched.h | 50 ++++++++++++++++ kernel/sched/stats.h | 8 +- kernel/sched/stop_task.c | 8 +- kernel/softirq.c | 5 +- kernel/time/Kconfig | 9 +++ kernel/time/tick-broadcast.c | 3 +- kernel/time/tick-common.c | 5 +- kernel/time/tick-sched.c | 132 +++++++++++++++++++++++++++++++++++++---- kernel/timer.c | 5 +- 22 files changed, 427 insertions(+), 84 deletions(-)
Signature · 1 line
1.7.5.4
I'll reply to this as I come up with comments.
First thing is, don't call it NO_HZ_FULL. A better name would be NO_HZ_CPU. I would like to reserve NO_HZ_FULL when we totally remove jiffies :-)
And the kconfig help should probably call it "Adaptive tickless" or "Tickless for single tasks". The full tickless system really sounds like we totally removed jiffies. It should explain it better. Something like:
"Adaptive tickless system"
With this option, you may designate CPUs that will turn off the periodic interrupt "tick" when only a single task is scheduled on the CPU. This is similar to NO_HZ where the tick is suspended when the CPU goes into idle. With this option, it takes it one step further. When only a single task is scheduled on the CPU, there scheduler does not need to keep track of time slices, as the running task does not need to be preempted for other tasks. Stopping the tick allows the task to avoid being interrupted by service routines by the kernel.
CPUs must be designated at time of boot via the kernel command line parameter (cpu_nohz) and must be a subset of the rcu_nocb parameter, which prevents RCU service routines from being called on the CPUs as well.
---
Something like that.
-- Steve
Quoted 5 lines · Steven Rostedt
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)
I don't think we want yet another config option named in a weird way.
What we want instead is to just split NO_HZ up into its conceptual parts:
CONFIG_NO_HZ_IDLE
CONFIG_NO_HZ_USER_SPACE
CONFIG_NO_HZ_KERNEL_SPACE
Where the current status quo is NO_HZ_IDLE=y, and Frederic is about to introduce NO_HZ_USER_SPACE=y. When jiffies get removed we get NO_HZ_KERNEL_SPACE=y.
The 'CONFIG_NO_HZ' meta-option, which we should leave for easy configurability and for compatibility, should get us the currently recommended default, which for the time being might be:
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_USER_SPACE is disabled
Btw., you could add CONFIG_NO_HZ_KERNEL_SPACE right away, just keep it false all the time. That would document our future plans pretty well.
Once CONFIG_NO_HZ_USER_SPACE is proven problem-free, we might default to:
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_USER_SPACE=y
The goal is to have this in the distant future:
CONFIG_NO_HZ=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_USER_SPACE=y
CONFIG_NO_HZ_KERNEL_SPACE=y
And eventually we might even be able to get rid of all the 3 variants, and only offer full-on/off.
Agreed?
Thanks,
Ingo
Quoted 1 line · Ingo Molnar
Agreed?
Yes and please also change the texts in Kconfig to accurately describe what happens to the timer tick.
Quoted 14 lines · Ingo Molnar
Quoted 5 lines · Steven Rostedt
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)I don't think we want yet another config option named in a weird way.
What we want instead is to just split
NO_HZup into its conceptual parts:
Quoted 7 lines
CONFIG_NO_HZ_IDLE
CONFIG_NO_HZ_USER_SPACE
CONFIG_NO_HZ_KERNEL_SPACEWhere the current status quo is NO_HZ_IDLE=y, and Frederic is about to introduce NO_HZ_USER_SPACE=y. When jiffies get removed we get NO_HZ_KERNEL_SPACE=y.
Saying NO_HZ_USER_SPACE is a bit of a misnomer. As we don't just stop the tick for user space, but it may remained stopped when entering the kernel. The rule is that when there's just a single task on a CPU, the tick can stop (no scheduling work needed). But if the task triggers something that may require a tick (like printk) then the tick will start again. But just going into the kernel does not designate a tick restart.
Maybe a better name would be NO_HZ_SINGLE_TASK ?
Quoted 6 lines
The '
CONFIG_NO_HZ' meta-option, which we should leave for easy configurability and for compatibility, should get us the currently recommended default, which for the time being might be:
Quoted 6 lines
CONFIG_NO_HZ_IDLE=y
#CONFIG_NO_HZ_USER_SPACEis disabledBtw., you could add
CONFIG_NO_HZ_KERNEL_SPACEright away, just keep it false all the time. That would document our future plans pretty well.
Maybe the removal of jiffies would be NO_HZ_COMPLETE?
-- Steve
2013/2/7 Ingo Molnar <[email protected]>:
Quoted 9 lines · Steven Rostedt
Quoted 6 lines
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)
Quoted 7 lines
I don't think we want yet another config option named in a weird way.
What we want instead is to just split
NO_HZup into its conceptual parts:
CONFIG_NO_HZ_IDLE
Renaming CONFIG_NO_HZ to CONFIG_NO_HZ_IDLE is something I considered. I was just worried about this option being present in many defconfig. Perhaps we can do that renaming and keep CONFIG_NO_HZ around a little while for backward compatibility (pretty much like what we've done for CONFIG_PERF_COUNTERS -> CONFIG_PERF_EVENTS).
Quoted 6 lines
CONFIG_NO_HZ_USER_SPACE
CONFIG_NO_HZ_KERNEL_SPACEWhere the current status quo is NO_HZ_IDLE=y, and Frederic is about to introduce NO_HZ_USER_SPACE=y. When jiffies get removed we get NO_HZ_KERNEL_SPACE=y.
Note on my tree I stop the tick on both rings. I believe that restarting the tick on kernel entry isn't something we should seriously consider. It would be a costly operation that may make things worse. And in fact there is no big difference. Just kernelspace has more opportunities to be disturbed (RCU IPIs, async timer/work scheduled by the kernel, etc...) and get its tick restarted sometimes.
Quoted 5 lines
The '
CONFIG_NO_HZ' meta-option, which we should leave for easy configurability and for compatibility, should get us the currently recommended default, which for the time being might be:
Ah looks like you considered the compatibility as well :)
Quoted 18 lines
CONFIG_NO_HZ_IDLE=y
#CONFIG_NO_HZ_USER_SPACEis disabledBtw., you could add
CONFIG_NO_HZ_KERNEL_SPACEright away, just keep it false all the time. That would document our future plans pretty well.Once
CONFIG_NO_HZ_USER_SPACEis proven problem-free, we might default to:
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_USER_SPACE=yThe goal is to have this in the distant future:
CONFIG_NO_HZ=y
Quoted 8 lines
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ_USER_SPACE=y
CONFIG_NO_HZ_KERNEL_SPACE=yAnd eventually we might even be able to get rid of all the 3 variants, and only offer full-on/off.
Agreed?
At least for now we seem to agree on CONFIG_NO_HZ_IDLE and keep CONFIG_NO_HZ for compatibility. Are you ok with that? If so I'll send a patch.
Quoted 3 lines · Frederic Weisbecker
At least for now we seem to agree on
CONFIG_NO_HZ_IDLEand keepCONFIG_NO_HZfor compatibility. Are you ok with that? If so I'll send a patch.
I believe that Ingo was suggesting to have CONFIG_NO_HZ give options to what type of config NO_HZ you want. Something like:
config NO_HZ
bool "Enable tickless support"
config NO_HZ_IDLE
bool "Stop tick when CPU is idle"
default y
depends on NO_HZ
config NO_HZ_TASK
bool "Stop tick on specified CPUs when single task is running" default n
depends on NO_HZ
That is, if you select NO_HZ, by default NO_HZ_IDLE is also selected. But in the kernel the NO_HZ_IDLE is used.
-- Steve
2013/2/7 Steven Rostedt <[email protected]>:
Quoted 5 lines
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)
I'm not sure we'll ever be able to completely remove the tick, even if jiffies is removed.
Ok in any case, NO_HZ_ADAPTIVE is probably a more accurate name.
Quoted 6 lines
And the kconfig help should probably call it "Adaptive tickless" or "Tickless for single tasks". The full tickless system really sounds like we totally removed jiffies. It should explain it better. Something like:
"Adaptive tickless system"
Right the problem with "tickless" is its meaning of absolute removal. "Full dynticks" is what I think reflect best what's happening.
Quoted 9 lines
With this option, you may designate CPUs that will turn off the periodic interrupt "tick" when only a single task is scheduled on the CPU. This is similar to
NO_HZwhere the tick is suspended when the CPU goes into idle. With this option, it takes it one step further. When only a single task is scheduled on the CPU, there scheduler does not need to keep track of time slices, as the running task does not need to be preempted for other tasks. Stopping the tick allows the task to avoid being interrupted by service routines by the kernel.
Quoted 8 lines
CPUs must be designated at time of boot via the kernel command line parameter (
cpu_nohz) and must be a subset of thercu_nocbparameter, which prevents RCU service routines from being called on the CPUs as well.---
Something like that.
I'm not convinced that "single task" must be a fundamental component of this. It's an implementation detail. We should be able to keep the tick off in the future when more than one task are on the runqueue and hrtick is on. May be this will never show up as a performance gain but we don't know yet.
Ok let's talk about that single task constraint in the Kconfig help so that the user knows the practical constraint as of today. But I suggest we keep that as an internal detail that we can deal with in the future.
Hm?
2013/2/7 Steven Rostedt <[email protected]>:
Quoted 9 lines · Frederic Weisbecker
Quoted 4 lines
At least for now we seem to agree on
CONFIG_NO_HZ_IDLEand keepCONFIG_NO_HZfor compatibility. Are you ok with that? If so I'll send a patch.I believe that Ingo was suggesting to have
CONFIG_NO_HZgive options to what type of configNO_HZyou want. Something like:
Quoted 7 lines
config
NO_HZ
bool "Enable tickless support"config
NO_HZ_IDLE
bool "Stop tick when CPU is idle"
default y
depends onNO_HZ
Sounds good!
Quoted 5 lines
config
NO_HZ_TASK
bool "Stop tick on specified CPUs when single task is running" default n
depends onNO_HZ
Ok I launched another debate about that single task thing. I wish we don't make it a fundamental component but rather an implementation detail that can be dynamically dealt with in the future. Anyway let's talk about that on my previous answer.
Quoted 3 lines
That is, if you select
NO_HZ, by defaultNO_HZ_IDLEis also selected. But in the kernel theNO_HZ_IDLEis used.
Yeah, nice idea!
Quoted 4 lines
-- Steve
Quoted 38 lines · Steven Rostedt
Quoted 14 lines · Ingo Molnar
Quoted 5 lines · Steven Rostedt
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)I don't think we want yet another config option named in a weird way.
What we want instead is to just split
NO_HZup into its conceptual parts:Quoted 7 lines
CONFIG_NO_HZ_IDLE
CONFIG_NO_HZ_USER_SPACE
CONFIG_NO_HZ_KERNEL_SPACEWhere the current status quo is NO_HZ_IDLE=y, and Frederic is about to introduce NO_HZ_USER_SPACE=y. When jiffies get removed we get NO_HZ_KERNEL_SPACE=y.
Saying
NO_HZ_USER_SPACEis a bit of a misnomer. As we don't just stop the tick for user space, but it may remained stopped when entering the kernel. The rule is that when there's just a single task on a CPU, the tick can stop (no scheduling work needed). But if the task triggers something that may require a tick (like printk) then the tick will start again. But just going into the kernel does not designate a tick restart.Maybe a better name would be
NO_HZ_SINGLE_TASK?Quoted 6 lines
The '
CONFIG_NO_HZ' meta-option, which we should leave for easy configurability and for compatibility, should get us the currently recommended default, which for the time being might be:
Quoted 8 lines
Quoted 6 lines
CONFIG_NO_HZ_IDLE=y
#CONFIG_NO_HZ_USER_SPACEis disabledBtw., you could add
CONFIG_NO_HZ_KERNEL_SPACEright away, just keep it false all the time. That would document our future plans pretty well.Maybe the removal of jiffies would be
NO_HZ_COMPLETE?
I suspect that removal of jiffies from the kernel will take a few stages, with RCU being one of the laggards for awhile. Making RCU's state machine depend wholly on process-based execution will take some care and experimentation, especially for extreme and corner-case workloads. For example, having RCU OOM the system just because a specific CPU was unable to run some RCU kthread for an extended time is something to be avoided. ;-)
Thanx, Paul
Quoted 10 lines · Frederic Weisbecker
I'm not convinced that "single task" must be a fundamental component of this. It's an implementation detail. We should be able to keep the tick off in the future when more than one task are on the runqueue and hrtick is on. May be this will never show up as a performance gain but we don't know yet.
Ok let's talk about that single task constraint in the
Kconfighelp so that the user knows the practical constraint as of today. But I suggest we keep that as an internal detail that we can deal with in the future.
Hmm, but isn't time slices still implemented by ticks? I would think implementing multiple tasks would be another huge change.
Maybe have:
NO_HZ_IDLENO_HZ_SINGLE_TASKNO_HZ_MULTI_TASKNO_HZ_COMPLETE
And as Ingo has suggested, maybe in the future we can remove SINGLE and MULTI and have just COMPLETE.
But anyway, the current method has a strict requirement of a single task, and that is user visible. I would want to keep the config name implying that requirement.
-- Steve
Quoted 9 lines · Frederic Weisbecker
Quoted 5 lines
config
NO_HZ_TASK
bool "Stop tick on specified CPUs when single task is running" default n
depends onNO_HZOk I launched another debate about that single task thing. I wish we don't make it a fundamental component but rather an implementation detail that can be dynamically dealt with in the future.
It's not just an implementation detail, as it is very visible to the user. If they want to take advantage of a task NO_HZ they have to go through a bit of loops to make sure only a single task is running on a CPU. We should be broadcasting this requirement to educate the users on exactly how they can take advantage of this feature.
Quoted 2 lines
Anyway let's
talk about that on my previous answer.
I already did ;-)
-- Steve
Quoted 7 lines · Paul E. McKenney
I suspect that removal of jiffies from the kernel will take a few stages, with RCU being one of the laggards for awhile. Making RCU's state machine depend wholly on process-based execution will take some care and experimentation, especially for extreme and corner-case workloads. For example, having RCU OOM the system just because a specific CPU was unable to run some RCU kthread for an extended time is something to be avoided. ;-)
Tickless doesn't mean no timeouts or periodic timers. I think we will always have some sort of dynamic tick when needed. It will just be more event driven then something that goes off constantly.
-- Steve
2013/2/7 Steven Rostedt <[email protected]>:
Quoted 8 lines · Frederic Weisbecker
Quoted 7 lines
I'm not convinced that "single task" must be a fundamental component of this. It's an implementation detail. We should be able to keep the tick off in the future when more than one task are on the runqueue and hrtick is on. May be this will never show up as a performance gain but we don't know yet.
Quoted 6 lines
Quoted 4 lines
Ok let's talk about that single task constraint in the
Kconfighelp so that the user knows the practical constraint as of today. But I suggest we keep that as an internal detail that we can deal with in the future.Hmm, but isn't time slices still implemented by ticks?
Not with hrtick.
Quoted 2 lines
I would think
implementing multiple tasks would be another huge change.
I don't think so. Really hrtick should take of everything.
Quoted 7 lines
Maybe have:
NO_HZ_IDLENO_HZ_SINGLE_TASKNO_HZ_MULTI_TASKNO_HZ_COMPLETE
I still see single task, multitask or complete as implementation constraints. Once we make hrtick support dynticks, it should be dynamically handled: if hrtick is enabled then stop the tick even on multitask, otherwise only stop it when we have one task.
Then when we remove jiffies, the complete coverage comes along.
Quoted 3 lines
And as Ingo has suggested, maybe in the future we can remove SINGLE and MULTI and have just COMPLETE.
But really, turning these constraints into single built-in optable choices doesn't make much sense to me.
Quoted 3 lines
But anyway, the current method has a strict requirement of a single task, and that is user visible. I would want to keep the config name implying that requirement.
As long as it's specified in the Kconfig help, does it matter? It''s a constraint amongst many others: you need to keep one CPU with a periodic tick, you need to avoid posix cpu timers, etc...
Quoted 10 lines · Steven Rostedt
Quoted 7 lines · Paul E. McKenney
I suspect that removal of jiffies from the kernel will take a few stages, with RCU being one of the laggards for awhile. Making RCU's state machine depend wholly on process-based execution will take some care and experimentation, especially for extreme and corner-case workloads. For example, having RCU OOM the system just because a specific CPU was unable to run some RCU kthread for an extended time is something to be avoided. ;-)
Quoted 3 lines
Tickless doesn't mean no timeouts or periodic timers. I think we will always have some sort of dynamic tick when needed. It will just be more event driven then something that goes off constantly.
As long as we don't end up replacing a single tick with multiple hrtimers (or whatever), ending up with more overhead and disruption than we started with. ;-)
Thanx, Paul
2013/2/7 Steven Rostedt <[email protected]>:
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines
Quoted 6 lines
config
NO_HZ_TASK
bool "Stop tick on specified CPUs when single task is running" default n
depends onNO_HZOk I launched another debate about that single task thing. I wish we don't make it a fundamental component but rather an implementation detail that can be dynamically dealt with in the future.
Quoted 5 lines
It's not just an implementation detail, as it is very visible to the user. If they want to take advantage of a task
NO_HZthey have to go through a bit of loops to make sure only a single task is running on a CPU. We should be broadcasting this requirement to educate the users on exactly how they can take advantage of this feature.
If you guys really insist I can make it CONFIG_NO_HZ_SINGLETASK. I don't mind that much. Then when we support hrtick we can rename it to NO_HZ_FULL or whatever.
Quoted 13 lines · Frederic Weisbecker
2013/2/7 Ingo Molnar <[email protected]>:
Quoted 12 lines · Steven Rostedt
Quoted 6 lines
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)I don't think we want yet another config option named in a weird way.
Quoted 8 lines
Quoted 4 lines
What we want instead is to just split
NO_HZup into its conceptual parts:
CONFIG_NO_HZ_IDLERenaming
CONFIG_NO_HZtoCONFIG_NO_HZ_IDLEis something I considered. I was just worried about this option being present in many defconfig.
I don't think renaming it is an option - it's present not just in defconfigs, but in various distro configs, etc.
But we can add new config variables and use the existing CONFIG_NO_HZ value to set their default values.
Quoted 3 lines
Perhaps we can do that renaming and keep
CONFIG_NO_HZaround a little while for backward compatibility (pretty much like what we've done forCONFIG_PERF_COUNTERS->CONFIG_PERF_EVENTS).
Yes.
Quoted 7 lines
Quoted 6 lines
CONFIG_NO_HZ_USER_SPACE
CONFIG_NO_HZ_KERNEL_SPACEWhere the current status quo is NO_HZ_IDLE=y, and Frederic is about to introduce NO_HZ_USER_SPACE=y. When jiffies get removed we get NO_HZ_KERNEL_SPACE=y.
Quoted 7 lines
Note on my tree I stop the tick on both rings. I believe that restarting the tick on kernel entry isn't something we should seriously consider. It would be a costly operation that may make things worse. And in fact there is no big difference. Just kernelspace has more opportunities to be disturbed (RCU IPIs, async timer/work scheduled by the kernel, etc...) and get its tick restarted sometimes.
Ok.
Could we just simplify things and make this an unconditional option of NO_HZ? Any reason why we'd want to make this configurable, other than debugging?
I'm worried about the proliferation of not easily separable config options. We already have way too many timer and scheduler options to begin with.
Quoted 3 lines
At least for now we seem to agree on
CONFIG_NO_HZ_IDLEand keepCONFIG_NO_HZfor compatibility. Are you ok with that? If so I'll send a patch.
What would be the name of the new config option?
Can we just keep CONFIG_NO_HZ and extend it with your bits, and make sure they work well?
Thanks,
Ingo
Quoted 1 line · Frederic Weisbecker
Not with hrtick.
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Quoted 3 lines · Ingo Molnar
Could we just simplify things and make this an unconditional option of
NO_HZ? Any reason why we'd want to make this configurable, other than debugging?
I think the worry is the overhead that is required to keep it active. It requires the context_tracking being enabled. Although, we may be able to have both working.
Frederic, can we switch between context_tracking timing and tick base at run time?
If we can have it enabled without overhead then I see no problem with it. We still need the boot time kernel parameter to implement it. Hmm, even if we can't dynamically switch between context_tracking and tick base, we could make that decision at boot up based off of the kernel parameters.
Quoted 4 lines
I'm worried about the proliferation of not easily separable config options. We already have way too many timer and scheduler options to begin with.
I agree.
Quoted 9 lines
Quoted 4 lines
At least for now we seem to agree on
CONFIG_NO_HZ_IDLEand keepCONFIG_NO_HZfor compatibility. Are you ok with that? If so I'll send a patch.What would be the name of the new config option?
Can we just keep
CONFIG_NO_HZand extend it with your bits, and make sure they work well?
As long as we do not introduce performance regressions. If we can keep it active without causing the system to slow down when not in use, then I think it should be always enabled if CONFIG_NO_HZ is selected.
-- Steve
Quoted 8 lines · Christoph Lameter
Quoted 1 line · Frederic Weisbecker
Not with hrtick.
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
No, it was simply buggy.
Thanks,
Ingo
Quoted 7 lines · Christoph Lameter
Quoted 1 line · Frederic Weisbecker
Not with hrtick.
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Yup.
-Mike
2013/2/7 Ingo Molnar <[email protected]>:
Quoted 25 lines · Frederic Weisbecker
Quoted 14 lines
2013/2/7 Ingo Molnar <[email protected]>:
Quoted 12 lines · Steven Rostedt
Quoted 6 lines
I'll reply to this as I come up with comments.
First thing is, don't call it
NO_HZ_FULL. A better name would beNO_HZ_CPU. I would like to reserveNO_HZ_FULLwhen we totally remove jiffies :-)I don't think we want yet another config option named in a weird way.
Quoted 8 lines
Quoted 4 lines
What we want instead is to just split
NO_HZup into its conceptual parts:
CONFIG_NO_HZ_IDLERenaming
CONFIG_NO_HZtoCONFIG_NO_HZ_IDLEis something I considered. I was just worried about this option being present in many defconfig.
Quoted 5 lines
I don't think renaming it is an option - it's present not just in defconfigs, but in various distro configs, etc.
But we can add new config variables and use the existing
CONFIG_NO_HZvalue to set their default values.
Sure.
Quoted 10 lines
Quoted 7 lines
Note on my tree I stop the tick on both rings. I believe that restarting the tick on kernel entry isn't something we should seriously consider. It would be a costly operation that may make things worse. And in fact there is no big difference. Just kernelspace has more opportunities to be disturbed (RCU IPIs, async timer/work scheduled by the kernel, etc...) and get its tick restarted sometimes.
Ok.
Quoted 7 lines
Could we just simplify things and make this an unconditional option of
NO_HZ? Any reason why we'd want to make this configurable, other than debugging?I'm worried about the proliferation of not easily separable config options. We already have way too many timer and scheduler options to begin with.
Like Steve said, this is for overhead reasons. The syscall uses the slow path so that's ok. But we add a callback to every exception, irq entry/exit, scheduler sched switch, signal handling, user and kernel preemption point. This all could be lowered using static keys but even that doesn't make me feel comfortable with this idea.
Moreover, for now this is going to be used only on extreme usecases such as real time and HPC. If we really have to merge this into an all-in-one nohz kconfig, I suggest we wait for the feature to mature a bit and prove that it can be useful further those specialized workloads, and also that we can ensure it's off-case overhead is not significant.
2013/2/7 Christoph Lameter <[email protected]>:
Quoted 7 lines · Frederic Weisbecker
Quoted 2 lines
Not with hrtick.
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Yeah Peter said that especially reprogramming the clock everytime we call schedule() was killing the performances. Now may be on some workloads, with the tick stopped, we can find some new results.
Quoted 5 lines · Frederic Weisbecker
2013/2/7 Christoph Lameter <[email protected]>:
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Not with hrtick.
Quoted 7 lines
Quoted 3 lines
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Yeah Peter said that especially reprogramming the clock everytime we call
schedule()was killing the performances. Now may be on some workloads, with the tick stopped, we can find some new results.
I could imagine this being dynamic. If the system isn't very loaded, and the scheduler is giving lots of time slices to tasks, then perhaps it could switch to a reprogramming the clock based scheduling. Or maybe, we could switch to a "skip ticks" method. That is, instead of completely disabling the tick, make the tick go off every other time or less, and use the NO_HZ code to calculate the missed ticks.
-- Steve
Quoted 14 lines · Steven Rostedt
Quoted 5 lines · Frederic Weisbecker
2013/2/7 Christoph Lameter <[email protected]>:
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Not with hrtick.
Quoted 7 lines
Quoted 3 lines
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Yeah Peter said that especially reprogramming the clock everytime we call
schedule()was killing the performances. Now may be on some workloads, with the tick stopped, we can find some new results.
Quoted 6 lines
I could imagine this being dynamic. If the system isn't very loaded, and the scheduler is giving lots of time slices to tasks, then perhaps it could switch to a reprogramming the clock based scheduling. Or maybe, we could switch to a "skip ticks" method. That is, instead of completely disabling the tick, make the tick go off every other time or less, and use the
NO_HZcode to calculate the missed ticks.
Ok that sounds good. Automatically reducing the HZ as much as possible would also quiet down the OS and be beneficial for low latency tasks. We are configuring the kernels here with the lowest HZ that the hardware allows to reduce the number of events that impact on the app. The main problem is that the network stack becomes flaky at low HZ.
Quoted 5 lines · On Fri, 8 Feb 2013 16:53:17 +0100 Frederic Weisbecker <[email protected]>…
2013/2/7 Christoph Lameter <[email protected]>:
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
Not with hrtick.
Quoted 8 lines
Quoted 3 lines
hrtick? Did we not already try that a couple of years back and it turned out that the overhead of constantly reprogramming a timer via the PCI bus was causing too much of a performance regression?
Yeah Peter said that especially reprogramming the clock everytime we call
schedule()was killing the performances. Now may be on some workloads, with the tick stopped, we can find some new results. --
I was a little apprehensive when you started talking about multiple tasks in Adaptive NOHZ mode on a core but the more I started thinking about it, I realized that we might end up in a cooperative multitasking mode with no tick at all going. Multiple SCHED_FIFO threads could run until blocking and another would be picked. Depends on well behaved threads of course, so probably many cases of users shooting off some toes with this...
Of course if you mix scheduling policies or have RT throttling turned on we'll need some sort of tick for preemption. But if we can keep the timer reprogramming down we may see some big wins for RT and HPC loads.
Quoted 11 lines · Clark Williams
I was a little apprehensive when you started talking about multiple tasks in Adaptive NOHZ mode on a core but the more I started thinking about it, I realized that we might end up in a cooperative multitasking mode with no tick at all going. Multiple
SCHED_FIFOthreads could run until blocking and another would be picked. Depends on well behaved threads of course, so probably many cases of users shooting off some toes with this...Of course if you mix scheduling policies or have RT throttling turned on we'll need some sort of tick for preemption. But if we can keep the timer reprogramming down we may see some big wins for RT and HPC loads.
We could tune the (hr)timer tick to have the same interval as the time slice interval for a process and make that constant for all processes on a hardware thread?
Quoted 11 lines · Frederic Weisbecker
Quoted 3 lines
I'm worried about the proliferation of not easily separable config options. We already have way too many timer and scheduler options to begin with.
Like Steve said, this is for overhead reasons. The syscall uses the slow path so that's ok. But we add a callback to every exception, irq entry/exit, scheduler sched switch, signal handling, user and kernel preemption point. This all could be lowered using static keys but even that doesn't make me feel comfortable with this idea.
Quoted 6 lines
Moreover, for now this is going to be used only on extreme usecases such as real time and HPC. If we really have to merge this into an all-in-one nohz kconfig, I suggest we wait for the feature to mature a bit and prove that it can be useful further those specialized workloads, and also that we can ensure it's off-case overhead is not significant.
I have no problems with making it an option initially - as long as the options are logically named and interconnected.
In terms of overhead, a big plus is the reduction in user-space execution overhead. At HZ=1000 we easily have 0.5%-1.0% overhead currently. That is a *lot* of overhead if the box does mostly user-space execution - which most boxes do, both servers and desktops - not HPC systems.
Thanks,
Ingo