Nohz cpusets (was: Nohz Tasks)
Subsystems (17)
cover Nohz cpusets (was: Nohz Tasks)
So it's still in draft stage. It's far from covering everything the periodic timer does but it has made some progress since last posting. So I think it's time now for another early release.
= What's that? =
On the mainline kernel we have a feature (CONFIG_NO_HZ) that is able to turn off the periodic scheduler tick when the CPU has nothing to do, namely when it's running the idle task.
The scheduler tick handles many things like RCU and scheduler internal state, jiffies accouting, wall time accounting, load accounting, cputime accounting, timer wheel, posix cpu timers, etc...
However by the time we run idle and the CPU is going to sleep, none of these things are useful for the CPU. We can then shut it down.
The benefit of this is for energy saving purposes. We avoid to wake up the CPU needlessly with these useless interrupts.
What this patchset do is to extend that feature to non idle cases, implementing some new kind of "adaptive nohz". But the purpose is different and the implementation too.
= How does that work =
It tries to handle all the things that the timer tick usually handle but using different tricks. Sometimes we can't really afford to avoid the periodic tick, but sometimes we can and if we do, we need to take some special care.
- We can't shutdown the tick if we have more than one task running, due to the need for the tick for preemption. But I believe that one day we can avoid the periodic tick for that and rather anticipate when the scheduler really needs the tick.
- We can't shutdown the tick if RCU needs to complete a grace period from the current CPU, or if it has callbacks to handle.
- We can't shutdown the tick if we have a posix cpu timer queued. Similarly to the preemption case, we should be able to anticipate that with a precise timer and avoid a periodic check based on HZ.
- Restart the tick when more than one non-idle task are in the runqueue.
- We need to handle process accounting, RCU, rq clock, task tick, etc...
And that patchset for now only handles a part of the whole needs.
= What's the interface =
We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.
= Why do we need that? =
There are at least two potential users of this feature:
* High performance computing: To optimize the throughput, some workloads involve running one task per CPU that mostly run in userspace. These tasks don't want and don't need to suffer from the overhead of the timer interrupt. It consumes CPU time and it trashes the CPU cache.
* Real time: Minimizing timer interrupts means less interrupts and thus less critical sections that usually induce latency.
= What's missing? =
Many things like handling of perf events, irq work, sched clock tick, runqueue clock, sched_class::task_tick(), rq clock, cpu load, ...
The handling of cputimes is also incomplete as there are other places that use the utime/stime. Process time accounting is globally incomplete.
But anyway the thing is moving forward. An early posting was just very needed at that step.
For those who want to play:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
nohz/cpuset-v1
Frederic Weisbecker (32):
nohz: Drop useless call in tick_nohz_start_idle()
nohz: Drop ts->idle_active
nohz: Drop useless ts->inidle check before rearming the tick
nohz: Separate idle sleeping time accounting from nohz switching
nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
nohz: Move idle ticks stats tracking out of nohz handlers
nohz: Rename ts->idle_tick to ts->last_tick
nohz: Move nohz load balancer selection into idle logic
nohz: Move ts->idle_calls into strict idle logic
nohz: Move next idle expiring time record into idle logic area
cpuset: Set up interface for nohz flag
nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
nohz: Adaptive tick stop and restart on nohz cpuset nohz/cpuset: Don't turn off the tick if rcu needs it nohz/cpuset: Restart tick when switching to idle task nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
x86: New cpuset nohz irq vector
nohz/cpuset: Don't stop the tick if posix cpu timers are running nohz/cpuset: Restart tick when nohz flag is cleared on cpuset nohz/cpuset: Restart the tick if printk needs it
rcu: Restart the tick on non-responding adaptive nohz CPUs
rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU nohz/cpuset: Account user and system times in adaptive nohz mode nohz/cpuset: Handle kernel entry/exit to account cputime nohz/cpuset: New API to flush cputimes on nohz cpusets nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader nohz/cpuset: Flush cputimes on procfs stat file read nohz/cpuset: Flush cputimes for getrusage() and times() syscalls
x86: Syscall hooks for nohz cpusets
x86: Exception hooks for nohz cpusets
rcu: Switch to extended quiescent state in userspace from nohz cpuset nohz/cpuset: Disable under some configs
arch/Kconfig | 3 + arch/arm/kernel/process.c | 4 +- arch/avr32/kernel/process.c | 4 +- arch/blackfin/kernel/process.c | 4 +- arch/microblaze/kernel/process.c | 4 +- arch/mips/kernel/process.c | 4 +- arch/powerpc/kernel/idle.c | 4 +- arch/powerpc/platforms/iseries/setup.c | 8 +- arch/s390/kernel/process.c | 4 +- arch/sh/kernel/idle.c | 4 +- arch/sparc/kernel/process_64.c | 4 +- arch/tile/kernel/process.c | 4 +- arch/um/kernel/process.c | 4 +- arch/unicore32/kernel/process.c | 4 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/entry_arch.h | 3 + arch/x86/include/asm/hw_irq.h | 6 + arch/x86/include/asm/irq_vectors.h | 2 + arch/x86/include/asm/smp.h | 11 + arch/x86/include/asm/thread_info.h | 10 +- arch/x86/kernel/entry_64.S | 4 + arch/x86/kernel/irqinit.c | 4 + arch/x86/kernel/process_32.c | 4 +- arch/x86/kernel/process_64.c | 5 +- arch/x86/kernel/ptrace.c | 10 + arch/x86/kernel/smp.c | 26 ++ arch/x86/kernel/traps.c | 22 +- arch/x86/mm/fault.c | 13 +- fs/proc/array.c | 2 + include/linux/cpuset.h | 29 ++ include/linux/kernel_stat.h | 2 + include/linux/posix-timers.h | 1 + include/linux/rcupdate.h | 1 + include/linux/sched.h | 10 +- include/linux/tick.h | 50 +++- init/Kconfig | 8 + kernel/cpuset.c | 105 +++++++ kernel/exit.c | 2 + kernel/posix-cpu-timers.c | 12 + kernel/printk.c | 17 +- kernel/rcutree.c | 28 ++- kernel/sched.c | 132 +++++++++- kernel/softirq.c | 6 +- kernel/sys.c | 6 + kernel/time/tick-sched.c | 479 ++++++++++++++++++++++++-------- kernel/time/timer_list.c | 4 +- kernel/timer.c | 8 +- 47 files changed, 897 insertions(+), 185 deletions(-)
Signature · 1 line
1.7.5.4
PATCH RESEND 1/32 nohz: Drop useless call in tick_nohz_start_idle()
The call to update_ts_time_stats() there is useless. All we need is to save the idle entry_time.
--- kernel/time/tick-sched.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-)
kernel/time/tick-sched.c +0 -3
@@ -185,9 +185,6 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts)ktime_t now; now =ktime_get (); - -update_ts_time_stats (cpu, ts, now,NULL ); - ts->idle_entrytime = now; ts->idle_active =1 ;sched_clock_idle_sleep_event ();
Signature · 1 line
1.7.5.4
Re: [PATCH 01/32 RESEND] nohz: Drop useless call in tick_nohz_start_idle()
Quoted 4 lines · Frederic Weisbecker
The call to
update_ts_time_stats()there is useless. All we need is to save the idleentry_time.
Would have been clearer if you just said the call was a NOP. The whole second sentence distracts and confuses as its irrelevant to the change at hand.
If you want to expand you can explain that its a NOP because update_ts_time_stats() requires either ts->idle_active and/or @last_update_time and our callsite has neither.
Although this assumes its never called when ts->idle_active is already set, is this so (likely)? Do we want a WARN_ON_ONCE() testing that assumption?
Re: [PATCH 01/32 RESEND] nohz: Drop useless call in tick_nohz_start_idle()
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
The call to
update_ts_time_stats()there is useless. All we need is to save the idleentry_time.
Quoted 7 lines
Would have been clearer if you just said the call was a NOP. The whole second sentence distracts and confuses as its irrelevant to the change at hand.
If you want to expand you can explain that its a NOP because
update_ts_time_stats()requires eitherts->idle_activeand/or @last_update_time and our callsite has neither.
Right, will fix the changelog.
Quoted 4 lines
Although this assumes its never called when
ts->idle_activeis already set, is this so (likely)? Do we want aWARN_ON_ONCE()testing that assumption?
Not sure. Looking at the ondemand cpufreq governor, it calls get_cpu_idle_time_us() from an initcall.
PATCH RESEND 2/32 nohz: Drop ts->idle_active
In reply to Frederic Weisbeckerts->idle_active is used to know if we want to account the idle sleep time. But ts->inidle is enough to check that.
So drop that field and use inidle instead. This simplifies the code.
--- include/linux/tick.h | 1 - kernel/time/tick-sched.c | 14 ++++++-------- 2 files changed, 6 insertions(+), 9 deletions(-)
include/linux/tick.h +0 -1
@@ -56,7 +56,6 @@ struct tick_sched {unsigned long idle_jiffies;unsigned long idle_calls;unsigned long idle_sleeps; -int idle_active;ktime_t idle_entrytime;ktime_t idle_waketime;ktime_t idle_exittime;
kernel/time/tick-sched.c +6 -8
@@ -157,7 +157,7 @@ update_ts_time_stats(int cpu, struct tick_sched *ts, ktime_t now, u64 *last_upda {ktime_t delta; -if (ts->idle_active) { +if (ts->inidle) { delta =ktime_sub (now, ts->idle_entrytime); ts->idle_sleeptime =ktime_add (ts->idle_sleeptime, delta);if (nr_iowait_cpu (cpu) >0 )
@@ -175,7 +175,6 @@ static void tick_nohz_stop_idle(int cpu, ktime_t now)struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);update_ts_time_stats (cpu, ts, now,NULL ); - ts->idle_active =0 ;sched_clock_idle_wakeup_event (0 ); }
@@ -186,7 +185,6 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts) now =ktime_get (); ts->idle_entrytime = now; - ts->idle_active =1 ;sched_clock_idle_sleep_event ();return now; }
@@ -502,11 +500,11 @@ void tick_nohz_restart_sched_tick(void)ktime_t now;local_irq_disable (); -if (ts->idle_active || (ts->inidle && ts->tick_stopped)) - now =ktime_get (); -if (ts->idle_active) +if (ts->inidle) { + now =ktime_get ();tick_nohz_stop_idle (cpu, now); + }if (!ts->inidle || !ts->tick_stopped) { ts->inidle =0 ;
@@ -677,10 +675,10 @@ static inline void tick_check_nohz(int cpu)struct tick_sched *ts = &per_cpu (tick_cpu_sched, cpu);ktime_t now; -if (!ts->idle_active && !ts->tick_stopped) +if (!ts->inidle && !ts->tick_stopped)return ; now =ktime_get (); -if (ts->idle_active) +if (ts->inidle)tick_nohz_stop_idle (cpu, now);if (ts->tick_stopped) {tick_nohz_update_jiffies (now);
Signature · 1 line
1.7.5.4
Re: [PATCH 02/32 RESEND] nohz: Drop ts->idle_active
Quoted 3 lines · Frederic Weisbecker
ts->idle_activeis used to know if we want to account the idle sleep time. Butts->inidleis enough to check that.
While possibly true, its not immediately obvious and no hints are supplied. For example: tick_check_nohz() would disable ->idle_active.. where is this mirrored in the ->inidle state.
Also, tick_nohz_stop_sched_tick() has this comment:
/* * Set ts->inidle unconditionally. Even if the system did not * switch to NOHZ mode the cpu frequency governers rely on the * update of the idle time accounting in tick_nohz_start_idle(). */ ts->inidle = 1;
Which suggest the ->inidle state doesn't accurately reflect things.
This is all rather hairy code, such changes really want more in terms of explanation.
Re: [PATCH 02/32 RESEND] nohz: Drop ts->idle_active
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
ts->idle_activeis used to know if we want to account the idle sleep time. Butts->inidleis enough to check that.While possibly true, its not immediately obvious and no hints are supplied. For example:
tick_check_nohz()would disable ->idle_active.. where is this mirrored in the ->inidle state.
Hmm, you're right. By the time we call tick_check_nohz() (irq_enter()) and tick_nohz_stop_sched_tick() (irq_exit()) there may be a softirq and then another hard irq that could update the idle time spuriously.
So the mapping inidle - idle_active is wrong.
Let's drop that patch.
Quoted 16 lines
Also,
tick_nohz_stop_sched_tick()has this comment:/* * Set ts->inidle unconditionally. Even if the system did not * switch to NOHZ mode the cpu frequency governers rely on the * update of the idle time accounting in tick_nohz_start_idle(). */ ts->inidle = 1;Which suggest the ->inidle state doesn't accurately reflect things.
This is all rather hairy code, such changes really want more in terms of explanation.
PATCH RESEND 3/32 nohz: Drop useless ts->inidle check before rearming the tick
In reply to Frederic WeisbeckerWe only need to check if we have ts->stopped to ensure the tick was stopped and we want to re-enable it. Checking ts->inidle there is useless.
--- kernel/time/tick-sched.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)
kernel/time/tick-sched.c +2 -4
@@ -504,16 +504,14 @@ void tick_nohz_restart_sched_tick(void)if (ts->inidle) { now =ktime_get ();tick_nohz_stop_idle (cpu, now); + ts->inidle =0 ; } -if (!ts->inidle || !ts->tick_stopped) { - ts->inidle =0 ; +if (!ts->tick_stopped) {local_irq_enable ();return ; } - ts->inidle =0 ; -rcu_exit_nohz ();/* Update jiffies first */
Signature · 1 line
1.7.5.4
Re: [PATCH 03/32 RESEND] nohz: Drop useless ts->inidle check before rearming the tick
Quoted 3 lines · Frederic Weisbecker
We only need to check if we have
ts->stoppedto ensure the tick was stopped and we want to re-enable it. Checkingts->inidlethere is useless.
/me goes la-la-la-la...
It would so help poor little me who hasn't stared at this code in detail for the past several days and is thus horridly confused if you'd expand your reasoning somewhat.
Re: [PATCH 03/32 RESEND] nohz: Drop useless ts->inidle check before rearming the tick
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
We only need to check if we have
ts->stoppedto ensure the tick was stopped and we want to re-enable it. Checkingts->inidlethere is useless./me goes la-la-la-la...
It would so help poor little me who hasn't stared at this code in detail for the past several days and is thus horridly confused if you'd expand your reasoning somewhat.
Sorry, I'm no big fan of writing changelogs and sometimes the lack it's unfortunately visible :)$
It needs to be refactored due to the previous patch beeing broken. But the rationale, indeed missing here, is that if you have ts->stopped then you have ts->inidle. Once you entered tick_nohz_stop_sched_tick() you have ts->inidle set and only once you reached that step the tick can be stopped, so the following check:
if (!ts->inidle || !ts->tick_stopped)
can be summed up with:
if (!ts->tick_stopped) {PATCH 4/32 nohz: Separate idle sleeping time accounting from nohz switching
In reply to Frederic WeisbeckerTo prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.
This implies to implement some new API to call when we enter/exit idle.
--- arch/arm/kernel/process.c | 4 +- arch/avr32/kernel/process.c | 4 +- arch/blackfin/kernel/process.c | 4 +- arch/microblaze/kernel/process.c | 4 +- arch/mips/kernel/process.c | 4 +- arch/powerpc/kernel/idle.c | 4 +- arch/powerpc/platforms/iseries/setup.c | 8 +- arch/s390/kernel/process.c | 4 +- arch/sh/kernel/idle.c | 4 +- arch/sparc/kernel/process_64.c | 4 +- arch/tile/kernel/process.c | 4 +- arch/um/kernel/process.c | 4 +- arch/unicore32/kernel/process.c | 4 +- arch/x86/kernel/process_32.c | 4 +- arch/x86/kernel/process_64.c | 5 +- include/linux/tick.h | 10 ++- kernel/softirq.c | 2 +- kernel/time/tick-sched.c | 102 ++++++++++++++++++------------- 18 files changed, 98 insertions(+), 81 deletions(-)
arch/arm/kernel/process.c +2 -2
@@ -182,7 +182,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();leds_event (led_idle_start);while (!need_resched ()) {#ifdef CONFIG_HOTPLUG_CPU
@@ -208,7 +208,7 @@ void cpu_idle(void) } }leds_event (led_idle_end); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/avr32/kernel/process.c +2 -2
@@ -34,10 +34,10 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ())cpu_idle_sleep (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/blackfin/kernel/process.c +2 -2
@@ -88,10 +88,10 @@ void cpu_idle(void)#endif if (!idle) idle = default_idle; -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ())idle (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/microblaze/kernel/process.c +2 -2
@@ -103,10 +103,10 @@ void cpu_idle(void)if (!idle) idle = default_idle; -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ())idle (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();
arch/mips/kernel/process.c +2 -2
@@ -56,7 +56,7 @@ void __noreturn cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched () &&cpu_online (cpu)) {#ifdef CONFIG_MIPS_MT_SMTC extern void smtc_idle_loop_hook (void );
@@ -77,7 +77,7 @@ void __noreturn cpu_idle(void) system_state == SYSTEM_BOOTING))play_dead ();#endif -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/powerpc/kernel/idle.c +2 -2
@@ -56,7 +56,7 @@ void cpu_idle(void)set_thread_flag (TIF_POLLING_NRFLAG );while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched () && !cpu_should_die ()) {ppc64_runlatch_off ();
@@ -93,7 +93,7 @@ void cpu_idle(void)HMT_medium ();ppc64_runlatch_on (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();if (cpu_should_die ())cpu_die ();
arch/powerpc/platforms/iseries/setup.c +4 -4
@@ -562,7 +562,7 @@ static void yield_shared_processor(void)static void iseries_shared_idle (void ) {while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched () && !hvlpevent_is_pending ()) {local_irq_disable ();ppc64_runlatch_off ();
@@ -576,7 +576,7 @@ static void iseries_shared_idle(void) }ppc64_runlatch_on (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();if (hvlpevent_is_pending ())process_iSeries_events ();
@@ -592,7 +592,7 @@ static void iseries_dedicated_idle(void)set_thread_flag (TIF_POLLING_NRFLAG );while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();if (!need_resched ()) {while (!need_resched ()) {ppc64_runlatch_off ();
@@ -609,7 +609,7 @@ static void iseries_dedicated_idle(void) }ppc64_runlatch_on (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/s390/kernel/process.c +2 -2
@@ -90,10 +90,10 @@ static void default_idle(void)void cpu_idle (void ) {for (;;) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ())default_idle (); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/sh/kernel/idle.c +2 -2
@@ -88,7 +88,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ()) {check_pgt_cache ();
@@ -109,7 +109,7 @@ void cpu_idle(void)start_critical_timings (); } -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/sparc/kernel/process_64.c +2 -2
@@ -95,12 +95,12 @@ void cpu_idle(void)set_thread_flag (TIF_POLLING_NRFLAG );while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched () && !cpu_is_offline (cpu))sparc64_yield (cpu); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();
arch/tile/kernel/process.c +2 -2
@@ -85,7 +85,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ()) {if (cpu_is_offline (cpu))BUG ();/* no HOTPLUG_CPU */
@@ -105,7 +105,7 @@ void cpu_idle(void)local_irq_enable ();current_thread_info ()->status |=TS_POLLING ; } -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/um/kernel/process.c +2 -2
@@ -245,10 +245,10 @@ void default_idle(void)if (need_resched ())schedule (); -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle (); nsecs =disable_timer ();idle_sleep (nsecs); -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle (); } }
arch/unicore32/kernel/process.c +2 -2
@@ -55,7 +55,7 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ()) {local_irq_disable ();stop_critical_timings ();
@@ -63,7 +63,7 @@ void cpu_idle(void)local_irq_enable ();start_critical_timings (); } -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/x86/kernel/process_32.c +2 -2
@@ -97,7 +97,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ()) {check_pgt_cache ();
@@ -112,7 +112,7 @@ void cpu_idle(void)pm_idle ();start_critical_timings (); } -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
arch/x86/kernel/process_64.c +2 -3
@@ -120,7 +120,7 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { -tick_nohz_stop_sched_tick (1 ); +tick_nohz_enter_idle ();while (!need_resched ()) {rmb ();
@@ -144,8 +144,7 @@ void cpu_idle(void)loops can be woken up without interrupt. */ __exit_idle (); } - -tick_nohz_restart_sched_tick (); +tick_nohz_exit_idle ();preempt_enable_no_resched ();schedule ();preempt_disable ();
include/linux/tick.h +6 -4
@@ -120,14 +120,16 @@ static inline int tick_oneshot_mode_active(void) { return 0; }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */ # ifdef CONFIG_NO_HZ -extern void tick_nohz_stop_sched_tick (int inidle); -extern void tick_nohz_restart_sched_tick (void ); +extern void tick_nohz_enter_idle (void ); +extern void tick_nohz_exit_idle (void ); +extern void tick_nohz_irq_exit (void );extern ktime_t tick_nohz_get_sleep_length (void );extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time);extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time);# else -static inline void tick_nohz_stop_sched_tick (int inidle) { } -static inline void tick_nohz_restart_sched_tick (void ) { } +static inline void tick_nohz_enter_idle (void ) { } +static inline void tick_nohz_exit_idle (void ) { } +static inline ktime_t tick_nohz_get_sleep_length (void ) {ktime_t len = { .tv64 =NSEC_PER_SEC /HZ };
kernel/softirq.c +1 -1
@@ -343,7 +343,7 @@ void irq_exit(void)#ifdef CONFIG_NO_HZ /* Make sure that timer wheel updates are propagated */ if (idle_cpu (smp_processor_id ()) && !in_interrupt () && !need_resched ()) -tick_nohz_stop_sched_tick (0 ); +tick_nohz_irq_exit ();#endif preempt_enable_no_resched (); }
kernel/time/tick-sched.c +59 -43
161 lines not on this page · raw message
Signature · 1 line
1.7.5.4
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
Quoted 5 lines · Frederic Weisbecker
To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.
This implies to implement some new API to call when we enter/exit idle.
I mean, I really love brevity, but you seem to just not state all the important bits ;-)
So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the tick_nohz_stop_sched_tick() call on going idle will simply not do anything.
If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.
This implies to implement some new API to call when we enter/exit idle.
I mean, I really love brevity, but you seem to just not state all the important bits ;-)
So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the
tick_nohz_stop_sched_tick()call on going idle will simply not do anything.
Quoted 4 lines
If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.
That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...
I could do these things conditionally using a some idle_cpu() checks but the end result would not be very proper.
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
Quoted 21 lines · Frederic Weisbecker
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.
This implies to implement some new API to call when we enter/exit idle.
I mean, I really love brevity, but you seem to just not state all the important bits ;-)
So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the
tick_nohz_stop_sched_tick()call on going idle will simply not do anything.Quoted 4 lines
If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.
Quoted 6 lines
That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...
I could do these things conditionally using a some
idle_cpu()checks but the end result would not be very proper.
Right, but you didn't explain any of that in the changelog. So the reasoning is that because tick_nohz_stop_sched_tick() does more than just stop the tick, and this extra work needs to be isolated to just the idle case, therefore we need hooks specific for the idle loop.
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frederic Weisbecker
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.
This implies to implement some new API to call when we enter/exit idle.
I mean, I really love brevity, but you seem to just not state all the important bits ;-)
So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the
tick_nohz_stop_sched_tick()call on going idle will simply not do anything.Quoted 4 lines
If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.
That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...
Quoted 7 lines
Quoted 2 lines
I could do these things conditionally using a some
idle_cpu()checks but the end result would not be very proper.Right, but you didn't explain any of that in the changelog. So the reasoning is that because
tick_nohz_stop_sched_tick()does more than just stop the tick, and this extra work needs to be isolated to just the idle case, therefore we need hooks specific for the idle loop.
Right, I'll update the changelog.
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
In reply to Frederic Weisbeckerinclude/linux/tick.h static inline int tick_oneshot_mode_active(void) { return 0; } · line 125
Quoted 2 lines · Frederic Weisbecker
include/linux/tick.h +2
-0+extern void tick_nohz_exit_idle (void ); +extern void tick_nohz_irq_exit (void );
For consistencies sake, pick either tick_nohz_exit_*() or tick_nohz_*_exit() but don't go mix and match (I prefer the former).
Also, there isn't a matching irq_enter() callback..
Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching
Quoted 6 lines · Peter Zijlstra
include/linux/tick.h static inline int tick_oneshot_mode_active(void) { return 0; } · line 125
Quoted 2 lines · Frederic Weisbecker
include/linux/tick.h +2
-0+extern void tick_nohz_exit_idle (void ); +extern void tick_nohz_irq_exit (void );For consistencies sake, pick either tick_nohz_exit_*() or tick_nohz_*_exit() but don't go mix and match (I prefer the former).
Agreed, I will.
Quoted 1 line
Also, there isn't a matching
irq_enter()callback..
Yeah, probably tick_check_nohz() should be renamed accordingly.
PATCH 5/32 nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
In reply to Frederic WeisbeckerTo prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
--- kernel/time/tick-sched.c | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-)
kernel/time/tick-sched.c +7 -4
@@ -385,7 +385,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now) ts->idle_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; ts->idle_jiffies = last_jiffies; -rcu_enter_nohz (); } ts->idle_sleeps++;
@@ -429,9 +428,13 @@ out:static void __tick_nohz_enter_idle (struct tick_sched *ts,int cpu) {ktime_t now; +int was_stopped = ts->tick_stopped; now =tick_nohz_start_idle (cpu, ts);tick_nohz_stop_sched_tick (now); + +if (!was_stopped && ts->tick_stopped) +rcu_enter_nohz (); }void tick_nohz_enter_idle (void )
@@ -499,8 +502,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)unsigned long ticks;#endif -rcu_exit_nohz (); -/* Update jiffies first */ select_nohz_load_balancer (0 );tick_do_update_jiffies64 (now);
@@ -547,8 +548,10 @@ void tick_nohz_exit_idle(void)tick_nohz_stop_idle (smp_processor_id (), now); ts->inidle =0 ; -if (ts->tick_stopped) +if (ts->tick_stopped) { +rcu_exit_nohz ();tick_nohz_restart_sched_tick (now, ts); + }local_irq_enable (); }
Signature · 1 line
1.7.5.4
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 5 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
But we can stop the tick from the kernel, not just userspace.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
But we can stop the tick from the kernel, not just userspace.
Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?
Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
Quoted 8 lines
Quoted 1 line
But we can stop the tick from the kernel, not just userspace.
Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?
Yeah. Either the interrupt returns to userspace and then we call rcu_enter_nohz() or we return to kernel space and then a further use of rcu will restart the tick.
Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick. But we need it as long as there is an RCU callback enqueued on some CPU.
Quoted 2 lines
Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?
This is why we have rcu_needs_cpu() and rcu_pending() checks before stopping the tick.
rcu_needs_cpu() checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.
rcu_pending() is there to know if another CPU started a grace period so we need the tick to complete it.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 23 lines · Frederic Weisbecker
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
Quoted 8 lines
Quoted 1 line
But we can stop the tick from the kernel, not just userspace.
Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?
Quoted 6 lines
Yeah. Either the interrupt returns to userspace and then we call
rcu_enter_nohz()or we return to kernel space and then a further use of rcu will restart the tick.Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick.
But but but, then how is it going to stop a grace period from happening? The grace period state is per-cpu and the whole state machine is tick driven.
Now some of the new RCU things go kick cpus with IPIs to push grace periods along, but I would expect you don't want that to happen either, the whole purpose here is to leave a cpu alone, unperturbed.
That means it has to be in an extended grace period when we stop the tick.
Quoted 2 lines
But we need it as long as there is an RCU callback enqueued on some CPU.
Well, no, only if there's one enqueued on this cpu because then we can't enter the extended grace period.
Quoted 11 lines
Quoted 2 lines
Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?
This is why we have
rcu_needs_cpu()andrcu_pending()checks before stopping the tick.
rcu_needs_cpu()checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.
rcu_pending()is there to know if another CPU started a grace period so we need the tick to complete it.
Hence the extended grace period, so we don't need to complete grace periods.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 28 lines · Peter Zijlstra
Quoted 27 lines · Frederic Weisbecker
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.
So we make the dyntick mode possible without always involving rcu extended quiescent state.
Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?
Quoted 8 lines
Quoted 1 line
But we can stop the tick from the kernel, not just userspace.
Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?
Yeah. Either the interrupt returns to userspace and then we call
rcu_enter_nohz()or we return to kernel space and then a further use of rcu will restart the tick.
Quoted 6 lines
Quoted 2 lines
Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick.
But but but, then how is it going to stop a grace period from happening? The grace period state is per-cpu and the whole state machine is tick driven.
But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.
OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.
Quoted 3 lines
Now some of the new RCU things go kick cpus with IPIs to push grace periods along, but I would expect you don't want that to happen either, the whole purpose here is to leave a cpu alone, unperturbed.
Sure we want the CPU to be unperturbed but not if that sacrifies correctness. As long as we run in the kernel we want to receive such IPIs to restart the tick as needed.
Quoted 2 lines
That means it has to be in an extended grace period when we stop the tick.
You mean extended quiescent state?
As a summary here is what we do:
- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.
- if we are in userspace we can run in extended quiescent state.
Quoted 6 lines
Quoted 3 lines
But we need it as long as there is an RCU callback enqueued on some CPU.
Well, no, only if there's one enqueued on this cpu because then we can't enter the extended grace period.
True if we are in userspace.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?
This is why we have
rcu_needs_cpu()andrcu_pending()checks before stopping the tick.
Quoted 8 lines
Quoted 5 lines
rcu_needs_cpu()checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.
rcu_pending()is there to know if another CPU started a grace period so we need the tick to complete it.Hence the extended grace period, so we don't need to complete grace periods.
I hope the above explanations made things more clear.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 3 lines · Frederic Weisbecker
But rcu read side critical sections (preemption disabled,
rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.
Note that with PREEMPT_RCU only the rcu_read_lock() is actually an rcu read side critical section, non of the others should be used as such. Relying on preempt_disable(), local_bh_disable() and similar is broken as per a long while ago.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
But rcu read side critical sections (preemption disabled,
rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.Note that with
PREEMPT_RCUonly thercu_read_lock()is actually an rcu read side critical section, non of the others should be used as such. Relying onpreempt_disable(),local_bh_disable()and similar is broken as per a long while ago.
Sure yeah.
My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
But rcu read side critical sections (preemption disabled,
rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.Note that with
PREEMPT_RCUonly thercu_read_lock()is actually an rcu read side critical section, non of the others should be used as such. Relying onpreempt_disable(),local_bh_disable()and similar is broken as per a long while ago.
Quoted 7 lines
Sure yeah.
My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.
But you wake up the tick for any callback - systemwide - right? That's a massive fail.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
But rcu read side critical sections (preemption disabled,
rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.Note that with
PREEMPT_RCUonly thercu_read_lock()is actually an rcu read side critical section, non of the others should be used as such. Relying onpreempt_disable(),local_bh_disable()and similar is broken as per a long while ago.Sure yeah.
Quoted 8 lines
Quoted 5 lines
My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.
But you wake up the tick for any callback - systemwide - right? That's a massive fail.
Until we find a better solution yeah.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
In reply to Frederic WeisbeckerQuoted 4 lines · Frederic Weisbecker
OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.
Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.
Requiring it to participate in the grace period state machine is a fail, plain and simple.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.
Quoted 7 lines
Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.
Requiring it to participate in the grace period state machine is a fail, plain and simple.
We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.
Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.
We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:
- Having explicit hooks in preempt_disable() and local_irq_restore() to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.
- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 15 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.
Quoted 7 lines
Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.
Requiring it to participate in the grace period state machine is a fail, plain and simple.
Quoted 8 lines
We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.
Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.
Well the simple fact is that rcu, when considered system-wide, is pretty much always busy, voiding any and all benefit you might want to gain.
Quoted 5 lines
We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:
Quoted 8 lines
- Having explicit hooks in
preempt_disable()andlocal_irq_restore()to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.
Right, so what we can do is keep a per-cpu context switch counter (I'm sure we have one someplace and we already have the rcu_note_context_switch() callback in case we need another) and have another cpu (outside of our extended nohz domain) drive our state machine.
But I'm sure Paul can say more sensible things than me here.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.
Quoted 7 lines
Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.
Requiring it to participate in the grace period state machine is a fail, plain and simple.
We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.
Quoted 5 lines
Quoted 2 lines
Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.
Well the simple fact is that rcu, when considered system-wide, is pretty much always busy, voiding any and all benefit you might want to gain.
With my testcase, a stupid userspace loop on a single CPU among 4, I actually see only few RCU activity. Especially as any other CPU is pretty much idle. There are some cases where it's not so pointless.
Quoted 14 lines
Quoted 5 lines
We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:
Quoted 8 lines
- Having explicit hooks in
preempt_disable()andlocal_irq_restore()to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.
Quoted 7 lines
Right, so what we can do is keep a per-cpu context switch counter (I'm sure we have one someplace and we already have the
rcu_note_context_switch()callback in case we need another) and have another cpu (outside of our extended nohz domain) drive our state machine.But I'm sure Paul can say more sensible things than me here.
Yeah I hope we can find some solution to minimize these IPIs.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
In reply to Frederic WeisbeckerQuoted 4 lines · Frederic Weisbecker
Quoted 2 lines
That means it has to be in an extended grace period when we stop the tick.
You mean extended quiescent state?
Yeah that :-)
Quoted 11 lines
As a summary here is what we do:
- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.
Quoted 1 line
- if we are in userspace we can run in extended quiescent state.
But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
That means it has to be in an extended grace period when we stop the tick.
You mean extended quiescent state?
Yeah that :-)
Quoted 11 lines
As a summary here is what we do:
- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.
Quoted 5 lines
Quoted 1 line
- if we are in userspace we can run in extended quiescent state.
But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.
No, we have a specific stacking in the irq:
rcu_irq_enter() disable tick... if (user) rcu_enter_nohz(); rcu_irq_exit() <-- extended quiescent state entry effective only there
And by the time we call rcu_irq_exit() and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal with do_notify_resume() which I have yet to handle).
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 36 lines · Frederic Weisbecker
Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
That means it has to be in an extended grace period when we stop the tick.
You mean extended quiescent state?
Yeah that :-)
Quoted 11 lines
As a summary here is what we do:
- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.
Quoted 5 lines
Quoted 1 line
- if we are in userspace we can run in extended quiescent state.
But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.
No, we have a specific stacking in the irq:
rcu_irq_enter() disable tick... if (user) rcu_enter_nohz(); rcu_irq_exit() <-- extended quiescent state entry effective only there
Quoted 3 lines
And by the time we call
rcu_irq_exit()and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal withdo_notify_resume()which I have yet to handle).
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Frederic Weisbecker
Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines
That means it has to be in an extended grace period when we stop the tick.
You mean extended quiescent state?
Yeah that :-)
Quoted 11 lines
As a summary here is what we do:
- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.
Quoted 5 lines
Quoted 1 line
- if we are in userspace we can run in extended quiescent state.
But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.
No, we have a specific stacking in the irq:
rcu_irq_enter() disable tick... if (user) rcu_enter_nohz(); rcu_irq_exit() <-- extended quiescent state entry effective only there
Quoted 8 lines
Quoted 3 lines
And by the time we call
rcu_irq_exit()and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal withdo_notify_resume()which I have yet to handle).See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 6 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 4 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Quoted 5 lines
Quoted 2 lines
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
At least if we are in userspace it does as long as we have no local rcu callbacks to handle. We have the rcu_pending() check but we can remove it for userspace.
Now for the kernel space I still think it's worth, for example when all other CPUs are idle, or when they don't queue that much RCU callbacks. In theory if we have a grace period to complete 100 times per second but HZ=1000 then we are going to avoid a lot of timer interrupts.
If we can't remove the timer, we can at least minimize it.
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
In reply to Frederic WeisbeckerQuoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 4 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Quoted 5 lines
Quoted 2 lines
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
So how about something like:
Assuming we are in rcu_nohz state; on kernel enter we leave rcu_nohz but don't start the tick, instead we assign another cpu to run our state machine.
On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter rcu_nohz state.
If between that time we did restart the tick, we take back our rcu state and skip the donate and rcu_nohz enter on kernel exit.
I really should go read all those docs Paul send me to see how insane the above is.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 6 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Quoted 8 lines
Quoted 2 lines
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
So how about something like:
Assuming we are in
rcu_nohzstate; on kernel enter we leavercu_nohzbut don't start the tick, instead we assign another cpu to run our state machine.
The nohz CPU still has to notice its own quiescent states. Now it could be an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.
Quoted 3 lines
On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter
rcu_nohzstate.
That's already what does rcu_enter_nohz().
Quoted 2 lines
If between that time we did restart the tick, we take back our rcu state and skip the donate and
rcu_nohzenter on kernel exit.
That's also what is done in this patchset. As soon as we re-enter the kernel or the tick had to be restarted before we re-enter the kernel, we call rcu_exit_nohz() that pulls back the CPU to the whole RCU machinery.
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 26 lines · Frederic Weisbecker
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 6 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
Quoted 7 lines
Quoted 5 lines
So how about something like:
Assuming we are in
rcu_nohzstate; on kernel enter we leavercu_nohzbut don't start the tick, instead we assign another cpu to run our state machine.The nohz CPU still has to notice its own quiescent states.
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
Quoted 4 lines
Now it could be
an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.
No, seriously, RCU should not, ever, need to re-enable the tick. Imagine a HPC workload where the system cores are also responsible for all IO and all the adaptive-nohz cores are simply crunching numbers. In that scenario you'll have a very high rcu usage because the system cores are all very busy arranging work for the computation cores.
Quoted 5 lines
Quoted 3 lines
On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter
rcu_nohzstate.That's already what does
rcu_enter_nohz().
Almost but not quite, it doesn't donate the callbacks for example (something it does do on hotplug -- and therefore any assumption the callback will in fact run on the cpu you submit it on is already broken).
Quoted 4 lines
Quoted 2 lines
If between that time we did restart the tick, we take back our rcu state and skip the donate and
rcu_nohzenter on kernel exit.That's also what is done in this patchset.
Its not, since you don't hand of the grace period detectoring you don't take it back now do you..
Quoted 2 lines
As soon as we re-enter the kernel
or the tick had to be restarted before we re-enter the kernel,
Another impossibility, you can only restart the tick from the kernel.
Quoted 2 lines
we call
rcu_exit_nohz()that pulls back the CPU to the whole RCU machinery.
But you then also start the tick again..
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Frederic Weisbecker
Quoted 27 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines
See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?
This: https://lkml.org/lkml/2011/6/23/744
It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)
OK, good ;-)
Quoted 4 lines
Quoted 3 lines
I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.
Quoted 5 lines
Quoted 2 lines
Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.
Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.
So how about something like:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Assuming we are in
rcu_nohzstate; on kernel enter we leavercu_nohzbut don't start the tick, instead we assign another cpu to run our state machine.The nohz CPU still has to notice its own quiescent states.
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.
Would be nice to hear about Paul's opinion on that.
Quoted 10 lines
Quoted 4 lines
Now it could be
an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.No, seriously, RCU should not, ever, need to re-enable the tick. Imagine a HPC workload where the system cores are also responsible for all IO and all the adaptive-nohz cores are simply crunching numbers. In that scenario you'll have a very high rcu usage because the system cores are all very busy arranging work for the computation cores.
Of course if we find a better way than having to restart this tick I'm all for doing that way.
That said if it requires some significant changes this should be done outside this patchset, as an optimization afterward may be, the patchset is already big while still missing very important features for now that the timer handles.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter
rcu_nohzstate.That's already what does
rcu_enter_nohz().Almost but not quite, it doesn't donate the callbacks for example (something it does do on hotplug -- and therefore any assumption the callback will in fact run on the cpu you submit it on is already broken).
Good to know, so that would avoid to restart the tick on call_rcu() ? Sounds good but again I think this should be done later.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
If between that time we did restart the tick, we take back our rcu state and skip the donate and
rcu_nohzenter on kernel exit.That's also what is done in this patchset.
Its not, since you don't hand of the grace period detectoring you don't take it back now do you..
So you are talking about grace period started locally due to local callbacks enqueued, right?
Quoted 4 lines
Quoted 2 lines
As soon as we re-enter the kernel
or the tick had to be restarted before we re-enter the kernel,Another impossibility, you can only restart the tick from the kernel.
Ok I meant it can be restarted from an interrupt interrupting userspace. I was talking about kernel enter/exit considering the new hooks brought (syscalls and exceptions).
Quoted 4 lines
Quoted 2 lines
we call
rcu_exit_nohz()that pulls back the CPU to the whole RCU machinery.But you then also start the tick again..
When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.
But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.
Quoted 4 lines
Quoted 1 line
But you then also start the tick again..
When we enter kernel? (minus interrupts)
No we only callrcu_exit_nohz().
So thinking more about all this:
rcu_exit_nohz() will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).
Then when we leave the kernel (or go idle) we re-enter rcu_nohz state, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.
I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.
So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.
Now I forgot where all you send IPIs from, and I'll go look at these patches once more.
As for call_rcu() for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.
But yeah, input from Paul would be nice...
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.
But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.
Ah, but it really is written, among other things, by the OOM killer. ;-)
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
But you then also start the tick again..
When we enter kernel? (minus interrupts)
No we only callrcu_exit_nohz().
Quoted 7 lines
So thinking more about all this:
rcu_exit_nohz()will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).
Lacking a tick, the CPU also fails to respond to state updates from other CPUs.
Quoted 4 lines
Then when we leave the kernel (or go idle) we re-enter
rcu_nohzstate, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.
This is true.
Quoted 2 lines
I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.
All the ways I know of to make PREEMPT_RCU live without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.
Or are you saying that PREEMPT_RCU does not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.
Quoted 6 lines
So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.
Now I forgot where all you send IPIs from, and I'll go look at these patches once more.
Quoted 6 lines
As for
call_rcu()for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.But yeah, input from Paul would be nice...
In the call_rcu() case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)
The reason for doing this is that people are enabling CONFIG_RCU_FAST_NO_HZ on systems that have no business enabling it. Bad choice of names on my part.
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
Quoted 2 lines
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.
But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.
Ah, but it really is written, among other things, by the OOM killer. ;-)
Well there is that of course :-) But I think the below argument relies on what we already have without requiring more.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
But you then also start the tick again..
When we enter kernel? (minus interrupts)
No we only callrcu_exit_nohz().So thinking more about all this:
Quoted 8 lines
Quoted 5 lines
rcu_exit_nohz()will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).Lacking a tick, the CPU also fails to respond to state updates from other CPUs.
I'm sure I'll have to go re-read your documents, but does that matter? If we would have had a tick we still couldn't have progressed since we wouldn't have scheduled etc.. so we would hold up GP completion any way.
Quoted 6 lines
Quoted 4 lines
Then when we leave the kernel (or go idle) we re-enter
rcu_nohzstate, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.This is true.
So suppose all other CPUs completed the GP and our CPU is the one holding things up, now I don't see rcu_enter_nohz() doing anything much at all, who is responsible for GP completion?
Quoted 9 lines
Quoted 2 lines
I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.
All the ways I know of to make
PREEMPT_RCUlive without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.
Quoted 3 lines
Or are you saying that
PREEMPT_RCUdoes not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.
I'm not saying anything much about PREEMPT_RCU, I voiced an ill-considered suspicion :-)
So in the nr_running=[0,1] case we're in rcu_nohz state when idle or when in userspace. The only interesting part is being in kernel space where we cannot be in rcu_nohz state because we might actually use pointers and thus have to stop callbacks from destroying state etc..
The only PREEMPT_RCU implementation I can recall is the counting one, and that one does indeed want a tick, because even in kernel space it could move things forward if the 'old' index counter reaches 0.
Now we could possibly add magic to rcu_read_unlock_special() to restart the tick in that case.
Now clearly all that might be non-applicable to the current one, will have to wrap my head around the current PREEMPT_RCU implementation some more.
Quoted 11 lines
Quoted 11 lines
So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.
Now I forgot where all you send IPIs from, and I'll go look at these patches once more.
As for
call_rcu()for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.
Quoted 6 lines
Quoted 1 line
But yeah, input from Paul would be nice...
In the
call_rcu()case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)
Hand your callback to someone else is one solution, but I'm not overly worried about re-starting the tick if we do call_rcu().
Quoted 3 lines
The reason for doing this is that people are enabling
CONFIG_RCU_FAST_NO_HZon systems that have no business enabling it. Bad choice of names on my part.
hehe :-)
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Paul E. McKenney
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines
Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.
If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.
Ah, but it really is written, among other things, by the OOM killer. ;-)
Well there is that of course :-) But I think the below argument relies on what we already have without requiring more.
Almost. ;-)
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 4 lines
Quoted 1 line
But you then also start the tick again..
When we enter kernel? (minus interrupts)
No we only callrcu_exit_nohz().So thinking more about all this:
rcu_exit_nohz()will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).
Quoted 6 lines
Quoted 2 lines
Lacking a tick, the CPU also fails to respond to state updates from other CPUs.
I'm sure I'll have to go re-read your documents, but does that matter? If we would have had a tick we still couldn't have progressed since we wouldn't have scheduled etc.. so we would hold up GP completion any way.
There are two phases to quiescent-state detection: (1) actually detecting the quiescent state and (2) reporting detection to the RCU core. If you turn off the tick at an inopportune time, you can have CPUs that have detected the tick, but not yet reported it.
You asked the follow-up question below, so please see below.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Then when we leave the kernel (or go idle) we re-enter
rcu_nohzstate, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.This is true.
So suppose all other CPUs completed the GP and our CPU is the one holding things up, now I don't see
rcu_enter_nohz()doing anything much at all, who is responsible for GP completion?
Any CPU that has RCU callbacks queued that are waiting for the current or some susequent grace period to complete are responsible for pushing the current grace period forward, hence the checks in the non-RCU_FAST_NO_HZ variants of rcu_needs_cpu(). This is why CPUs with callbacks that are not yet done cannot currently disable the tick -- because we need at least one CPU to detect the fact that dyntick-idle CPUs are in fact in extended quiescent states.
Again, I believe that I can do better, hence the in-progress rewrite of RCU_FAST_NO_HZ. Either that or get most people to stop using it, and given its name, getting people to stop using it is likely an exercise in futility. "But it is FAST, and that is good, and it involve NO_HZ, which saves energy, which is also good. Therefore, I will enable it -everywhere-!!!"
Sigh. It will be much easier to rewrite it, ugly corner cases nonwithstanding. :-(
Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.
All the ways I know of to make
PREEMPT_RCUlive without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.
Quoted 6 lines
Quoted 3 lines
Or are you saying that
PREEMPT_RCUdoes not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.I'm not saying anything much about
PREEMPT_RCU, I voiced an ill-considered suspicion :-)
;-)
Quoted 4 lines
So in the nr_running=[0,1] case we're in
rcu_nohzstate when idle or when in userspace. The only interesting part is being in kernel space where we cannot be inrcu_nohzstate because we might actually use pointers and thus have to stop callbacks from destroying state etc..
Yep!
Quoted 6 lines
The only
PREEMPT_RCUimplementation I can recall is the counting one, and that one does indeed want a tick, because even in kernel space it could move things forward if the 'old' index counter reaches 0.Now we could possibly add magic to
rcu_read_unlock_special()to restart the tick in that case.
Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...
I don't currently have an opinion about the advisability of restarting the tick from hardIRQ handlers, but I do feel the need to point out the the possibility.
Quoted 3 lines
Now clearly all that might be non-applicable to the current one, will have to wrap my head around the current
PREEMPT_RCUimplementation some more.
Indeed, the documentation is going much more slowly than I would like...
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.
Now I forgot where all you send IPIs from, and I'll go look at these patches once more.
As for
call_rcu()for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.But yeah, input from Paul would be nice...
Quoted 7 lines
Quoted 4 lines
In the
call_rcu()case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)Hand your callback to someone else is one solution, but I'm not overly worried about re-starting the tick if we do
call_rcu().
As long as the handoff doesn't turn into a battery-killing game of RCU-callback hot potato. And I am seriously concerned about this possibility.
I will be updating the Dyntick-Idle doc to cover the new RCU_FAST_NO_HZ algorithm if and when I get it into human-readable form.
Quoted 5 lines
Quoted 3 lines
The reason for doing this is that people are enabling
CONFIG_RCU_FAST_NO_HZon systems that have no business enabling it. Bad choice of names on my part.hehe :-)
Sigh! Scanning dyntick-idle state on a system with 256 CPUs. What could possibly go wrong? :-/
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines
Now we could possibly add magic to
rcu_read_unlock_special()to restart the tick in that case.Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...
Uhm.. good point ;-) yes that will be somewhat detrimental to the overall OS health if we attempt such a thing.
Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines
Now we could possibly add magic to
rcu_read_unlock_special()to restart the tick in that case.Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...
Uhm.. good point ;-) yes that will be somewhat detrimental to the overall OS health if we attempt such a thing.
But if you don't mind, I would very much like to use your question as a Quick Quiz. It should be a good one. ;-)
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
Quoted 3 lines · Paul E. McKenney
But if you don't mind, I would very much like to use your question as a Quick Quiz. It should be a good one. ;-)
No problem.. :-)
PATCH 6/32 nohz: Move idle ticks stats tracking out of nohz handlers
In reply to Frederic WeisbeckerIdle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.
--- kernel/time/tick-sched.c | 34 +++++++++++++++++++--------------- 1 files changed, 19 insertions(+), 15 deletions(-)
kernel/time/tick-sched.c +19 -15
@@ -384,7 +384,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now) ts->idle_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; - ts->idle_jiffies = last_jiffies; } ts->idle_sleeps++;
@@ -433,8 +432,10 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu) now =tick_nohz_start_idle (cpu, ts);tick_nohz_stop_sched_tick (now); -if (!was_stopped && ts->tick_stopped) +if (!was_stopped && ts->tick_stopped) { + ts->idle_jiffies = ts->last_jiffies;rcu_enter_nohz (); + } }void tick_nohz_enter_idle (void )
@@ -498,16 +499,26 @@ static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)static void tick_nohz_restart_sched_tick (ktime_t now,struct tick_sched *ts) {int cpu =smp_processor_id (); -#ifndef CONFIG_VIRT_CPU_ACCOUNTING -unsigned long ticks; -#endif /* Update jiffies first */ select_nohz_load_balancer (0 );tick_do_update_jiffies64 (now);cpumask_clear_cpu (cpu, nohz_cpu_mask); +touch_softlockup_watchdog (); +/* +* Cancel the scheduled timer and restore the tick +*/ + ts->tick_stopped =0 ; + ts->idle_exittime = now; + +tick_nohz_restart (ts, now); +} + +static void tick_nohz_account_idle_ticks (struct tick_sched *ts) +{#ifndef CONFIG_VIRT_CPU_ACCOUNTING +unsigned long ticks;/* * We stopped the tick in idle. Update process times would miss the * time we slept as update_process_times does only a 1 tick
@@ -520,15 +531,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)if (ticks && ticks <LONG_MAX )account_idle_ticks (ticks);#endif - -touch_softlockup_watchdog (); -/* -* Cancel the scheduled timer and restore the tick -*/ - ts->tick_stopped =0 ; - ts->idle_exittime = now; - -tick_nohz_restart (ts, now); }void tick_nohz_exit_idle (void )
@@ -551,6 +553,7 @@ void tick_nohz_exit_idle(void)if (ts->tick_stopped) {rcu_exit_nohz ();tick_nohz_restart_sched_tick (now, ts); +tick_nohz_account_idle_ticks (ts); }local_irq_enable ();
@@ -766,7 +769,8 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)*/ if (ts->tick_stopped) {touch_softlockup_watchdog (); - ts->idle_jiffies++; +if (idle_cpu (cpu)) + ts->idle_jiffies++; }update_process_times (user_mode (regs));profile_tick (CPU_PROFILING );
Signature · 1 line
1.7.5.4
Re: [PATCH 06/32] nohz: Move idle ticks stats tracking out of nohz handlers
Quoted 3 lines · Frederic Weisbecker
Idle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.
Are you trying to say:
Currently idle time tracking is part of the nohz state tracking, separate this so that we can disable the tick while we're non-idle?
If so, how does idle time tracking work on a !NOHZ kernel, surely such things can be idle too..
Re: [PATCH 06/32] nohz: Move idle ticks stats tracking out of nohz handlers
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
Idle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.
Are you trying to say:
Currently idle time tracking is part of the nohz state tracking, separate this so that we can disable the tick while we're non-idle?
Right.
Quoted 3 lines
If so, how does idle time tracking work on a !NOHZ kernel, surely such things can be idle too..
In that case the tick calls account_process_tick() which delegates to account_idle_time() if we are idle on tick.
That's also used in NOHZ kernels in fact.
In every tick we call account_process_tick(), but when the tick is stopped we save its value in ts->idle_jiffies and when it is restarted we account jiffies - ts->idle_jiffies.
PATCH 7/32 nohz: Rename ts->idle_tick to ts->last_tick
In reply to Frederic WeisbeckerNow that idle and nohz logic are getting split, idle_tick becomes a misnomer when it takes a field name to save the last tick before switching to nohz.
Call it last_tick instead. This changes a bit the timer list stat export so we need to increase its version.
--- include/linux/tick.h | 8 ++++---- kernel/time/tick-sched.c | 4 ++-- kernel/time/timer_list.c | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/tick.h +4 -4
@@ -30,10 +30,10 @@ enum tick_nohz_mode {* struct tick_sched - sched tick emulation and no idle tick control/stats * @sched_timer: hrtimer to schedule the periodic tick in high * resolution mode -* @idle_tick: Store the last idle tick expiry time when the tick -* timer is modified for idle sleeps. This is necessary +* @last_tick: Store the last tick expiry time when the tick +* timer is modified for nohz sleeps. This is necessary * to resume the tick timer operation in the timeline -* when the CPU returns from idle +* when the CPU returns from nohz sleep. * @tick_stopped: Indicator that the idle tick has been stopped * @idle_jiffies: jiffies at the entry to idle for idle time accounting * @idle_calls: Total number of idle calls
@@ -50,7 +50,7 @@ struct tick_sched {struct hrtimer sched_timer;unsigned long check_clocks;enum tick_nohz_mode nohz_mode; -ktime_t idle_tick; +ktime_t last_tick;int inidle;int tick_stopped;unsigned long idle_jiffies;
kernel/time/tick-sched.c +2 -2
@@ -382,7 +382,7 @@ static void tick_nohz_stop_sched_tick(ktime_t now)if (!ts->tick_stopped) {select_nohz_load_balancer (1 ); - ts->idle_tick =hrtimer_get_expires (&ts->sched_timer); + ts->last_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; }
@@ -468,7 +468,7 @@ ktime_t tick_nohz_get_sleep_length(void)static void tick_nohz_restart (struct tick_sched *ts,ktime_t now) {hrtimer_cancel (&ts->sched_timer); -hrtimer_set_expires (&ts->sched_timer, ts->idle_tick); +hrtimer_set_expires (&ts->sched_timer, ts->last_tick);while (1 ) {/* Forward the time to expire in the future */
kernel/time/timer_list.c +2 -2
@@ -167,7 +167,7 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now) {struct tick_sched *ts =tick_get_tick_sched (cpu);P (nohz_mode); -P_ns (idle_tick); +P_ns (last_tick);P (tick_stopped);P (idle_jiffies);P (idle_calls);
@@ -259,7 +259,7 @@ static int timer_list_show(struct seq_file *m, void *v)u64 now =ktime_to_ns (ktime_get ());int cpu; -SEQ_printf (m,"Timer List Version: v0.6\n" ); +SEQ_printf (m,"Timer List Version: v0.7\n" );SEQ_printf (m,"HRTIMER_MAX_CLOCK_BASES: %d\n" ,HRTIMER_MAX_CLOCK_BASES );SEQ_printf (m,"now at %Ld nsecs\n" , (unsigned long long )now);
Signature · 1 line
1.7.5.4
PATCH 8/32 nohz: Move nohz load balancer selection into idle logic
In reply to Frederic WeisbeckerWe want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.
--- kernel/time/tick-sched.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-)
kernel/time/tick-sched.c +2 -3
@@ -380,8 +380,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now)* the scheduler tick in nohz_restart_sched_tick. */ if (!ts->tick_stopped) { -select_nohz_load_balancer (1 ); - ts->last_tick =hrtimer_get_expires (&ts->sched_timer); ts->tick_stopped =1 ; }
@@ -434,6 +432,7 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)if (!was_stopped && ts->tick_stopped) { ts->idle_jiffies = ts->last_jiffies; +select_nohz_load_balancer (1 );rcu_enter_nohz (); } }
@@ -501,7 +500,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)int cpu =smp_processor_id ();/* Update jiffies first */ -select_nohz_load_balancer (0 );tick_do_update_jiffies64 (now);cpumask_clear_cpu (cpu, nohz_cpu_mask);
@@ -552,6 +550,7 @@ void tick_nohz_exit_idle(void)if (ts->tick_stopped) {rcu_exit_nohz (); +select_nohz_load_balancer (0 );tick_nohz_restart_sched_tick (now, ts);tick_nohz_account_idle_ticks (ts); }
Signature · 1 line
1.7.5.4
Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic
Quoted 2 lines · Frederic Weisbecker
We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.
Again, the important part is missing, why is this correct?
I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.
AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.
Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.
Again, the important part is missing, why is this correct?
Quoted 8 lines
I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.
AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.
Right, we have the support for trigger_load_balance() in scheduler_tick() that is still missing.
What about using that CPU that has to stay awake with a periodic tick to handle jiffies? We could force that CPU to be the idle load balancer. The problem is perhaps to find the right frequency for doing that because we have all the rq to handle.
Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic
Quoted 16 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker
We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.
Again, the important part is missing, why is this correct?
Quoted 8 lines
I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.
AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.
Quoted 7 lines
Right, we have the support for
trigger_load_balance()inscheduler_tick()that is still missing.What about using that CPU that has to stay awake with a periodic tick to handle jiffies? We could force that CPU to be the idle load balancer. The problem is perhaps to find the right frequency for doing that because we have all the rq to handle.
If this CPU can also be the RCU grace-period advancer of last resort, that would make it easier to arrive at an improved RCU_FAST_NO_HZ.
Thanx, Paul
PATCH 9/32 nohz: Move ts->idle_calls into strict idle logic
In reply to Frederic WeisbeckerSplit the nohz switch in two parts, a first that checks if we can really stop the tick, and another that actually stop it. This way we can pull out idle_calls stat incrementation into strict idle logic.
--- kernel/time/tick-sched.c | 87 ++++++++++++++++++++++++--------------------- 1 files changed, 46 insertions(+), 41 deletions(-)
kernel/time/tick-sched.c +46 -41
@@ -249,48 +249,14 @@ EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);* Called either from the idle loop or from irq_exit() when an idle period was * just interrupted by an interrupt which did not cause a reschedule. */ -static void tick_nohz_stop_sched_tick (ktime_t now) +static void tick_nohz_stop_sched_tick (ktime_t now,int cpu,struct tick_sched *ts) {unsigned long seq, last_jiffies, next_jiffies, delta_jiffies; -struct tick_sched *ts;ktime_t last_update, expires;struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta; -int cpu; - - cpu =smp_processor_id (); - ts = &per_cpu (tick_cpu_sched, cpu); - -/* -* If this cpu is offline and it is the one which updates -* jiffies, then give up the assignment and let it be taken by -* the cpu which runs the tick timer next. If we don't drop -* this here the jiffies might be stale and do_timer() never -* invoked. -*/ -if (unlikely (!cpu_online (cpu))) { -if (cpu == tick_do_timer_cpu) - tick_do_timer_cpu =TICK_DO_TIMER_NONE ; - } -if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) -return ; - -if (need_resched ()) -return ; - -if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { -static int ratelimit; - -if (ratelimit <10 ) { -printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , - (unsigned int )local_softirq_pending ()); - ratelimit++; - } -return ; - } - ts->idle_calls++;/* Read jiffies and the time when jiffies were updated last */ do { seq = read_seqbegin(&xtime_lock);
@@ -422,18 +388,57 @@ out: ts->sleep_length =ktime_sub (dev->next_event, now); } +static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts) +{ +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =TICK_DO_TIMER_NONE ; + } + +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +return false; + +if (need_resched ()) +return false; + +if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { +static int ratelimit; + +if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , + (unsigned int )local_softirq_pending ()); + ratelimit++; + } +return false; + } + +return true; +} +static void __tick_nohz_enter_idle (struct tick_sched *ts,int cpu) {ktime_t now; -int was_stopped = ts->tick_stopped; now =tick_nohz_start_idle (cpu, ts); -tick_nohz_stop_sched_tick (now); -if (!was_stopped && ts->tick_stopped) { - ts->idle_jiffies = ts->last_jiffies; -select_nohz_load_balancer (1 ); -rcu_enter_nohz (); +if (tick_nohz_can_stop_tick (cpu, ts)) { +int was_stopped = ts->tick_stopped; + + ts->idle_calls++; +tick_nohz_stop_sched_tick (now, cpu, ts); + +if (!was_stopped && ts->tick_stopped) { + ts->idle_jiffies = ts->last_jiffies; +select_nohz_load_balancer (1 ); +rcu_enter_nohz (); + } } }
Signature · 1 line
1.7.5.4
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23
-0+static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts) +{ +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =TICK_DO_TIMER_NONE ; + } + +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +return false; + +if (need_resched ()) +return false; + +if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { +static int ratelimit; +
kernel/time/tick-sched.c out: · line 423
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , + (unsigned int )local_softirq_pending ()); + ratelimit++; + } +return false; + } + +return true; +}
Why aren't rcu_needs_cpu(), printk_needs_cpu() and arch_needs_cpu() not in there?
That are typical 'can we go sleep now?' functions.
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23
-0+static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts) +{ +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =TICK_DO_TIMER_NONE ; + } + +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +return false; + +if (need_resched ()) +return false; + +if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { +static int ratelimit; +kernel/time/tick-sched.c out: · line 423
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , + (unsigned int )local_softirq_pending ()); + ratelimit++; + } +return false; + } + +return true; +}Why aren't
rcu_needs_cpu(),printk_needs_cpu()andarch_needs_cpu()not in there?
Quoted 1 line
That are typical 'can we go sleep now?' functions.
Because when one of these functions are positive, the ts->next_jiffies and ts->last_jiffies stats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.
We still want tick_nohz_stop_sched_tick() to have the *needs_cpu() checks so that they can restore a HZ periodic behaviour on interrupt return if needed.
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 39 lines · Frederic Weisbecker
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23
-0+static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts) +{ +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =TICK_DO_TIMER_NONE ; + } + +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +return false; + +if (need_resched ()) +return false; + +if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { +static int ratelimit; +kernel/time/tick-sched.c out: · line 423
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , + (unsigned int )local_softirq_pending ()); + ratelimit++; + } +return false; + } + +return true; +}Why aren't
rcu_needs_cpu(),printk_needs_cpu()andarch_needs_cpu()not in there?
Quoted 5 lines
Quoted 1 line
That are typical 'can we go sleep now?' functions.
Because when one of these functions are positive, the
ts->next_jiffiesandts->last_jiffiesstats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.
Then call it tick_nohz_can_stop_tick_idle() or so, and create tick_nohz_can_stop_tick() to deal with all stuff.
Quoted 3 lines
We still want
tick_nohz_stop_sched_tick()to have the*needs_cpu()checks so that they can restore a HZ periodic behaviour on interrupt return if needed.
Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 40 lines · Peter Zijlstra
Quoted 39 lines · Frederic Weisbecker
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23
-0+static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts) +{ +/* +* If this cpu is offline and it is the one which updates +* jiffies, then give up the assignment and let it be taken by +* the cpu which runs the tick timer next. If we don't drop +* this here the jiffies might be stale and do_timer() never +* invoked. +*/ +if (unlikely (!cpu_online (cpu))) { +if (cpu == tick_do_timer_cpu) + tick_do_timer_cpu =TICK_DO_TIMER_NONE ; + } + +if (unlikely (ts->nohz_mode ==NOHZ_MODE_INACTIVE )) +return false; + +if (need_resched ()) +return false; + +if (unlikely (local_softirq_pending () &&cpu_online (cpu))) { +static int ratelimit; +kernel/time/tick-sched.c out: · line 423
Quoted 10 lines
kernel/time/tick-sched.c +10
-0+if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , + (unsigned int )local_softirq_pending ()); + ratelimit++; + } +return false; + } + +return true; +}Why aren't
rcu_needs_cpu(),printk_needs_cpu()andarch_needs_cpu()not in there?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
That are typical 'can we go sleep now?' functions.
Because when one of these functions are positive, the
ts->next_jiffiesandts->last_jiffiesstats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.Then call it
tick_nohz_can_stop_tick_idle()or so, and createtick_nohz_can_stop_tick()to deal with all stuff.
Yeah I need to have a deeper look into these checks.
Quoted 8 lines
Quoted 4 lines
We still want
tick_nohz_stop_sched_tick()to have the*needs_cpu()checks so that they can restore a HZ periodic behaviour on interrupt return if needed.Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.
I don't see a good reason to duplicate that logic with a pure restart from the IPI.
Quoted 7 lines
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when
tick_nohz_can_stop_tick()returns false. If it returns true, then I expect anything that needs it to re-enable it.
Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines
Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then
tick_nohz_stop_sched_tick()handles the cases with*needs_cpu()very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.
Quoted 2 lines
I don't see a good reason to duplicate that logic with a pure restart from the IPI.
That sounds like an optimization, and should thus be done later.
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when
tick_nohz_can_stop_tick()returns false. If it returns true, then I expect anything that needs it to re-enable it.Yeah. In the case of
need_resched()in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending ornohz_mode, I'm not sure...
softirqs shouldn't be pending when you go into nohz mode..
That is, I'm really not seeing what's wrong with the very simple:
if (tick_nohz_can_stop_tick())
tick_nohz_stop_tick();
and relying on everybody who invalidates tick_nohz_can_stop_tick(), to do:
tick_nohz_start_tick();
I'm also not quite sure why you always IPI, is that to avoid lock inversions?
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines
Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then
tick_nohz_stop_sched_tick()handles the cases with*needs_cpu()very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.
Quoted 4 lines
Quoted 2 lines
I don't see a good reason to duplicate that logic with a pure restart from the IPI.
That sounds like an optimization, and should thus be done later.
The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when
tick_nohz_can_stop_tick()returns false. If it returns true, then I expect anything that needs it to re-enable it.
Quoted 5 lines
Quoted 3 lines
Yeah. In the case of
need_resched()in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending ornohz_mode, I'm not sure...softirqs shouldn't be pending when you go into nohz mode..
You mean it can't happen or we don't want that to happen?
Quoted 4 lines
That is, I'm really not seeing what's wrong with the very simple:
Quoted 8 lines
if (
tick_nohz_can_stop_tick())tick_nohz_stop_tick();and relying on everybody who invalidates
tick_nohz_can_stop_tick(), to do:tick_nohz_start_tick();
May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.
Quoted 3 lines
I'm also not quite sure why you always IPI, is that to avoid lock inversions?
Exactly! I think I wrote that to some changelog but I'm not sure. I'll check that.
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 15 lines · Frederic Weisbecker
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines
Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then
tick_nohz_stop_sched_tick()handles the cases with*needs_cpu()very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I don't see a good reason to duplicate that logic with a pure restart from the IPI.
That sounds like an optimization, and should thus be done later.
The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.
care to point me to the relevant code, because I can't remember a half-assed nohz state.. then again, maybe I didn't look hard enough.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when
tick_nohz_can_stop_tick()returns false. If it returns true, then I expect anything that needs it to re-enable it.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Yeah. In the case of
need_resched()in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending ornohz_mode, I'm not sure...softirqs shouldn't be pending when you go into nohz mode..
You mean it can't happen or we don't want that to happen?
We don't want that.. going into nohz with pending softirqs means the softirqs will be delayed for an unknown amount of time, this should not occur.
tick_nohz_stop_sched_tick() has:
if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
static int ratelimit;
if (ratelimit < 10) {
printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
(unsigned int) local_softirq_pending());
ratelimit++;
}
goto end;
}which should warn us if this ever was to occur.
Quoted 8 lines
Quoted 8 lines
That is, I'm really not seeing what's wrong with the very simple:
if (
tick_nohz_can_stop_tick())tick_nohz_stop_tick();
Quoted 7 lines
Quoted 4 lines
and relying on everybody who invalidates
tick_nohz_can_stop_tick(), to do:tick_nohz_start_tick();May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.
How exactly do idle and non-idle differ? Its about stopping the tick, regardless of if we're idle or not if someone needs the thing we need to start it again.
Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
Quoted 17 lines · Peter Zijlstra
Quoted 14 lines · Frederic Weisbecker
Quoted 3 lines
Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.
Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then
tick_nohz_stop_sched_tick()handles the cases with*needs_cpu()very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.I don't see a good reason to duplicate that logic with a pure restart from the IPI.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
That sounds like an optimization, and should thus be done later.
The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.
care to point me to the relevant code, because I can't remember a half-assed nohz state.. then again, maybe I didn't look hard enough.
It's in tick_nohz_stop_sched_tick():
if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
arch_needs_cpu(cpu)) {
next_jiffies = last_jiffies + 1;
delta_jiffies = 1;
} else {
/* Get the next timer wheel timer */
next_jiffies = get_next_timer_interrupt(last_jiffies);
delta_jiffies = next_jiffies - last_jiffies;
}There are two cases:
- tick_nohz_stop_sched_tick() is called from idle. If one of the *_needs_cpu() is true, then the tick is not stopped. It doesn't even enter the nohz logic.
- tick_nohz_stop_sched_tick() is called from an interrupt exit while we are idle. If one of the *_needs_cpu() is true, then if the tick was previously stopped, we program the next timer to be in 1 jiffy but without exiting nohz mode. Otherwise if the tick was disabled, then we don't stop the tick.
In fact those *_needs_cpu() are treated like timer list timer or hrtimers that happen in one jiffy from now.
When we are in such a scheme:
tick_nohz_stop_sched_tick(); while (!need_resched()) hlt(); tick_nohz_restart_sched_tick();
It avoids to go back and forth between hz/nohz logic from inside the loop.
Quoted 33 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...
I would expect the tick not to be stopped when
tick_nohz_can_stop_tick()returns false. If it returns true, then I expect anything that needs it to re-enable it.Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Yeah. In the case of
need_resched()in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending ornohz_mode, I'm not sure...softirqs shouldn't be pending when you go into nohz mode..
You mean it can't happen or we don't want that to happen?
We don't want that.. going into nohz with pending softirqs means the softirqs will be delayed for an unknown amount of time, this should not occur.
tick_nohz_stop_sched_tick()has:if (unlikely(local_softirq_pending() && cpu_online(cpu))) { static int ratelimit; if (ratelimit < 10) { printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n", (unsigned int) local_softirq_pending()); ratelimit++; } goto end; }
Quoted 1 line
which should warn us if this ever was to occur.
Yeah. Moreover that check is weird because if softirqs are pending at that time, then the softirqd thread should be woken up. Because either we come from idle and we are not in an interrupt (softirq raised would use softirqd in not interrupt exit) or we come from an interrupt and the softirqs have been executed. If there were too much of them, softirqd would have been woken up as well.
Thus the need_resched() condition that comes before should have exited the function already.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
That is, I'm really not seeing what's wrong with the very simple:
if (
tick_nohz_can_stop_tick())tick_nohz_stop_tick();and relying on everybody who invalidates
tick_nohz_can_stop_tick(), to do:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
tick_nohz_start_tick();May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.
How exactly do idle and non-idle differ? Its about stopping the tick, regardless of if we're idle or not if someone needs the thing we need to start it again.
It may differ because of the way the idle loop is made, considering some things as 1 jiffy timer to avoid a hard whole tick restart.
PATCH 10/32 nohz: Move next idle expiring time record into idle logic area
In reply to Frederic WeisbeckerMove next idle expiry time record and idle sleeps tracking into idle entry functions as they are not generic nohz stats.
--- kernel/time/tick-sched.c | 21 ++++++++++++--------- 1 files changed, 12 insertions(+), 9 deletions(-)
kernel/time/tick-sched.c +12 -9
@@ -249,10 +249,10 @@ EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);* Called either from the idle loop or from irq_exit() when an idle period was * just interrupted by an interrupt which did not cause a reschedule. */ -static void tick_nohz_stop_sched_tick (ktime_t now,int cpu,struct tick_sched *ts) +static ktime_t tick_nohz_stop_sched_tick (ktime_t now,int cpu,struct tick_sched *ts) {unsigned long seq, last_jiffies, next_jiffies, delta_jiffies; -ktime_t last_update, expires; +ktime_t last_update, expires, ret = { .tv64 =0 };struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev;u64 time_delta;
@@ -338,6 +338,8 @@ static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *tif (ts->tick_stopped &&ktime_equal (expires, dev->next_event)) goto out; + ret = expires; +/* * nohz_stop_sched_tick can be called several times before * the nohz_restart_sched_tick is called. This happens when
@@ -350,11 +352,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *t ts->tick_stopped =1 ; } - ts->idle_sleeps++; - -/* Mark expires */ - ts->idle_expires = expires; -/* * If the expiration time == KTIME_MAX, then * in this case we simply stop the tick timer.
@@ -386,6 +383,8 @@ out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; ts->sleep_length =ktime_sub (dev->next_event, now); + +return ret; }static bool tick_nohz_can_stop_tick (int cpu,struct tick_sched *ts)
@@ -424,7 +423,7 @@ static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)static void __tick_nohz_enter_idle (struct tick_sched *ts,int cpu) { -ktime_t now; +ktime_t now, expires; now =tick_nohz_start_idle (cpu, ts);
@@ -432,7 +431,11 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)int was_stopped = ts->tick_stopped; ts->idle_calls++; -tick_nohz_stop_sched_tick (now, cpu, ts); + expires =tick_nohz_stop_sched_tick (now, cpu, ts); +if (expires.tv64 >0LL ) { + ts->idle_sleeps++; + ts->idle_expires = expires; + }if (!was_stopped && ts->tick_stopped) { ts->idle_jiffies = ts->last_jiffies;
Signature · 1 line
1.7.5.4
PATCH 11/32 cpuset: Set up interface for nohz flag
In reply to Frederic WeisbeckerPrepare the interface to implement the nohz cpuset flag. This flag, once set, will tell the system to try to shutdown the periodic timer tick when possible.
We use here a per cpu refcounter. As long as a CPU is contained into at least one cpuset that has the nohz flag set, it is part of the set of CPUs that run into adaptive nohz mode.
--- arch/Kconfig | 3 ++ include/linux/cpuset.h | 22 ++++++++++++++++++++ init/Kconfig | 8 +++++++ kernel/cpuset.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 85 insertions(+), 0 deletions(-)
arch/Kconfig +3 -0
@@ -172,6 +172,9 @@ config HAVE_ARCH_JUMP_LABEL bool config HAVE_ARCH_MUTEX_CPU_RELAX + bool + +config HAVE_CPUSETS_NO_HZ bool config HAVE_RCU_TABLE_FREE
include/linux/cpuset.h +22 -0
@@ -244,4 +244,26 @@ static inline void put_mems_allowed(void)#endif /* !CONFIG_CPUSETS */ +#ifdef CONFIG_CPUSETS_NO_HZ + +DECLARE_PER_CPU (int, cpu_adaptive_nohz_ref); + +static inline bool cpuset_cpu_adaptive_nohz (int cpu) +{ +if (per_cpu (cpu_adaptive_nohz_ref, cpu) >0 ) +return true; + +return false; +} + +static inline bool cpuset_adaptive_nohz (void ) +{ +if (__get_cpu_var (cpu_adaptive_nohz_ref) >0 ) +return true; + +return false; +} + +#endif /* CONFIG_CPUSETS_NO_HZ */ +#endif /* _LINUX_CPUSET_H */
init/Kconfig +8 -0
@@ -624,6 +624,14 @@ config PROC_PID_CPUSET depends on CPUSETS default y +config CPUSETS_NO_HZ + bool "Tickless cpusets" + depends on CPUSETS && HAVE_CPUSETS_NO_HZ + help + This options let you apply a nohz property to a cpuset such + that the periodic timer tick tries to be avoided when possible on + the concerned CPUs. + config CGROUP_CPUACCT bool "Simple CPU accounting cgroup subsystem" help
kernel/cpuset.c +52 -0
100 lines not on this page · raw message
Signature · 1 line
1.7.5.4
PATCH 12/32 nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
In reply to Frederic WeisbeckerTry to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
--- kernel/time/tick-sched.c | 52 ++++++++++++++++++++++++++++++++++++--------- 1 files changed, 41 insertions(+), 11 deletions(-)
kernel/time/tick-sched.c +41 -11
@@ -20,6 +20,7 @@#include <linux/profile.h> #include <linux/sched.h> #include <linux/module.h> +#include <linux/cpuset.h> #include <asm/irq_regs.h>
@@ -729,6 +730,45 @@ void tick_check_idle(int cpu)tick_check_nohz (cpu); } +#ifdef CONFIG_CPUSETS_NO_HZ + +/* +* Take the timer duty if nobody is taking care of it. +* If a CPU already does and and it's in a nohz cpuset, +* then take the charge so that it can switch to nohz mode. +*/ +static void tick_do_timer_check_handler (int cpu) +{ +int handler = tick_do_timer_cpu; + +if (unlikely (handler ==TICK_DO_TIMER_NONE )) { + tick_do_timer_cpu = cpu; + }else { +if (!cpuset_adaptive_nohz () && +cpuset_cpu_adaptive_nohz (handler)) + tick_do_timer_cpu = cpu; + } +} + +#else + +static void tick_do_timer_check_handler (int cpu) +{ +#ifdef CONFIG_NO_HZ +/* +* Check if the do_timer duty was dropped. We don't care about +* concurrency: This happens only when the cpu in charge went +* into a long sleep. If two cpus happen to assign themself to +* this duty, then the jiffies update is still serialized by +* xtime_lock. +*/ +if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) + tick_do_timer_cpu = cpu; +#endif +} + +#endif /* CONFIG_CPUSETS_NO_HZ */ +/* * High resolution timer specific code */
@@ -745,17 +785,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)ktime_t now =ktime_get ();int cpu =smp_processor_id (); -#ifdef CONFIG_NO_HZ -/* -* Check if the do_timer duty was dropped. We don't care about -* concurrency: This happens only when the cpu in charge went -* into a long sleep. If two cpus happen to assign themself to -* this duty, then the jiffies update is still serialized by -* xtime_lock. -*/ -if (unlikely (tick_do_timer_cpu ==TICK_DO_TIMER_NONE )) - tick_do_timer_cpu = cpu; -#endif +tick_do_timer_check_handler (cpu);/* Check, if the jiffies need an update */ if (tick_do_timer_cpu == cpu)
Signature · 1 line
1.7.5.4
Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
Quoted 4 lines · Frederic Weisbecker
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
You and Dmitiri might want to get together:
lkml.kernel.org/r/[email protected]
Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
You and Dmitiri might want to get together:
lkml.kernel.org/r/[email protected]
Right!
There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the do_timer duty and adaptive nohz CPUs run with a stale jiffies and walltime.
I need to handle that.
Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
You and Dmitiri might want to get together:
lkml.kernel.org/r/[email protected]
Quoted 7 lines
Right!
There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the
do_timerduty and adaptive nohz CPUs run with a stale jiffies and walltime.I need to handle that.
Frederic,
Please note that the patch that Peter references (lkml.kernel.org/r/[email protected]) supports only the nohz=off case.
Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
In reply to Frederic WeisbeckerQuoted 9 lines · Frederic Weisbecker
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
You and Dmitiri might want to get together:
Quoted 7 lines
Quoted 1 line
lkml.kernel.org/r/[email protected]
Right!
There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the
do_timerduty and adaptive nohz CPUs run with a stale jiffies and walltime.
Doesn't nohz already deal with the case of all cpus being idle? In that case the cpu that wakes up first gets to play catch up on irq_enter() or so.
Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.
You and Dmitiri might want to get together:
lkml.kernel.org/r/[email protected]
Right!
Quoted 7 lines
Quoted 3 lines
There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the
do_timerduty and adaptive nohz CPUs run with a stale jiffies and walltime.Doesn't nohz already deal with the case of all cpus being idle? In that case the cpu that wakes up first gets to play catch up on
irq_enter()or so.
Sure and that works for the nohz idle case. But that's not enough anymore in the case of adaptive nohz CPUs. They can run for a while without the tick and if nobody else maintains a tick either then jiffies and walltime are not maintained anymore.
PATCH 13/32 nohz: Adaptive tick stop and restart on nohz cpuset
In reply to Frederic WeisbeckerWhen a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.
Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.
This assumes we are using TTWU_QUEUE sched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.
I also need to look at the ARCH_WANT_INTERRUPTS_ON_CTXW case and the remote wakeups.
--- include/linux/cpuset.h | 4 +++ include/linux/sched.h | 6 ++++ include/linux/tick.h | 12 ++++++++- init/Kconfig | 2 +- kernel/sched.c | 35 +++++++++++++++++++++++++ kernel/softirq.c | 4 +- kernel/time/tick-sched.c | 63 ++++++++++++++++++++++++++++++++++++++------- 7 files changed, 112 insertions(+), 14 deletions(-)
include/linux/cpuset.h +4 -0
@@ -264,6 +264,10 @@ static inline bool cpuset_adaptive_nohz(void)return false; } +extern void cpuset_update_nohz (void ); +#else +static inline void cpuset_update_nohz (void ) { } +#endif /* CONFIG_CPUSETS_NO_HZ */ #endif /* _LINUX_CPUSET_H */
include/linux/sched.h +6 -0
@@ -2652,6 +2652,12 @@ static inline void inc_syscw(struct task_struct *tsk)#define TASK_SIZE_OF (tsk) TASK_SIZE#endif +#ifdef CONFIG_CPUSETS_NO_HZ +extern bool cpuset_nohz_can_stop_tick (void ); +#else +static inline bool cpuset_nohz_can_stop_tick (void ) {return false; } +#endif +#ifdef CONFIG_MM_OWNER extern void mm_update_next_owner (struct mm_struct *mm);extern void mm_init_owner (struct mm_struct *mm,struct task_struct *p);
include/linux/tick.h +11 -1
@@ -122,11 +122,21 @@ static inline int tick_oneshot_mode_active(void) { return 0; }# ifdef CONFIG_NO_HZ extern void tick_nohz_enter_idle (void );extern void tick_nohz_exit_idle (void ); +extern void tick_nohz_restart_sched_tick (void );extern void tick_nohz_irq_exit (void );extern ktime_t tick_nohz_get_sleep_length (void );extern u64 get_cpu_idle_time_us (int cpu,u64 *last_update_time);extern u64 get_cpu_iowait_time_us (int cpu,u64 *last_update_time); -# else + +#ifdef CONFIG_CPUSETS_NO_HZ +DECLARE_PER_CPU (int, task_nohz_mode); + +extern int tick_nohz_adaptive_mode (void ); +#else /* !CPUSETS_NO_HZ */ +static inline int tick_nohz_adaptive_mode (void ) {return 0 ; } +#endif /* CPUSETS_NO_HZ */ + +# else /* !NO_HZ */ static inline void tick_nohz_enter_idle (void ) { }static inline void tick_nohz_exit_idle (void ) { }
init/Kconfig +1 -1
@@ -626,7 +626,7 @@ config PROC_PID_CPUSET config CPUSETS_NO_HZ bool "Tickless cpusets" - depends on CPUSETS && HAVE_CPUSETS_NO_HZ + depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS help This options let you apply a nohz property to a cpuset such that the periodic timer tick tries to be avoided when possible on
kernel/sched.c +35 -0
@@ -2433,6 +2433,38 @@ static void update_avg(u64 *avg, u64 sample) }#endif +#ifdef CONFIG_CPUSETS_NO_HZ +DEFINE_PER_CPU (int, task_nohz_mode); + +bool cpuset_nohz_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +} + +static void cpuset_nohz_restart_tick (void ) +{ +__get_cpu_var (task_nohz_mode) =0 ; +tick_nohz_restart_sched_tick (); +} + +void cpuset_update_nohz (void ) +{ +if (!tick_nohz_adaptive_mode ()) +return ; + +if (!cpuset_nohz_can_stop_tick ()) +cpuset_nohz_restart_tick (); +} +#endif +static void ttwu_stat (struct task_struct *p,int cpu,int wake_flags) {
@@ -2560,6 +2592,8 @@ static void sched_ttwu_pending(void)ttwu_do_activate (rq, p,0 ); } +cpuset_update_nohz (); +raw_spin_unlock (&rq->lock); }
@@ -2620,6 +2654,7 @@ static void ttwu_queue(struct task_struct *p, int cpu)raw_spin_lock (&rq->lock);ttwu_do_activate (rq, p,0 ); +cpuset_update_nohz ();raw_spin_unlock (&rq->lock); }
kernel/softirq.c +2 -2
@@ -297,7 +297,7 @@ void irq_enter(void)int cpu =smp_processor_id ();rcu_irq_enter (); -if (idle_cpu (cpu) && !in_interrupt ()) { +if ((idle_cpu (cpu) ||tick_nohz_adaptive_mode ()) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd * here, as softirq will be serviced on return from interrupt.
@@ -342,7 +342,7 @@ void irq_exit(void)rcu_irq_exit ();#ifdef CONFIG_NO_HZ /* Make sure that timer wheel updates are propagated */ -if (idle_cpu (smp_processor_id ()) && !in_interrupt () && !need_resched ()) +if (!in_interrupt ())tick_nohz_irq_exit ();#endif preempt_enable_no_resched ();
kernel/time/tick-sched.c +53 -10
113 lines not on this page · raw message
Signature · 1 line
1.7.5.4
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 7 lines · Frederic Weisbecker
When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.
Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.
Shouldn't you first put the syscall hooks in place before allowing the tick to be switched off? It seems this patch is somewhat too early in the series.
Quoted 5 lines
This assumes we are using
TTWU_QUEUEsched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.
We could certainly remove that feature flag and always use it, it was mostly a transition debug switch in case something didn't work or performance issues were found due to this.
Quoted 2 lines
I also need to look at the
ARCH_WANT_INTERRUPTS_ON_CTXWcase and the remote wakeups.
Well, ideally we'd simply get rid of that, rmk has some preliminary patches in that direction.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.
Quoted 7 lines
Quoted 3 lines
Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.
Shouldn't you first put the syscall hooks in place before allowing the tick to be switched off? It seems this patch is somewhat too early in the series.
I don't think it's necessary, that part doesn't depend on userspace hooks. The whole thing is enabled very late anyway.
Quoted 9 lines
Quoted 5 lines
This assumes we are using
TTWU_QUEUEsched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.We could certainly remove that feature flag and always use it, it was mostly a transition debug switch in case something didn't work or performance issues were found due to this.
Ok, good.
Quoted 5 lines
Quoted 2 lines
I also need to look at the
ARCH_WANT_INTERRUPTS_ON_CTXWcase and the remote wakeups.Well, ideally we'd simply get rid of that, rmk has some preliminary patches in that direction.
Great!
Thanks.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
In reply to Frederic Weisbeckerkernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450
Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12
-0+bool cpuset_nohz_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +}
int sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 16 lines · Peter Zijlstra
kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450
Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12
-0+bool cpuset_nohz_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +}int
sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.
tick_nohz_stop_sched_tick() already handles that by keeping a periodic behaviour if one of these conditions are met.
It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 18 lines · Frederic Weisbecker
Quoted 16 lines · Peter Zijlstra
kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450
Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12
-0+bool cpuset_nohz_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +}int
sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.
Quoted 2 lines
tick_nohz_stop_sched_tick()already handles that by keeping a periodic behaviour if one of these conditions are met.
What? tick_nohz_stop_sched_tick() most surely cannot access struct rq, so it cannot do the nr_running test.
Quoted 2 lines
It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.
Again, what?
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
Quoted 16 lines · Peter Zijlstra
kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450
Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12
-0+bool cpuset_nohz_can_stop_tick (void ) +{ +struct rq *rq; + + rq =this_rq (); + +/* More than one running task need preemption */ +if (rq->nr_running >1 ) +return false; + +return true; +}int
sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.
Quoted 5 lines
Quoted 2 lines
tick_nohz_stop_sched_tick()already handles that by keeping a periodic behaviour if one of these conditions are met.What?
tick_nohz_stop_sched_tick()most surely cannot access struct rq, so it cannot do thenr_runningtest.
I was talking about {rcu,printk,arch}_needs_cpu() functions.
Quoted 5 lines
Quoted 3 lines
It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.
Again, what?
If the tick is stopped then an irq fires and something calls printk() or call_rcu() then on interrupt return, tick_nohz_stop_sched_tick() checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 3 lines · Frederic Weisbecker
tick_nohz_stop_sched_tick()checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.
tick_nohz_stop_sched_tick() should not restore stuff, it should at worst fail to stop but never enable that's just weird.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
tick_nohz_stop_sched_tick()checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.
tick_nohz_stop_sched_tick()should not restore stuff, it should at worst fail to stop but never enable that's just weird.
It's not really enablement, it's just nohz behaviour but the next tick is in HZ :o)
Like you said before it's an optimization. I can do the things differently for idle and non-idle cases there but I'm not sure it's really a good thing.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
Quoted 6 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
tick_nohz_stop_sched_tick()checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.
Quoted 7 lines
Quoted 2 lines
tick_nohz_stop_sched_tick()should not restore stuff, it should at worst fail to stop but never enable that's just weird.It's not really enablement, it's just nohz behaviour but the next tick is in HZ :o)
Like you said before it's an optimization.
I'm still not feeling very confident about all that..
Quoted 2 lines
I can do the things differently
for idle and non-idle cases there but I'm not sure it's really a good thing.
its not, but I'm very sure I've lost you on why that should be the case.
Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset
In reply to Frederic WeisbeckerQuoted 29 lines · Frederic Weisbecker
kernel/sched.c +18
-0+static void cpuset_nohz_restart_tick (void ) +{ +__get_cpu_var (task_nohz_mode) =0 ; +tick_nohz_restart_sched_tick (); +} + +void cpuset_update_nohz (void ) +{ +if (!tick_nohz_adaptive_mode ()) +return ; + +if (!cpuset_nohz_can_stop_tick ()) +cpuset_nohz_restart_tick (); +} +#endif +static void ttwu_stat (struct task_struct *p,int cpu,int wake_flags) {@@ -2560,6 +2592,8 @@ static void sched_ttwu_pending(void)ttwu_do_activate (rq, p,0 ); } +cpuset_update_nohz (); +raw_spin_unlock (&rq->lock); }
kernel/sched.c static void ttwu_queue(struct task_struct *p, int cpu) · line 2659
Quoted 6 lines
kernel/sched.c +1
-0@@ -2620,6 +2654,7 @@ static void ttwu_queue(struct task_struct *p, int cpu)raw_spin_lock (&rq->lock);ttwu_do_activate (rq, p,0 ); +cpuset_update_nohz ();raw_spin_unlock (&rq->lock); }
That really has nothing to do with cpusets, why doesn't that live in the tick_nohz_* namespace? Something like tick_nohz_sched_wakeup() or so.
PATCH 14/32 nohz/cpuset: Don't turn off the tick if rcu needs it
In reply to Frederic WeisbeckerIf RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.
--- include/linux/rcupdate.h | 1 + kernel/rcutree.c | 3 +-- kernel/sched.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/rcupdate.h +1 -0
@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_pending (int cpu);struct notifier_block ;#ifdef CONFIG_NO_HZ
kernel/rcutree.c +1 -2
@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;module_param (rcu_cpu_stall_suppress, int,0644 );static void force_quiescent_state (struct rcu_state *rsp,int relaxed); -static int rcu_pending (int cpu);/* * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)* by the current CPU, returning 1 if so. This function is part of the * RCU implementation; it is -not- an exported member of the RCU API. */ -static int rcu_pending (int cpu) +int rcu_pending (int cpu) {return __rcu_pending (&rcu_sched_state, &per_cpu (rcu_sched_data, cpu)) || __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
kernel/sched.c +14 -0
@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);bool cpuset_nohz_can_stop_tick (void ) {struct rq *rq; +int cpu; rq =this_rq ();
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)if (rq->nr_running >1 )return false; + cpu =smp_processor_id (); + +/* +* FIXME: will probably be removed soon as it's +* already checked from tick_nohz_stop_sched_tick() +*/ +if (rcu_needs_cpu (cpu)) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (cpu)) +return false; +return true; }
Signature · 1 line
1.7.5.4
Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it
Quoted 2 lines · Frederic Weisbecker
If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we
s/dynctik/dyntick/ ;-)
Quoted 3 lines
are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.
One question below...
Thanx, Paul
Quoted 68 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Anton Blanchard <[email protected]>Cc Avi Kivity <[email protected]>Cc Ingo Molnar <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Paul E . McKenney <[email protected]>Cc Paul Menage <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Tim Pepper <[email protected]>--- include/linux/rcupdate.h | 1 + kernel/rcutree.c | 3 +-- kernel/sched.c | 14 ++++++++++++++ 3 files changed, 16 insertions(+), 2 deletions(-)include/linux/rcupdate.h +1
-0@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_pending (int cpu);struct notifier_block ;#ifdef CONFIG_NO_HZ kernel/rcutree.c +1
-2@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;module_param (rcu_cpu_stall_suppress, int,0644 );static void force_quiescent_state (struct rcu_state *rsp,int relaxed); -static int rcu_pending (int cpu);/* * Return the number of RCU-sched batches processed thus far for debug & stats. @@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)* by the current CPU, returning 1 if so. This function is part of the * RCU implementation; it is -not- an exported member of the RCU API. */ -static int rcu_pending (int cpu) +int rcu_pending (int cpu) {return __rcu_pending (&rcu_sched_state, &per_cpu (rcu_sched_data, cpu)) || __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||kernel/sched.c +2
-0@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);bool cpuset_nohz_can_stop_tick (void ) {struct rq *rq; +int cpu; rq =this_rq ();@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)if (rq->nr_running >1 )return false; + cpu =smp_processor_id ();
kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460
Quoted 10 lines
kernel/sched.c +10
-0+ +/* +* FIXME: will probably be removed soon as it's +* already checked from tick_nohz_stop_sched_tick() +*/ +if (rcu_needs_cpu (cpu)) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (cpu))
This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)
But force_quiescent_state() will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?
Quoted 8 lines
kernel/sched.c +2
-0+return false; +return true; }Signature · 1 line
1.7.5.4
Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Frederic Weisbecker
If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we
s/dynctik/dyntick/ ;-)
Heh! :)
Quoted 49 lines
Quoted 49 lines
include/linux/rcupdate.h +1
-0@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_pending (int cpu);struct notifier_block ;#ifdef CONFIG_NO_HZ kernel/rcutree.c +1
-2@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;module_param (rcu_cpu_stall_suppress, int,0644 );static void force_quiescent_state (struct rcu_state *rsp,int relaxed); -static int rcu_pending (int cpu);/* * Return the number of RCU-sched batches processed thus far for debug & stats. @@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)* by the current CPU, returning 1 if so. This function is part of the * RCU implementation; it is -not- an exported member of the RCU API. */ -static int rcu_pending (int cpu) +int rcu_pending (int cpu) {return __rcu_pending (&rcu_sched_state, &per_cpu (rcu_sched_data, cpu)) || __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||kernel/sched.c +1
-0@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);bool cpuset_nohz_can_stop_tick (void ) {struct rq *rq; +int cpu; rq =this_rq ();@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)if (rq->nr_running >1 )return false;
Quoted 15 lines
kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460
Quoted 11 lines
kernel/sched.c +11
-0+ cpu =smp_processor_id (); + +/* +* FIXME: will probably be removed soon as it's +* already checked from tick_nohz_stop_sched_tick() +*/ +if (rcu_needs_cpu (cpu)) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (cpu))This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)
Yeah this can happen. This is called from the timer interrupt or from an IPI. We can be in any kind of rcu critical section.
Quoted 3 lines
But
force_quiescent_state()will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?
Yeah we should receive IPIs from CPUs that need us. This can be an optimization though. No need to run into a cycle of on timers shutdown/restart if we can complete something right away.
Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it
Quoted 77 lines · Frederic Weisbecker
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Frederic Weisbecker
If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we
s/dynctik/dyntick/ ;-)
Heh! :)
Quoted 49 lines
Quoted 49 lines
include/linux/rcupdate.h +1
-0@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_pending (int cpu);struct notifier_block ;#ifdef CONFIG_NO_HZ kernel/rcutree.c +1
-2@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;module_param (rcu_cpu_stall_suppress, int,0644 );static void force_quiescent_state (struct rcu_state *rsp,int relaxed); -static int rcu_pending (int cpu);/* * Return the number of RCU-sched batches processed thus far for debug & stats. @@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)* by the current CPU, returning 1 if so. This function is part of the * RCU implementation; it is -not- an exported member of the RCU API. */ -static int rcu_pending (int cpu) +int rcu_pending (int cpu) {return __rcu_pending (&rcu_sched_state, &per_cpu (rcu_sched_data, cpu)) || __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||kernel/sched.c +1
-0@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);bool cpuset_nohz_can_stop_tick (void ) {struct rq *rq; +int cpu; rq =this_rq ();@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)if (rq->nr_running >1 )return false;Quoted 15 lines
kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460
Quoted 11 lines
kernel/sched.c +11
-0+ cpu =smp_processor_id (); + +/* +* FIXME: will probably be removed soon as it's +* already checked from tick_nohz_stop_sched_tick() +*/ +if (rcu_needs_cpu (cpu)) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (cpu))This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)
Yeah this can happen. This is called from the timer interrupt or from an IPI. We can be in any kind of rcu critical section.
Quoted 7 lines
Quoted 2 lines
But
force_quiescent_state()will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?Yeah we should receive IPIs from CPUs that need us. This can be an optimization though. No need to run into a cycle of on timers shutdown/restart if we can complete something right away.
Never mind... I was confusing this with rcu_needs_cpu(), which should not be called from within an RCU-sched or RCU-bh read-side critical section.
It is plenty fine to call rcu_pending() from within RCU read-side critical sections.
Thanx, Paul
Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it
In reply to Frederic WeisbeckerQuoted 9 lines · Frederic Weisbecker
kernel/sched.c +5
-0@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)if (rq->nr_running >1 )return false; + cpu =smp_processor_id (); + +/* +* FIXME: will probably be removed soon as it's +* already checked from tick_nohz_stop_sched_tick()
kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2464
Quoted 10 lines
kernel/sched.c +8
-0+*/ +if (rcu_needs_cpu (cpu)) +return false; + +/* Is there a grace period to complete ? */ +if (rcu_pending (cpu)) +return false; +return true; }
This really shouldn't live in sched.c, also I would have expected tick_nohz_can_stop_tick() to do this, its named like it does this.
PATCH 15/32 nohz/cpuset: Restart tick when switching to idle task
In reply to Frederic WeisbeckerIdeally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.
That optimization requires some minor tweaks here and there though, lets handle that later.
--- kernel/sched.c | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-)
kernel/sched.c +13 -0
@@ -2477,6 +2477,18 @@ void cpuset_update_nohz(void)if (!cpuset_nohz_can_stop_tick ())cpuset_nohz_restart_tick (); } + +static void cpuset_nohz_task_switch (struct task_struct *next) +{ +int cpu =smp_processor_id (); + +if (tick_nohz_adaptive_mode () && next ==idle_task (cpu)) +cpuset_nohz_restart_tick (); +} +#else +static void cpuset_nohz_task_switch (struct task_struct *next) +{ +}#endif static void
@@ -3023,6 +3035,7 @@ static inline void prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { +cpuset_nohz_task_switch (next);sched_info_switch (prev, next);perf_event_task_sched_out (prev, next);fire_sched_out_preempt_notifiers (prev, next);
Signature · 1 line
1.7.5.4
Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task
Quoted 6 lines · Frederic Weisbecker
Ideally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.
That optimization requires some minor tweaks here and there though, lets handle that later.
You have a knack for confusing changelogs.. so basically you say:
Restart the tick when we switch to idle.
Now all that needs is an explanation of why..
Also, please drop the whole cpuset_nohz_ stuff, this really isn't about cpusets, cpusets simly provide the interface, the functionality lives in the tick_nohz_ namespace.
Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Ideally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.
Quoted 8 lines
Quoted 2 lines
That optimization requires some minor tweaks here and there though, lets handle that later.
You have a knack for confusing changelogs.. so basically you say:
Restart the tick when we switch to idle.
Now all that needs is an explanation of why..
In the end we don't want to restart the tick when we switch to idle. But to support that I'll need to tweak various things on idle time accounting, otherwise the [cpu] time spent in idle is going to be accounted as system time.
That's definetly headed to be temporary.
Quoted 3 lines
Also, please drop the whole
cpuset_nohz_stuff, this really isn't about cpusets, cpusets simly provide the interface, the functionality lives in thetick_nohz_namespace.
Agreed this sucks.
Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task
Quoted 10 lines · Frederic Weisbecker
Quoted 5 lines
... so basically you say:
Restart the tick when we switch to idle.
Now all that needs is an explanation of why..
In the end we don't want to restart the tick when we switch to idle. But to support that I'll need to tweak various things on idle time accounting, otherwise the [cpu] time spent in idle is going to be accounted as system time.
Ok, so then the changelog should say so, preferably with a little bit more detail on how exactly the idle time accounting would go tits-up.
PATCH 16/32 nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
In reply to Frederic WeisbeckerWake up a CPU when a timer list timer is enqueued there and the CPU is in adaptive nohz mode. Sending an IPI to it makes it reconsidering the next timer to program on top of recent updates.
--- include/linux/sched.h | 4 ++-- kernel/sched.c | 33 ++++++++++++++++++++++++++++++++- kernel/time/tick-sched.c | 5 ++++- kernel/timer.c | 2 +- 4 files changed, 39 insertions(+), 5 deletions(-)
include/linux/sched.h +2 -2
@@ -1947,9 +1947,9 @@ static inline void idle_task_exit(void) {}#endif #if defined(CONFIG_NO_HZ ) && defined(CONFIG_SMP ) -extern void wake_up_idle_cpu (int cpu); +extern void wake_up_nohz_cpu (int cpu);#else -static inline void wake_up_idle_cpu (int cpu) { } +static inline void wake_up_nohz_cpu (int cpu) { }#endif extern unsigned int sysctl_sched_latency;
kernel/sched.c +32 -1
@@ -1234,7 +1234,7 @@ unlock:* account when the CPU goes back to idle and evaluates the timer * wheel for the next timer event. */ -void wake_up_idle_cpu (int cpu) +static void wake_up_idle_cpu (int cpu) {struct rq *rq =cpu_rq (cpu);
@@ -1264,6 +1264,37 @@ void wake_up_idle_cpu(int cpu) smp_send_reschedule(cpu); } + +static bool wake_up_cpuset_nohz_cpu (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ +/* Ensure task_nohz_mode update is visible */ +smp_rmb (); +/* +* Even if task_nohz_mode is set concurrently, what +* matters is that by the time we do that check, we know +* that the CPU has not reached tick_nohz_stop_sched_tick(). +* As we are holding the base->lock and that lock needs +* to be taken by tick_nohz_stop_sched_tick() we know +* we are preceding it and it will see our update +* synchronously. Thus we know we don't need to send an +* IPI to that CPU. +*/ +if (per_cpu (task_nohz_mode, cpu)) { +smp_cpuset_update_nohz (cpu); +return true; + } +#endif + +return false; +} + +void wake_up_nohz_cpu (int cpu) +{ +if (!wake_up_cpuset_nohz_cpu (cpu)) +wake_up_idle_cpu (cpu); +} +#endif /* CONFIG_NO_HZ */ static u64 sched_avg_period (void )
kernel/time/tick-sched.c +4 -1
@@ -767,8 +767,11 @@ static void tick_nohz_cpuset_stop_tick(int user)if (!cpuset_adaptive_nohz () ||tick_nohz_adaptive_mode ())return ; -if (cpuset_nohz_can_stop_tick ()) +if (cpuset_nohz_can_stop_tick ()) {__get_cpu_var (task_nohz_mode) =1 ; +/* Nohz mode must be visible to wake_up_nohz_cpu() */ +smp_wmb (); + } }/*
kernel/timer.c +1 -1
@@ -880,7 +880,7 @@ void add_timer_on(struct timer_list *timer, int cpu)* makes sure that a CPU on the way to idle can not evaluate * the timer wheel. */ -wake_up_idle_cpu (cpu); +wake_up_nohz_cpu (cpu);spin_unlock_irqrestore (&base->lock, flags); }EXPORT_SYMBOL_GPL (add_timer_on);
Signature · 1 line
1.7.5.4
Re: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
Quoted 11 lines · Frederic Weisbecker
Wake up a CPU when a timer list timer is enqueued there and the CPU is in adaptive nohz mode. Sending an IPI to it makes it reconsidering the next timer to program on top of recent updates.
include/linux/sched.h | 4 ++-- kernel/sched.c | 33 ++++++++++++++++++++++++++++++++- kernel/time/tick-sched.c | 5 ++++- kernel/timer.c | 2 +- 4 files changed, 39 insertions(+), 5 deletions(-)
So here I would have expected timer_needs_cpu() and an addition to tick_nohz_can_stop_tick(). Why does sched.c get touched at all?
Also, all the delta_jiffies stuff in the current tick_nohz_stop_sched_tick() deals with this, why duplicate the logic?
Peter Zijlstra
maintainerRe: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
In reply to Frederic WeisbeckerQuoted 4 lines · Peter Zijlstra
Also, all the
delta_jiffiesstuff in the currenttick_nohz_stop_sched_tick()deals with this, why duplicate the logic?
Damn, n/m this is about new timers.. tick_nohz_new_timer() then.
Re: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Also, all the
delta_jiffiesstuff in the currenttick_nohz_stop_sched_tick()deals with this, why duplicate the logic?Damn, n/m this is about new timers..
tick_nohz_new_timer()then.
I can call that tick_nohz_new_timer() if you prefer yeah. I personally don't mind either name.
PATCH 17/32 x86: New cpuset nohz irq vector
In reply to Frederic WeisbeckerWe need a way to send an irq IPI (or local) in any case and asynchronously in order to restart the tick for CPUs in nohz adaptive mode
Generic smp operations don't fit into this because they need interrupts to be enabled and they try to execute the functions in place if the dest CPU is the current one. But we always need the function to be executed in irq context so it happens quickly and restarting the tick doesn't mess up with random lock scenarios in place.
In fact this is a temporary solution, what we really need is an irq work subsystem that supports remote enqueuing.
--- arch/x86/include/asm/entry_arch.h | 3 +++ arch/x86/include/asm/hw_irq.h | 6 ++++++ arch/x86/include/asm/irq_vectors.h | 2 ++ arch/x86/include/asm/smp.h | 11 +++++++++++ arch/x86/kernel/entry_64.S | 4 ++++ arch/x86/kernel/irqinit.c | 4 ++++ arch/x86/kernel/smp.c | 26 ++++++++++++++++++++++++++ 7 files changed, 56 insertions(+), 0 deletions(-)
arch/x86/include/asm/entry_arch.h +3 -0
@@ -10,6 +10,9 @@* through the ICC by us (IPIs) */ #ifdef CONFIG_SMP +#ifdef CONFIG_CPUSETS_NO_HZ +BUILD_INTERRUPT(cpuset_update_nohz_interrupt,CPUSET_UPDATE_NOHZ_VECTOR) +#endif BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)BUILD_INTERRUPT (call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR )
arch/x86/include/asm/hw_irq.h +6 -0
@@ -34,6 +34,9 @@ extern void irq_work_interrupt(void);extern void spurious_interrupt (void );extern void thermal_interrupt (void );extern void reschedule_interrupt (void ); +#ifdef CONFIG_CPUSETS_NO_HZ +extern void cpuset_update_nohz_interrupt (void ); +#endif extern void mce_self_interrupt (void );extern void invalidate_interrupt (void );
@@ -153,6 +156,9 @@ extern asmlinkage void smp_irq_move_cleanup_interrupt(void);#endif #ifdef CONFIG_SMP extern void smp_reschedule_interrupt (struct pt_regs *); +#ifdef CONFIG_CPUSETS_NO_HZ +extern void smp_cpuset_update_nohz_interrupt (struct pt_regs *); +#endif extern void smp_call_function_interrupt (struct pt_regs *);extern void smp_call_function_single_interrupt (struct pt_regs *);#ifdef CONFIG_X86_32
arch/x86/include/asm/irq_vectors.h +2 -0
@@ -117,6 +117,8 @@/* Xen vector callback to receive events in a HVM domain */ #define XEN_HVM_EVTCHN_CALLBACK 0xf3 +#define CPUSET_UPDATE_NOHZ_VECTOR 0xf2 +/* * Local APIC timer IRQ vector is on a different priority level, * to work around the 'lost local interrupt if more than 2 IRQ
arch/x86/include/asm/smp.h +11 -0
@@ -70,6 +70,10 @@ struct smp_ops {void (*stop_other_cpus)(int wait);void (*smp_send_reschedule)(int cpu); +#ifdef CONFIG_CPUSETS_NO_HZ +void (*smp_cpuset_update_nohz)(int cpu); +#endif +int (*cpu_up)(unsigned cpu);int (*cpu_disable)(void );void (*cpu_die)(unsigned int cpu);
@@ -138,6 +142,13 @@ static inline void smp_send_reschedule(int cpu) smp_ops.smp_send_reschedule (cpu); } +static inline void smp_cpuset_update_nohz (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ + smp_ops.smp_cpuset_update_nohz (cpu); +#endif +} +static inline void arch_send_call_function_single_ipi (int cpu) { smp_ops.send_call_func_single_ipi (cpu);
arch/x86/kernel/entry_64.S +4 -0
@@ -996,6 +996,10 @@ apicinterrupt CALL_FUNCTION_VECTOR \ call_function_interrupt smp_call_function_interrupt apicinterrupt RESCHEDULE_VECTOR \ reschedule_interrupt smp_reschedule_interrupt +#ifdef CONFIG_CPUSETS_NO_HZ +apicinterrupt CPUSET_UPDATE_NOHZ_VECTOR \ + cpuset_update_nohz_interrupt smp_cpuset_update_nohz_interrupt +#endif #endif apicinterrupt ERROR_APIC_VECTOR \
arch/x86/kernel/irqinit.c +4 -0
@@ -172,6 +172,10 @@ static void __init smp_intr_init(void)*/ alloc_intr_gate (RESCHEDULE_VECTOR , reschedule_interrupt); +#ifdef CONFIG_CPUSETS_NO_HZ +alloc_intr_gate (CPUSET_UPDATE_NOHZ_VECTOR , cpuset_update_nohz_interrupt); +#endif +/* IPIs for invalidation */ #define ALLOC_INVTLB_VEC (NR ) \ alloc_intr_gate(INVALIDATE_TLB_VECTOR_START+NR, \
arch/x86/kernel/smp.c +26 -0
@@ -22,6 +22,7 @@#include <linux/interrupt.h> #include <linux/cpu.h> #include <linux/gfp.h> +#include <linux/cpuset.h> #include <asm/mtrr.h> #include <asm/tlbflush.h>
@@ -121,6 +122,17 @@ static void native_smp_send_reschedule(int cpu) apic->send_IPI_mask (cpumask_of (cpu),RESCHEDULE_VECTOR ); } +#ifdef CONFIG_CPUSETS_NO_HZ +static void native_smp_cpuset_update_nohz (int cpu) +{ +if (unlikely (cpu_is_offline (cpu))) { +WARN_ON (1 ); +return ; + } + apic->send_IPI_mask (cpumask_of (cpu),CPUSET_UPDATE_NOHZ_VECTOR ); +} +#endif +void native_send_call_func_single_ipi (int cpu) { apic->send_IPI_mask (cpumask_of (cpu),CALL_FUNCTION_SINGLE_VECTOR );
@@ -206,6 +218,17 @@ void smp_reschedule_interrupt(struct pt_regs *regs)*/ } +#ifdef CONFIG_CPUSETS_NO_HZ +void smp_cpuset_update_nohz_interrupt (struct pt_regs *regs) +{ +ack_APIC_irq (); +irq_enter (); +cpuset_update_nohz (); +inc_irq_stat (irq_call_count); +irq_exit (); +} +#endif +void smp_call_function_interrupt (struct pt_regs *regs) {ack_APIC_irq ();
@@ -231,6 +254,9 @@ struct smp_ops smp_ops = { .stop_other_cpus = native_stop_other_cpus, .smp_send_reschedule = native_smp_send_reschedule, +#ifdef CONFIG_CPUSETS_NO_HZ + .smp_cpuset_update_nohz = native_smp_cpuset_update_nohz, +#endif .cpu_up = native_cpu_up, .cpu_die = native_cpu_die,
Signature · 1 line
1.7.5.4
PATCH 18/32 nohz/cpuset: Don't stop the tick if posix cpu timers are running
In reply to Frederic WeisbeckerIf either a per thread or a per process posix cpu timer is running, don't stop the tick.
TODO: restart the tick if it is stopped and a posix cpu timer is enqueued. Check we probably need a memory barrier for the per process posix timer that can be enqueued from another task of the group.
--- include/linux/posix-timers.h | 1 + kernel/posix-cpu-timers.c | 12 ++++++++++++ kernel/sched.c | 4 ++++ 3 files changed, 17 insertions(+), 0 deletions(-)
include/linux/posix-timers.h +1 -0
@@ -116,6 +116,7 @@ int posix_timer_event(struct k_itimer *timr, int si_private);void posix_cpu_timer_schedule (struct k_itimer *timer);void run_posix_cpu_timers (struct task_struct *task); +bool posix_cpu_timers_running (struct task_struct *tsk);void posix_cpu_timers_exit (struct task_struct *task);void posix_cpu_timers_exit_group (struct task_struct *task);
kernel/posix-cpu-timers.c +12 -0
@@ -6,6 +6,7 @@#include <linux/posix-timers.h> #include <linux/errno.h> #include <linux/math64.h> +#include <linux/cpuset.h> #include <asm/uaccess.h> #include <linux/kernel_stat.h> #include <trace/events/timer.h>
@@ -1300,6 +1301,17 @@ static inline int fastpath_timer_check(struct task_struct *tsk)return 0 ; } +bool posix_cpu_timers_running (struct task_struct *tsk) +{ +if (!task_cputime_zero (&tsk->cputime_expires)) +return true; + +if (tsk->signal->cputimer.running) +return true; + +return false; +} +/* * This is called from the timer interrupt handler. The irq handler has * already updated our counts. We need to check if any timers fire now.
kernel/sched.c +4 -0
@@ -71,6 +71,7 @@#include <linux/ctype.h> #include <linux/ftrace.h> #include <linux/slab.h> +#include <linux/posix-timers.h> #include <asm/tlb.h> #include <asm/irq_regs.h>
@@ -2491,6 +2492,9 @@ bool cpuset_nohz_can_stop_tick(void)if (rcu_pending (cpu))return false; +if (posix_cpu_timers_running (current)) +return false; +return true; }
Signature · 1 line
1.7.5.4
Re: [PATCH 18/32] nohz/cpuset: Don't stop the tick if posix cpu timers are running
Quoted 7 lines · Frederic Weisbecker
If either a per thread or a per process posix cpu timer is running, don't stop the tick.
TODO: restart the tick if it is stopped and a posix cpu timer is enqueued. Check we probably need a memory barrier for the per process posix timer that can be enqueued from another task of the group.
it would
Quoted 10 lines
+++
b/kernel/sched.ckernel/sched.c +1
-0@@ -71,6 +71,7 @@#include <linux/ctype.h> #include <linux/ftrace.h> #include <linux/slab.h> +#include <linux/posix-timers.h> #include <asm/tlb.h> #include <asm/irq_regs.h>
kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2499
Quoted 8 lines
kernel/sched.c +3
-0@@ -2491,6 +2492,9 @@ bool cpuset_nohz_can_stop_tick(void)if (rcu_pending (cpu))return false; +if (posix_cpu_timers_running (current)) +return false; +return true; }
Doesn't belong here, go poke at tick_nohz_can_stop_tick().
PATCH 19/32 nohz/cpuset: Restart tick when nohz flag is cleared on cpuset
In reply to Frederic WeisbeckerIssue an IPI to restart the tick on a CPU that belongs to a cpuset when its nohz flag gets cleared.
--- include/linux/cpuset.h | 1 + kernel/cpuset.c | 21 +++++++++++++++++++++ kernel/sched.c | 8 ++++++++ 3 files changed, 30 insertions(+), 0 deletions(-)
include/linux/cpuset.h +1 -0
@@ -265,6 +265,7 @@ static inline bool cpuset_adaptive_nohz(void) }extern void cpuset_update_nohz (void ); +extern void cpuset_exit_nohz_interrupt (void *unused);#else static inline void cpuset_update_nohz (void ) { }
kernel/cpuset.c +21 -0
@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (int, cpu_adaptive_nohz_ref); +static void cpu_exit_nohz (int cpu) +{ +preempt_disable (); +smp_call_function_single (cpu, cpuset_exit_nohz_interrupt, +NULL , true); +preempt_enable (); +} +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) {int cpu;
@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)per_cpu (cpu_adaptive_nohz_ref, cpu) +=1 ; else per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1; + + val =per_cpu (cpu_adaptive_nohz_ref, cpu); + +if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +*/ +smp_mb (); +cpu_exit_nohz (cpu); + } } }#else
kernel/sched.c +8 -0
@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)cpuset_nohz_restart_tick (); } +void cpuset_exit_nohz_interrupt (void *unused) +{ +if (!tick_nohz_adaptive_mode ()) +return ; + +cpuset_nohz_restart_tick (); +} +static void cpuset_nohz_task_switch (struct task_struct *next) {int cpu =smp_processor_id ();
Signature · 1 line
1.7.5.4
Re: [PATCH 19/32] nohz/cpuset: Restart tick when nohz flag is cleared on cpuset
Quoted 41 lines · Frederic Weisbecker
+++
b/kernel/cpuset.ckernel/cpuset.c +21
-0@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (int, cpu_adaptive_nohz_ref); +static void cpu_exit_nohz (int cpu) +{ +preempt_disable (); +smp_call_function_single (cpu, cpuset_exit_nohz_interrupt, +NULL , true); +preempt_enable (); +} +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) {int cpu;@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)per_cpu (cpu_adaptive_nohz_ref, cpu) +=1 ; else per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1; + + val =per_cpu (cpu_adaptive_nohz_ref, cpu); + +if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +*/ +smp_mb (); +cpu_exit_nohz (cpu); + } } }#else kernel/sched.c +0
-0
kernel/sched.c void cpuset_update_nohz(void) · line 2522
Quoted 10 lines
kernel/sched.c +7
-0@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)cpuset_nohz_restart_tick (); } +void cpuset_exit_nohz_interrupt (void *unused) +{ +if (!tick_nohz_adaptive_mode ()) +return ; + +cpuset_nohz_restart_tick (); +}
You do this just to annoy me, right? Why doesn't it live in cpuset.c where you use it?
Re: [PATCH 19/32] nohz/cpuset: Restart tick when nohz flag is cleared on cpuset
Quoted 45 lines · Peter Zijlstra
Quoted 44 lines · Frederic Weisbecker
+++
b/kernel/cpuset.ckernel/cpuset.c +21
-0@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (int, cpu_adaptive_nohz_ref); +static void cpu_exit_nohz (int cpu) +{ +preempt_disable (); +smp_call_function_single (cpu, cpuset_exit_nohz_interrupt, +NULL , true); +preempt_enable (); +} +static void update_nohz_cpus (struct cpuset *old_cs,struct cpuset *cs) {int cpu;@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)per_cpu (cpu_adaptive_nohz_ref, cpu) +=1 ; else per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1; + + val =per_cpu (cpu_adaptive_nohz_ref, cpu); + +if (!val) { +/* +* The update to cpu_adaptive_nohz_ref must be +* visible right away. So that once we restart the tick +* from the IPI, it won't be stopped again due to cache +* update lag. +*/ +smp_mb (); +cpu_exit_nohz (cpu); + } } }#else kernel/sched.c +0
-0@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)cpuset_nohz_restart_tick (); }
Quoted 10 lines
kernel/sched.c void cpuset_update_nohz(void) · line 2522
Quoted 7 lines
kernel/sched.c +7
-0+void cpuset_exit_nohz_interrupt (void *unused) +{ +if (!tick_nohz_adaptive_mode ()) +return ; + +cpuset_nohz_restart_tick (); +}You do this just to annoy me, right? Why doesn't it live in
cpuset.cwhere you use it?
Right, I'll move all those things and fix the naming / namespace.
PATCH 20/32 nohz/cpuset: Restart the tick if printk needs it
In reply to Frederic WeisbeckerIf we are in nohz adaptive mode and printk is called, the tick is missing to wake up the logger. We need to restart the tick when that happens.
--- kernel/printk.c | 17 ++++++++++++++++- 1 files changed, 16 insertions(+), 1 deletions(-)
kernel/printk.c +16 -1
@@ -41,6 +41,7 @@#include <linux/cpu.h> #include <linux/notifier.h> #include <linux/rculist.h> +#include <linux/tick.h> #include <asm/uaccess.h>
@@ -1220,8 +1221,22 @@ int printk_needs_cpu(int cpu)void wake_up_klogd (void ) { -if (waitqueue_active (&log_wait)) +unsigned long flags; + +if (waitqueue_active (&log_wait)) {this_cpu_write (printk_pending,1 ); +/* Make it visible from any interrupt from now */ +barrier (); +/* +* It's safe to check that even if interrupts are not disabled. +* If we enable nohz adaptive mode concurrently, we'll the +* printk_pending value and thus keep a periodic tick behaviour. +* Unless it's possible tick_nohz_adaptive_mode() reads its value +* before the barrier() ? +*/ +if (tick_nohz_adaptive_mode ()) +smp_cpuset_update_nohz (smp_processor_id ()); + } }/**
Signature · 1 line
1.7.5.4
PATCH 21/32 rcu: Restart the tick on non-responding adaptive nohz CPUs
In reply to Frederic WeisbeckerWhen a CPU in adaptive nohz mode doesn't respond to complete a grace period, issue it a specific IPI so that it restarts the tick and chases a quiescent state.
--- kernel/rcutree.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
kernel/rcutree.c +17 -0
@@ -50,6 +50,7 @@#include <linux/wait.h> #include <linux/kthread.h> #include <linux/prefetch.h> +#include <linux/cpuset.h> #include "rcutree.h"
@@ -295,6 +296,20 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)#ifdef CONFIG_SMP +static void cpuset_update_rcu_cpu (int cpu) +{ +#ifdef CONFIG_CPUSETS_NO_HZ +unsigned long flags; + +local_irq_save (flags); + +if (cpuset_cpu_adaptive_nohz (cpu)) +smp_cpuset_update_nohz (cpu); + +local_irq_restore (flags); +#endif +} +/* * If the specified CPU is offline, tell the caller that it is in * a quiescent state. Otherwise, whack it with a reschedule IPI.
@@ -317,6 +332,8 @@ static int rcu_implicit_offline_qs(struct rcu_data *rdp)return 1 ; } +cpuset_update_rcu_cpu (rdp->cpu); +/* If preemptible RCU, no point in sending reschedule IPI. */ if (rdp->preemptible)return 0 ;
Signature · 1 line
1.7.5.4
PATCH 22/32 rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU
In reply to Frederic WeisbeckerIf we enqueue an rcu callback, we need the CPU tick to stay alive until we take care of those by completing the appropriate grace period.
Thus, when we call_rcu(), send a self IPI that checks rcu_needs_cpu() so that we restore a periodic tick behaviour that can take care of everything.
--- kernel/rcutree.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
kernel/rcutree.c +8 -0
@@ -51,6 +51,7 @@#include <linux/kthread.h> #include <linux/prefetch.h> #include <linux/cpuset.h> +#include <linux/tick.h> #include "rcutree.h"
@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), rdp->nxttail[RCU_NEXT_TAIL ] = &head->next; rdp->qlen++; +/* Restart the timer if needed to handle the callbacks */ +if (tick_nohz_adaptive_mode ()) { +/* Make updates on nxtlist visible to self IPI */ +barrier (); +smp_cpuset_update_nohz (smp_processor_id ()); + } +/* If interrupts were disabled, don't dive into RCU core. */ if (irqs_disabled_flags (flags)) {local_irq_restore (flags);
Signature · 1 line
1.7.5.4
Re: [PATCH 22/32] rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU
Quoted 7 lines · Frederic Weisbecker
If we enqueue an rcu callback, we need the CPU tick to stay alive until we take care of those by completing the appropriate grace period.
Thus, when we
call_rcu(), send a self IPI that checksrcu_needs_cpu()so that we restore a periodic tick behaviour that can take care of everything.
One question below.
Thanx, Paul
Quoted 29 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Anton Blanchard <[email protected]>Cc Avi Kivity <[email protected]>Cc Ingo Molnar <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Paul E . McKenney <[email protected]>Cc Paul Menage <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Tim Pepper <[email protected]>--- kernel/rcutree.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)kernel/rcutree.c +1
-0@@ -51,6 +51,7 @@#include <linux/kthread.h> #include <linux/prefetch.h> #include <linux/cpuset.h> +#include <linux/tick.h> #include "rcutree.h"
kernel/rcutree.c __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · line 1556
Quoted 10 lines
kernel/rcutree.c +7
-0@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), rdp->nxttail[RCU_NEXT_TAIL ] = &head->next; rdp->qlen++; +/* Restart the timer if needed to handle the callbacks */ +if (tick_nohz_adaptive_mode ()) { +/* Make updates on nxtlist visible to self IPI */ +barrier (); +smp_cpuset_update_nohz (smp_processor_id ()); + } +
But this must be happening in a system call or interrupt handler, right? If so, won't we get a chance to check things on exit from the system call or interrupt? Or are you hooking only into syscall entry?
Quoted 6 lines
kernel/rcutree.c +0
-0/* If interrupts were disabled, don't dive into RCU core. */ if (irqs_disabled_flags (flags)) {local_irq_restore (flags);Signature · 1 line
1.7.5.4
Re: [PATCH 22/32] rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU
Quoted 4 lines · Paul E. McKenney
Quoted 4 lines
kernel/rcutree.c +0
-0@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), rdp->nxttail[RCU_NEXT_TAIL ] = &head->next; rdp->qlen++;
Quoted 11 lines
kernel/rcutree.c __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · line 1556
Quoted 7 lines
kernel/rcutree.c +7
-0+/* Restart the timer if needed to handle the callbacks */ +if (tick_nohz_adaptive_mode ()) { +/* Make updates on nxtlist visible to self IPI */ +barrier (); +smp_cpuset_update_nohz (smp_processor_id ()); + } +But this must be happening in a system call or interrupt handler, right? If so, won't we get a chance to check things on exit from the system call or interrupt? Or are you hooking only into syscall entry?
Sure in theory when we call call_rcu() we are in the kernel and thus not very far from a syscall/exception/irq exit into userspace or a scheduling out.
We could rely on that assumption and I bet it won't ever be a problem in practice. I just wanted to ensure we don't spend too much time before handling that callback.
Quoted 6 lines
Quoted 6 lines
kernel/rcutree.c +0
-0/* If interrupts were disabled, don't dive into RCU core. */ if (irqs_disabled_flags (flags)) {local_irq_restore (flags);Signature · 1 line
1.7.5.4
PATCH 23/32 nohz/cpuset: Account user and system times in adaptive nohz mode
In reply to Frederic WeisbeckerIf we are not running the tick, we are not anymore regularly counting the cputime at every jiffies.
Lay the ground to count that cputime from the points that require it. Start by catching up from timer interrupts and when we schedule out a process. We record the last jiffies and from which ring we saved it and compute the difference later when we can catch up.
For now it assumes we haven't switched to another ring while we were running nohz.
TODO: wrap operation on jiffies?
--- include/linux/kernel_stat.h | 2 ++ include/linux/tick.h | 11 +++++++++++ kernel/sched.c | 23 +++++++++++++++++++++++ kernel/time/tick-sched.c | 39 +++++++++++++++++++++++++++++++++++++++ kernel/timer.c | 6 ++++-- 5 files changed, 79 insertions(+), 2 deletions(-)
include/linux/kernel_stat.h +2 -0
@@ -114,7 +114,9 @@ static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)extern unsigned long long task_delta_exec (struct task_struct *);extern void account_user_time (struct task_struct *,cputime_t ,cputime_t ); +extern void account_user_jiffies (struct task_struct *,unsigned long );extern void account_system_time (struct task_struct *,int ,cputime_t ,cputime_t ); +extern void account_system_jiffies (struct task_struct *,unsigned long );extern void account_steal_time (cputime_t );extern void account_idle_time (cputime_t );
include/linux/tick.h +11 -0
@@ -26,6 +26,12 @@ enum tick_nohz_mode { NOHZ_MODE_HIGHRES, }; +enum tick_saved_jiffies { +JIFFIES_SAVED_NONE , +JIFFIES_SAVED_USER , +JIFFIES_SAVED_SYS , +}; +/** * struct tick_sched - sched tick emulation and no idle tick control/stats * @sched_timer: hrtimer to schedule the periodic tick in high
@@ -60,6 +66,8 @@ struct tick_sched {ktime_t idle_waketime;ktime_t idle_exittime;ktime_t idle_sleeptime; +enum tick_saved_jiffies saved_jiffies_whence; +unsigned long saved_jiffies;ktime_t iowait_sleeptime;ktime_t sleep_length;unsigned long last_jiffies;
@@ -132,8 +140,11 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);DECLARE_PER_CPU (int, task_nohz_mode);extern int tick_nohz_adaptive_mode (void ); +extern bool tick_nohz_account_tick (void ); +extern void tick_nohz_flush_current_times (void );#else /* !CPUSETS_NO_HZ */ static inline int tick_nohz_adaptive_mode (void ) {return 0 ; } +static inline bool tick_nohz_account_tick (void ) {return false; }#endif /* CPUSETS_NO_HZ */ # else /* !NO_HZ */
kernel/sched.c +23 -0
@@ -2500,6 +2500,7 @@ bool cpuset_nohz_can_stop_tick(void)static void cpuset_nohz_restart_tick (void ) { +tick_nohz_flush_current_times ();__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick (); }
@@ -3838,6 +3839,17 @@ void account_user_time(struct task_struct *p, cputime_t cputime,acct_update_integrals (p); } +void account_user_jiffies (struct task_struct *p,unsigned long count) +{ +cputime_t delta_cputime, delta_scaled; + +if (count) { + delta_cputime =jiffies_to_cputime (count); + delta_scaled =cputime_to_scaled (count); +account_user_time (p, delta_cputime, delta_scaled); + } +} +/* * Account guest cpu time to a process. * @p: the process that the cpu time gets accounted to
@@ -3922,6 +3934,17 @@ void account_system_time(struct task_struct *p, int hardirq_offset,__account_system_time (p, cputime, cputime_scaled, target_cputime64); } +void account_system_jiffies (struct task_struct *p,unsigned long count) +{ +cputime_t delta_cputime, delta_scaled; + +if (count) { + delta_cputime =jiffies_to_cputime (count); + delta_scaled =cputime_to_scaled (count); +account_system_time (p,0 , delta_cputime, delta_scaled); + } +} +/* * Account for involuntary wait time. * @cputime: the cpu time spent in involuntary wait
kernel/time/tick-sched.c +39 -0
@@ -764,6 +764,8 @@ int tick_nohz_adaptive_mode(void)static void tick_nohz_cpuset_stop_tick (int user) { +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +if (!cpuset_adaptive_nohz () ||tick_nohz_adaptive_mode ())return ;
@@ -771,6 +773,13 @@ static void tick_nohz_cpuset_stop_tick(int user)__get_cpu_var (task_nohz_mode) =1 ;/* Nohz mode must be visible to wake_up_nohz_cpu() */ smp_wmb (); + +WARN_ON_ONCE (ts->saved_jiffies_whence !=JIFFIES_SAVED_NONE ); + ts->saved_jiffies = jiffies; +if (user) + ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; +else + ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; } }
@@ -792,6 +801,36 @@ static void tick_do_timer_check_handler(int cpu) } } +bool tick_nohz_account_tick (void ) +{ +struct tick_sched *ts; +unsigned long delta_jiffies; + +if (!tick_nohz_adaptive_mode ()) +return false; + + ts = &__get_cpu_var (tick_cpu_sched); + + delta_jiffies = jiffies - ts->saved_jiffies; +if (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS ) +account_system_jiffies (current, delta_jiffies); +else +account_user_jiffies (current, delta_jiffies); + + ts->saved_jiffies = jiffies; + +return true; +} + +void tick_nohz_flush_current_times (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +tick_nohz_account_tick (); + + ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; +} +#else static void tick_nohz_cpuset_stop_tick (int user) { }
kernel/timer.c +4 -2
@@ -1288,8 +1288,10 @@ void update_process_times(int user_tick)struct task_struct *p = current;int cpu =smp_processor_id (); -/* Note: this timer irq context must be accounted for as well. */ -account_process_tick (p, user_tick); +if (!tick_nohz_account_tick ()) { +/* Note: this timer irq context must be accounted for as well. */ +account_process_tick (p, user_tick); + }run_local_timers ();rcu_check_callbacks (cpu, user_tick);printk_tick ();
Signature · 1 line
1.7.5.4
PATCH 24/32 nohz/cpuset: Handle kernel entry/exit to account cputime
In reply to Frederic WeisbeckerProvide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.
The new APIs are:
- tick_nohz_enter_kernel() (called when we enter a syscall)
- tick_nohz_exit_kernel() (called when we exit a syscall)
- tick_nohz_enter_exception() (called when we enter any exception, trap, faults...but not irqs)
- tick_nohz_exit_exception() (called when we exit any exception)
Hooks into syscalls are typically driven by the TIF_NOHZ thread flag.
In addition, we use the value returned by user_mode(regs) from the timer interrupt to know where we are. Nonetheless, we can rely on user_mode(regs) != 0 to know we are in userspace, but we can't rely on user_mode(regs) == 0 to know we are in the system.
Consider the following scenario: we stop the tick after syscall return, so we set TIF_NOHZ but the syscall exit hook is behind us. If we haven't yet returned to userspace, then we have user_mode(regs) == 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.
The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with user_mode(regs) == 1 fires. Kernel threads don't have this worry.
This sucks but for now I have no better solution. Let's hope we can find better.
TODO: wrap operation on jiffies?
--- include/linux/tick.h | 8 +++ kernel/sched.c | 1 + kernel/time/tick-sched.c | 114 ++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 109 insertions(+), 14 deletions(-)
include/linux/tick.h +8 -0
@@ -139,10 +139,18 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);#ifdef CONFIG_CPUSETS_NO_HZ DECLARE_PER_CPU (int, task_nohz_mode); +extern void tick_nohz_enter_kernel (void ); +extern void tick_nohz_exit_kernel (void ); +extern void tick_nohz_enter_exception (struct pt_regs *regs); +extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (void );#else /* !CPUSETS_NO_HZ */ +static inline void tick_nohz_enter_kernel (void ) { } +static inline void tick_nohz_exit_kernel (void ) { } +static inline void tick_nohz_enter_exception (struct pt_regs *regs) { } +static inline void tick_nohz_exit_exception (struct pt_regs *regs) { }static inline int tick_nohz_adaptive_mode (void ) {return 0 ; }static inline bool tick_nohz_account_tick (void ) {return false; }#endif /* CPUSETS_NO_HZ */
kernel/sched.c +1 -0
@@ -2503,6 +2503,7 @@ static void cpuset_nohz_restart_tick(void)tick_nohz_flush_current_times ();__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick (); +clear_thread_flag (TIF_NOHZ ); }void cpuset_update_nohz (void )
kernel/time/tick-sched.c +100 -14
164 lines not on this page · raw message
Signature · 1 line
1.7.5.4
Re: [PATCH 24/32] nohz/cpuset: Handle kernel entry/exit to account cputime
Quoted 35 lines · Frederic Weisbecker
Provide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.
The new APIs are:
-
tick_nohz_enter_kernel()(called when we enter a syscall)
-tick_nohz_exit_kernel()(called when we exit a syscall)
-tick_nohz_enter_exception()(called when we enter any exception, trap, faults...but not irqs)
-tick_nohz_exit_exception()(called when we exit any exception)Hooks into syscalls are typically driven by the
TIF_NOHZthread flag.In addition, we use the value returned by
user_mode(regs)from the timer interrupt to know where we are. Nonetheless, we can rely onuser_mode(regs)!= 0 to know we are in userspace, but we can't rely onuser_mode(regs)== 0 to know we are in the system.Consider the following scenario: we stop the tick after syscall return, so we set
TIF_NOHZbut the syscall exit hook is behind us. If we haven't yet returned to userspace, then we haveuser_mode(regs)== 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with
user_mode(regs)== 1 fires. Kernel threads don't have this worry.
Quoted 4 lines
This sucks but for now I have no better solution. Let's hope we can find better.
TODO: wrap operation on jiffies?
Hmmm... Does the RCU dyntick-idle code need to know about exception entry and exit?
Thanx, Paul
Quoted 230 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Anton Blanchard <[email protected]>Cc Avi Kivity <[email protected]>Cc Ingo Molnar <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Paul E . McKenney <[email protected]>Cc Paul Menage <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Tim Pepper <[email protected]>--- include/linux/tick.h | 8 +++ kernel/sched.c | 1 + kernel/time/tick-sched.c | 114 ++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 109 insertions(+), 14 deletions(-)include/linux/tick.h +8
-0@@ -139,10 +139,18 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);#ifdef CONFIG_CPUSETS_NO_HZ DECLARE_PER_CPU (int, task_nohz_mode); +extern void tick_nohz_enter_kernel (void ); +extern void tick_nohz_exit_kernel (void ); +extern void tick_nohz_enter_exception (struct pt_regs *regs); +extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (void );#else /* !CPUSETS_NO_HZ */ +static inline void tick_nohz_enter_kernel (void ) { } +static inline void tick_nohz_exit_kernel (void ) { } +static inline void tick_nohz_enter_exception (struct pt_regs *regs) { } +static inline void tick_nohz_exit_exception (struct pt_regs *regs) { }static inline int tick_nohz_adaptive_mode (void ) {return 0 ; }static inline bool tick_nohz_account_tick (void ) {return false; }#endif /* CPUSETS_NO_HZ */ kernel/sched.c +1
-0@@ -2503,6 +2503,7 @@ static void cpuset_nohz_restart_tick(void)tick_nohz_flush_current_times ();__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick (); +clear_thread_flag (TIF_NOHZ ); }void cpuset_update_nohz (void )kernel/time/tick-sched.c +100
-14@@ -595,8 +595,9 @@ void tick_nohz_irq_exit(void)if (ts->inidle && !need_resched ())__tick_nohz_enter_idle (ts, cpu);else if (tick_nohz_adaptive_mode () && !idle_cpu (cpu)) { -if (tick_nohz_can_stop_tick (cpu, ts)) -tick_nohz_stop_sched_tick (ktime_get (), cpu, ts); +if (ts->saved_jiffies_whence !=JIFFIES_SAVED_NONE + &&tick_nohz_can_stop_tick (cpu, ts)) +tick_nohz_stop_sched_tick (ktime_get (), cpu, ts); } }@@ -757,6 +758,74 @@ void tick_check_idle(int cpu)#ifdef CONFIG_CPUSETS_NO_HZ +void tick_nohz_exit_kernel (void ) +{ +unsigned long flags; +struct tick_sched *ts; +unsigned long delta_jiffies; + +local_irq_save (flags); + +if (!tick_nohz_adaptive_mode ()) { +local_irq_restore (flags); +return ; + } + + ts = &__get_cpu_var (tick_cpu_sched); + +WARN_ON_ONCE (ts->saved_jiffies_whence ==JIFFIES_SAVED_USER ); + +if (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS ) { + delta_jiffies = jiffies - ts->saved_jiffies; +account_system_jiffies (current, delta_jiffies); + } + + ts->saved_jiffies = jiffies; + ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; + +local_irq_restore (flags); +} + +void tick_nohz_enter_kernel (void ) +{ +unsigned long flags; +struct tick_sched *ts; +unsigned long delta_jiffies; + +local_irq_save (flags); + +if (!tick_nohz_adaptive_mode ()) { +local_irq_restore (flags); +return ; + } + + ts = &__get_cpu_var (tick_cpu_sched); + +WARN_ON_ONCE (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS ); + +if (ts->saved_jiffies_whence ==JIFFIES_SAVED_USER ) { + delta_jiffies = jiffies - ts->saved_jiffies; +account_user_jiffies (current, delta_jiffies); + } + + ts->saved_jiffies = jiffies; + ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; + +local_irq_restore (flags); +} + +void tick_nohz_enter_exception (struct pt_regs *regs) +{ +if (user_mode (regs)) +tick_nohz_enter_kernel (); +} + +void tick_nohz_exit_exception (struct pt_regs *regs) +{ +if (user_mode (regs)) +tick_nohz_exit_kernel (); +} +int tick_nohz_adaptive_mode (void ) {return __get_cpu_var (task_nohz_mode);@@ -766,20 +835,33 @@ static void tick_nohz_cpuset_stop_tick(int user) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); -if (!cpuset_adaptive_nohz () ||tick_nohz_adaptive_mode ()) +if (!cpuset_adaptive_nohz ())return ; +if (tick_nohz_adaptive_mode ()) { +if (user && ts->saved_jiffies_whence ==JIFFIES_SAVED_NONE ) { + ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; + ts->saved_jiffies = jiffies; + } + +return ; + } +if (cpuset_nohz_can_stop_tick ()) {__get_cpu_var (task_nohz_mode) =1 ;/* Nohz mode must be visible to wake_up_nohz_cpu() */ smp_wmb (); +set_thread_flag (TIF_NOHZ );WARN_ON_ONCE (ts->saved_jiffies_whence !=JIFFIES_SAVED_NONE ); - ts->saved_jiffies = jiffies; -if (user) + +if (user) { ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; -else + ts->saved_jiffies = jiffies; + }else if (!current->mm) { ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; + ts->saved_jiffies = jiffies; + } } }@@ -803,7 +885,7 @@ static void tick_do_timer_check_handler(int cpu)bool tick_nohz_account_tick (void ) { -struct tick_sched *ts; +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);unsigned long delta_jiffies;if (!tick_nohz_adaptive_mode ())@@ -811,11 +893,15 @@ bool tick_nohz_account_tick(void) ts = &__get_cpu_var (tick_cpu_sched); +if (ts->saved_jiffies_whence ==JIFFIES_SAVED_NONE ) +return false; + delta_jiffies = jiffies - ts->saved_jiffies; -if (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS ) -account_system_jiffies (current, delta_jiffies); -else + +if (ts->saved_jiffies_whence ==JIFFIES_SAVED_USER )account_user_jiffies (current, delta_jiffies); +else +account_system_jiffies (current, delta_jiffies); ts->saved_jiffies = jiffies;@@ -825,12 +911,12 @@ bool tick_nohz_account_tick(void)void tick_nohz_flush_current_times (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +unsigned long delta_jiffies; +struct pt_regs *regs; -tick_nohz_account_tick (); - - ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; +if (tick_nohz_account_tick ()) + ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; } -#else static void tick_nohz_cpuset_stop_tick (int user) { }Signature · 1 line
1.7.5.4
Re: [PATCH 24/32] nohz/cpuset: Handle kernel entry/exit to account cputime
Quoted 39 lines · Paul E. McKenney
Quoted 38 lines · Frederic Weisbecker
Provide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.
The new APIs are:
-
tick_nohz_enter_kernel()(called when we enter a syscall)
-tick_nohz_exit_kernel()(called when we exit a syscall)
-tick_nohz_enter_exception()(called when we enter any exception, trap, faults...but not irqs)
-tick_nohz_exit_exception()(called when we exit any exception)Hooks into syscalls are typically driven by the
TIF_NOHZthread flag.In addition, we use the value returned by
user_mode(regs)from the timer interrupt to know where we are. Nonetheless, we can rely onuser_mode(regs)!= 0 to know we are in userspace, but we can't rely onuser_mode(regs)== 0 to know we are in the system.Consider the following scenario: we stop the tick after syscall return, so we set
TIF_NOHZbut the syscall exit hook is behind us. If we haven't yet returned to userspace, then we haveuser_mode(regs)== 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with
user_mode(regs)== 1 fires. Kernel threads don't have this worry.This sucks but for now I have no better solution. Let's hope we can find better.
Quoted 6 lines
Quoted 1 line
TODO: wrap operation on jiffies?
Hmmm... Does the RCU dyntick-idle code need to know about exception entry and exit?
Thanx, Paul
At that time it doesn't because we don't yet call rcu_enter_nohz() when switching to userspace. Instead we shutdown the tick and restart it when needed when a remote CPU sends us an IPI to complete a grace period.
The patch that switches to extended qs is the 31/32 and it handles syscalls and exceptions as well.
I wanted to have support on rcu extended quiescent states late in the patchset so that it's considered as an incremental feature and not a core piece of the adaptive nohz (ie: it's no mandatory thing, just an optimization). This way we can use cpuset nohz without that rcu extended quiescent state feature and hence make that small part bisectable.
Patch 30 activates support for cpuset nohz (support from x86). Patch 31 activates the rcu extended quiescent state support in userspace as a bonus.
PATCH 25/32 nohz/cpuset: New API to flush cputimes on nohz cpusets
In reply to Frederic WeisbeckerProvide a new API that sends an IPI to every CPUs included in nohz cpusets in order to flush their cputimes. It's going to be useful for those that want to see accurate cputimes on a nohz cpuset.
--- include/linux/cpuset.h | 2 ++ include/linux/tick.h | 2 +- kernel/cpuset.c | 34 +++++++++++++++++++++++++++++++++- kernel/sched.c | 2 +- kernel/time/tick-sched.c | 4 ++-- 5 files changed, 39 insertions(+), 5 deletions(-)
include/linux/cpuset.h +2 -0
@@ -265,9 +265,11 @@ static inline bool cpuset_adaptive_nohz(void) }extern void cpuset_update_nohz (void ); +extern void cpuset_nohz_flush_cputimes (void );extern void cpuset_exit_nohz_interrupt (void *unused);#else static inline void cpuset_update_nohz (void ) { } +static inline void cpuset_nohz_flush_cputimes (void ) { }#endif /* CONFIG_CPUSETS_NO_HZ */
include/linux/tick.h +1 -1
@@ -145,7 +145,7 @@ extern void tick_nohz_enter_exception(struct pt_regs *regs);extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void );extern bool tick_nohz_account_tick (void ); -extern void tick_nohz_flush_current_times (void ); +extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */ static inline void tick_nohz_enter_kernel (void ) { }static inline void tick_nohz_exit_kernel (void ) { }
kernel/cpuset.c +33 -1
@@ -59,6 +59,7 @@#include <linux/mutex.h> #include <linux/workqueue.h> #include <linux/cgroup.h> +#include <linux/tick.h> /* * Workqueue for cpuset related tasks.
@@ -1199,6 +1200,23 @@ static void cpuset_change_flag(struct task_struct *tsk,DEFINE_PER_CPU (int, cpu_adaptive_nohz_ref); +static cpumask_t nohz_cpuset_mask; + +static void flush_cputime_interrupt (void *unused) +{ +tick_nohz_flush_current_times (false); +} + +void cpuset_nohz_flush_cputimes (void ) +{ +preempt_disable (); +smp_call_function_many (&nohz_cpuset_mask, flush_cputime_interrupt, +NULL , true); +preempt_enable (); +/* Make the utime/stime updates visible */ +smp_mb (); +} +static void cpu_exit_nohz (int cpu) {preempt_disable ();
@@ -1223,7 +1241,15 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs) val =per_cpu (cpu_adaptive_nohz_ref, cpu); -if (!val) { +if (val ==1 ) { +cpumask_set_cpu (cpu, &nohz_cpuset_mask); +/* +* The mask update needs to be visible right away +* so that this CPU is part of the cputime IPI +* update right now. +*/ +smp_mb (); + }else if (!val) {/* * The update to cpu_adaptive_nohz_ref must be * visible right away. So that once we restart the tick
@@ -1232,6 +1258,12 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)*/ smp_mb ();cpu_exit_nohz (cpu); +/* +* Now that the tick has been restarted and cputimes +* flushed, we don't need anymore to be part of the +* cputime flush IPI. +*/ +cpumask_clear_cpu (cpu, &nohz_cpuset_mask); } } }
kernel/sched.c +1 -1
@@ -2500,7 +2500,7 @@ bool cpuset_nohz_can_stop_tick(void)static void cpuset_nohz_restart_tick (void ) { -tick_nohz_flush_current_times (); +tick_nohz_flush_current_times (true);__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ );
kernel/time/tick-sched.c +2 -2
@@ -908,13 +908,13 @@ bool tick_nohz_account_tick(void) return true; } -void tick_nohz_flush_current_times (void ) +void tick_nohz_flush_current_times (bool restart_tick) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched);unsigned long delta_jiffies;struct pt_regs *regs; -if (tick_nohz_account_tick ()) +if (tick_nohz_account_tick () && restart_tick) ts->saved_jiffies_whence =JIFFIES_SAVED_NONE ; }#else
Signature · 1 line
1.7.5.4
PATCH 26/32 nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader
In reply to Frederic WeisbeckerWhen we wait for a zombie task, flush the cputimes on nohz cpusets in case we are waiting for a group leader that has threads running in nohz CPUs. This way thread_group_times() doesn't report stale values.
<doubts>
If I understood well the code, by the time we call that thread_group_times(), we may have childs that are still running, so this is necessary. But I need to check deeper.
</doubts>
--- kernel/exit.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/exit.c +2 -0
@@ -51,6 +51,7 @@#include <trace/events/sched.h> #include <linux/hw_breakpoint.h> #include <linux/oom.h> +#include <linux/cpuset.h> #include <asm/uaccess.h> #include <asm/unistd.h>
@@ -1262,6 +1263,7 @@ static int wait_task_zombie(struct wait_opts *wo, struct task_struct *p)* group, which consolidates times for all threads in the * group including the group leader. */ +cpuset_nohz_flush_cputimes ();thread_group_times (p, &tgutime, &tgstime);spin_lock_irq (&p->real_parent->sighand->siglock); psig = p->real_parent->signal;
Signature · 1 line
1.7.5.4
PATCH 27/32 nohz/cpuset: Flush cputimes on procfs stat file read
In reply to Frederic WeisbeckerWhen we read a process's procfs stat file, we need to flush the cputimes of the tasks running in nohz cpusets in case some childs in the thread group are running there.
--- fs/proc/array.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
fs/proc/array.c +2 -0
@@ -397,6 +397,8 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns, cutime = cstime = utime = stime = cputime_zero; cgtime = gtime = cputime_zero; +/* For thread group times */ +cpuset_nohz_flush_cputimes ();if (lock_task_sighand (task, &flags)) {struct signal_struct *sig = task->signal;
Signature · 1 line
1.7.5.4
PATCH 28/32 nohz/cpuset: Flush cputimes for getrusage() and times() syscalls
In reply to Frederic WeisbeckerBoth syscalls need to iterate through the thread group to get the cputimes. As some threads of the group may be running on nohz cpuset, we need to flush the cputimes there.
--- kernel/sys.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/sys.c +6 -0
@@ -43,6 +43,7 @@#include <linux/syscalls.h> #include <linux/kprobes.h> #include <linux/user_namespace.h> +#include <linux/cpuset.h> #include <linux/kmsg_dump.h>
@@ -908,6 +909,8 @@ void do_sys_times(struct tms *tms) {cputime_t tgutime, tgstime, cutime, cstime; +cpuset_nohz_flush_cputimes (); +spin_lock_irq (¤t->sighand->siglock);thread_group_times (current, &tgutime, &tgstime); cutime = current->signal->cutime;
@@ -1536,6 +1539,9 @@ static void k_getrusage(struct task_struct *p, int who, struct rusage *r) goto out; } +/* For thread_group_times */ +cpuset_nohz_flush_cputimes (); +if (!lock_task_sighand (p, &flags))return ;
Signature · 1 line
1.7.5.4
PATCH 29/32 x86: Syscall hooks for nohz cpusets
In reply to Frederic WeisbeckerAdd syscall hooks to notify syscall entry and exit on CPUs running in adative nohz mode.
--- arch/x86/include/asm/thread_info.h | 10 +++++++--- arch/x86/kernel/ptrace.c | 10 ++++++++++ 2 files changed, 17 insertions(+), 3 deletions(-)
arch/x86/include/asm/thread_info.h +7 -3
@@ -87,6 +87,7 @@ struct thread_info {#define TIF_NOTSC 16/* TSC is not accessible in userland */ #define TIF_IA32 17/* 32bit process */ #define TIF_FORK 18/* ret_from_fork */ +#define TIF_NOHZ 19/* in nohz userspace mode */ #define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_DEBUG 21/* uses debug registers */ #define TIF_IO_BITMAP 22/* uses I/O bitmap */
@@ -110,6 +111,7 @@ struct thread_info {#define _TIF_NOTSC (1 << TIF_NOTSC)#define _TIF_IA32 (1 << TIF_IA32)#define _TIF_FORK (1 << TIF_FORK) +#define _TIF_NOHZ (1 << TIF_NOHZ)#define _TIF_DEBUG (1 << TIF_DEBUG)#define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP)#define _TIF_FREEZE (1 << TIF_FREEZE)
@@ -121,12 +123,13 @@ struct thread_info {/* work to do in syscall_trace_enter() */ #define _TIF_WORK_SYSCALL_ENTRY \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_EMU | _TIF_SYSCALL_AUDIT | \ - _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT) + _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* work to do in syscall_trace_leave() */ #define _TIF_WORK_SYSCALL_EXIT \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SINGLESTEP | \ - _TIF_SYSCALL_TRACEPOINT) + _TIF_SYSCALL_TRACEPOINT | _TIF_NOHZ)/* work to do on interrupt/exception return */ #define _TIF_WORK_MASK \
@@ -136,7 +139,8 @@ struct thread_info {/* work to do on any return to user space */ #define _TIF_ALLWORK_MASK \ - ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT) + ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* Only used for 64 bit */ #define _TIF_DO_NOTIFY_MASK \
arch/x86/kernel/ptrace.c +10 -0
@@ -21,6 +21,7 @@#include <linux/signal.h> #include <linux/perf_event.h> #include <linux/hw_breakpoint.h> +#include <linux/tick.h> #include <asm/uaccess.h> #include <asm/pgtable.h>
@@ -1385,6 +1386,9 @@ long syscall_trace_enter(struct pt_regs *regs) {long ret =0 ; +/* Notify nohz task syscall early so the rest can use rcu */ +tick_nohz_enter_kernel (); +/* * If we stepped into a sysenter/syscall insn, it trapped in * kernel mode; do_debug() cleared TF and set TIF_SINGLESTEP.
@@ -1446,4 +1450,10 @@ void syscall_trace_leave(struct pt_regs *regs) !test_thread_flag (TIF_SYSCALL_EMU );if (step ||test_thread_flag (TIF_SYSCALL_TRACE ))tracehook_report_syscall_exit (regs, step); + +/* +* Notify nohz task exit syscall at last so the rest can +* use rcu. +*/ +tick_nohz_exit_kernel (); }
Signature · 1 line
1.7.5.4
PATCH 30/32 x86: Exception hooks for nohz cpusets
In reply to Frederic WeisbeckerAdd necessary hooks to x86 exception for nohz cpusets support. It includes traps, page fault, debug exceptions, etc...
TODO: handle do_notify_resume(). Not an exception but it's executed at the end of the syscall/irq path so it's beyond our hooks but shouldn't be considered as userspace code.
--- arch/x86/Kconfig | 1 + arch/x86/kernel/traps.c | 22 +++++++++++++++------- arch/x86/mm/fault.c | 13 +++++++++++-- 3 files changed, 27 insertions(+), 9 deletions(-)
arch/x86/Kconfig +1 -0
@@ -69,6 +69,7 @@ config X86 select IRQ_FORCED_THREADING select USE_GENERIC_SMP_HELPERS if SMP select HAVE_BPF_JIT if (X86_64 && NET) + select HAVE_CPUSETS_NO_HZ config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)
arch/x86/kernel/traps.c +15 -7
@@ -26,6 +26,7 @@#include <linux/sched.h> #include <linux/timer.h> #include <linux/init.h> +#include <linux/tick.h> #include <linux/bug.h> #include <linux/nmi.h> #include <linux/mm.h>
@@ -456,24 +457,28 @@ void restart_nmi(void)/* May run on IST stack. */ dotraplinkage void __kprobes do_int3(struct pt_regs *regs, long error_code) { +tick_nohz_enter_exception (regs); +#ifdef CONFIG_KGDB_LOW_LEVEL_TRAP if (kgdb_ll_trap (DIE_INT3 ,"int3" , regs, error_code,3 ,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;#endif /* CONFIG_KGDB_LOW_LEVEL_TRAP */ #ifdef CONFIG_KPROBES if (notify_die (DIE_INT3 ,"int3" , regs, error_code,3 ,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;#else if (notify_die (DIE_TRAP ,"int3" , regs, error_code,3 ,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;#endif preempt_conditional_sti (regs);do_trap (3 ,SIGTRAP ,"int3" , regs, error_code,NULL );preempt_conditional_cli (regs); +exit: +tick_nohz_exit_exception (regs); }#ifdef CONFIG_X86_64
@@ -534,6 +539,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)unsigned long dr6;int si_code; +tick_nohz_enter_exception (regs); +get_debugreg (dr6,6 );/* Filter out all the reserved bits which are preset to 1 */
@@ -549,7 +556,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)/* Catch kmemcheck conditions first of all! */ if ((dr6 &DR_STEP ) &&kmemcheck_trap (regs)) -return ; + goto exit;/* DR6 may or may not be cleared by the CPU */ set_debugreg (0 ,6 );
@@ -564,7 +571,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)if (notify_die (DIE_DEBUG ,"debug" , regs,PTR_ERR (&dr6), error_code,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;/* It's safe to allow irq's after DR6 has been saved */ preempt_conditional_sti (regs);
@@ -573,7 +580,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)handle_vm86_trap ((struct kernel_vm86_regs *) regs, error_code,1 );preempt_conditional_cli (regs); -return ; + goto exit; }/*
@@ -593,7 +600,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)send_sigtrap (tsk, regs, error_code, si_code);preempt_conditional_cli (regs); -return ; +exit: +tick_nohz_exit_exception (regs); }/*
arch/x86/mm/fault.c +11 -2
@@ -13,6 +13,7 @@#include <linux/perf_event.h> /* perf_sw_event */ #include <linux/hugetlb.h> /* hstate_index_to_shift */ #include <linux/prefetch.h> /* prefetchw */ +#include <linux/tick.h> #include <asm/traps.h> /* dotraplinkage, ... */ #include <asm/pgalloc.h> /* pgd_*(), ... */
@@ -971,8 +972,8 @@ static int fault_in_kernel_space(unsigned long address)* and the problem, and then passes it off to one of the appropriate * routines. */ -dotraplinkage void __kprobes -do_page_fault (struct pt_regs *regs,unsigned long error_code) +static void __kprobes +__do_page_fault (struct pt_regs *regs,unsigned long error_code) {struct vm_area_struct *vma;struct task_struct *tsk;
@@ -1180,3 +1181,11 @@ good_area:up_read (&mm->mmap_sem); } + +dotraplinkage void __kprobes +do_page_fault (struct pt_regs *regs,unsigned long error_code) +{ +tick_nohz_enter_exception (regs); +__do_page_fault (regs, error_code); +tick_nohz_exit_exception (regs); +}
Signature · 1 line
1.7.5.4
PATCH 31/32 rcu: Switch to extended quiescent state in userspace from nohz cpuset
In reply to Frederic WeisbeckerWhen we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.
However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.
To perform this, we just to use the RCU extended quiescent state APIs on the following points:
- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.
- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.
TODO: hook into do_notify_resume() because we may have called rcu_enter_nohz() from syscall exit hook, but we might call do_notify_resume() right after, which may use RCU.
--- include/linux/tick.h | 2 ++ kernel/sched.c | 1 + kernel/time/tick-sched.c | 21 +++++++++++++++++++++ 3 files changed, 24 insertions(+), 0 deletions(-)
include/linux/tick.h +2 -0
@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);#ifdef CONFIG_CPUSETS_NO_HZ DECLARE_PER_CPU (int, task_nohz_mode); +DECLARE_PER_CPU (int, nohz_task_ext_qs);extern void tick_nohz_enter_kernel (void );extern void tick_nohz_exit_kernel (void );extern void tick_nohz_enter_exception (struct pt_regs *regs);extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void ); +extern void tick_nohz_cpu_exit_qs (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */
kernel/sched.c +1 -0
@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); +tick_nohz_cpu_exit_qs (); }void cpuset_update_nohz (void )
kernel/time/tick-sched.c +21 -0
@@ -757,6 +757,7 @@ void tick_check_idle(int cpu) }#ifdef CONFIG_CPUSETS_NO_HZ +DEFINE_PER_CPU (int, nohz_task_ext_qs);void tick_nohz_exit_kernel (void ) {
@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void) ts->saved_jiffies = jiffies; ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz (); +local_irq_restore (flags); }
@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)return ; } +if (__get_cpu_var (nohz_task_ext_qs) ==1 ) { +__get_cpu_var (nohz_task_ext_qs) =0 ; +rcu_exit_nohz (); + } + ts = &__get_cpu_var (tick_cpu_sched);WARN_ON_ONCE (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS );
@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)local_irq_restore (flags); } +void tick_nohz_cpu_exit_qs (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (__get_cpu_var (nohz_task_ext_qs)) { +rcu_exit_nohz (); +__get_cpu_var (nohz_task_ext_qs) =0 ; + } +} +void tick_nohz_enter_exception (struct pt_regs *regs) {if (user_mode (regs))
@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)if (user) { ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; ts->saved_jiffies = jiffies; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz (); }else if (!current->mm) { ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; ts->saved_jiffies = jiffies;
Signature · 1 line
1.7.5.4
Re: [PATCH 31/32] rcu: Switch to extended quiescent state in userspace from nohz cpuset
Quoted 17 lines · Frederic Weisbecker
When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.
However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.
To perform this, we just to use the RCU extended quiescent state APIs on the following points:
- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.
Quoted 8 lines
- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.
TODO: hook into
do_notify_resume()because we may have calledrcu_enter_nohz()from syscall exit hook, but we might calldo_notify_resume()right after, which may use RCU.
I don't see exactly how the exception path works, but this does reassure me a bit on the syscall path.
Thanx, Paul
Quoted 62 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Anton Blanchard <[email protected]>Cc Avi Kivity <[email protected]>Cc Ingo Molnar <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Paul E . McKenney <[email protected]>Cc Paul Menage <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Tim Pepper <[email protected]>--- include/linux/tick.h | 2 ++ kernel/sched.c | 1 + kernel/time/tick-sched.c | 21 +++++++++++++++++++++ 3 files changed, 24 insertions(+), 0 deletions(-)include/linux/tick.h +2
-0@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);#ifdef CONFIG_CPUSETS_NO_HZ DECLARE_PER_CPU (int, task_nohz_mode); +DECLARE_PER_CPU (int, nohz_task_ext_qs);extern void tick_nohz_enter_kernel (void );extern void tick_nohz_exit_kernel (void );extern void tick_nohz_enter_exception (struct pt_regs *regs);extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void ); +extern void tick_nohz_cpu_exit_qs (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */ kernel/sched.c +1
-0@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); +tick_nohz_cpu_exit_qs (); }void cpuset_update_nohz (void )kernel/time/tick-sched.c +1
-0@@ -757,6 +757,7 @@ void tick_check_idle(int cpu) }#ifdef CONFIG_CPUSETS_NO_HZ +DEFINE_PER_CPU (int, nohz_task_ext_qs);void tick_nohz_exit_kernel (void ) {
kernel/time/tick-sched.c void tick_nohz_exit_kernel(void) · line 788
Quoted 5 lines
kernel/time/tick-sched.c +2
-0@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void) ts->saved_jiffies = jiffies; ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz ();
OK, I was wondering how this was going to work if RCU didn't know about kernel entry/exit. Whew!!! ;-)
Quoted 34 lines
kernel/time/tick-sched.c +16
-0+local_irq_restore (flags); }@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)return ; } +if (__get_cpu_var (nohz_task_ext_qs) ==1 ) { +__get_cpu_var (nohz_task_ext_qs) =0 ; +rcu_exit_nohz (); + } + ts = &__get_cpu_var (tick_cpu_sched);WARN_ON_ONCE (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS );@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)local_irq_restore (flags); } +void tick_nohz_cpu_exit_qs (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (__get_cpu_var (nohz_task_ext_qs)) { +rcu_exit_nohz (); +__get_cpu_var (nohz_task_ext_qs) =0 ; + } +} +void tick_nohz_enter_exception (struct pt_regs *regs) {if (user_mode (regs))
kernel/time/tick-sched.c static void tick_nohz_cpuset_stop_tick(int user) · line 881
Quoted 5 lines
kernel/time/tick-sched.c +2
-0@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)if (user) { ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; ts->saved_jiffies = jiffies; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz ();
When entering an exception, shouldn't we call rcu_exit_nohz() rather than rcu_exit_nohz()? Or is this a "didn't really mean an exception" code path?
Quoted 11 lines
kernel/time/tick-sched.c +0
-0}else if (!current->mm) { ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; ts->saved_jiffies = jiffies;Signature · 7 lines
1.7.5.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 31/32] rcu: Switch to extended quiescent state in userspace from nohz cpuset
Quoted 23 lines · Paul E. McKenney
Quoted 22 lines · Frederic Weisbecker
When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.
However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.
To perform this, we just to use the RCU extended quiescent state APIs on the following points:
- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.
- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.
Quoted 6 lines
Quoted 3 lines
TODO: hook into
do_notify_resume()because we may have calledrcu_enter_nohz()from syscall exit hook, but we might calldo_notify_resume()right after, which may use RCU.I don't see exactly how the exception path works, but this does reassure me a bit on the syscall path.
On the syscall path we directly call tick_nohz_enter,exit_kernel() and that takes care of all the rcu trickies.
The exception paths call tick_nohz_enter,exit_exception() which are essentially conditional wrappers around tick_nohz_enter,exit_kernel() after checking user_mode(regs). Hmm now I realize I can't rely on user_mode() to know if we should exit or not rcu extended quiescent state because even if user_mode(regs) != 1, we may not have yet called the syscall exit hook and thus not exited rcu quiescent state. So tick_nohz_enter_exception() may forget to call rcu_exit_nohz() sometimes.
I need to fix that.
Quoted 129 lines
Thanx, Paul
Quoted 62 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Andrew Morton <[email protected]>Cc Anton Blanchard <[email protected]>Cc Avi Kivity <[email protected]>Cc Ingo Molnar <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Paul E . McKenney <[email protected]>Cc Paul Menage <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Tim Pepper <[email protected]>--- include/linux/tick.h | 2 ++ kernel/sched.c | 1 + kernel/time/tick-sched.c | 21 +++++++++++++++++++++ 3 files changed, 24 insertions(+), 0 deletions(-)include/linux/tick.h +2
-0@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);#ifdef CONFIG_CPUSETS_NO_HZ DECLARE_PER_CPU (int, task_nohz_mode); +DECLARE_PER_CPU (int, nohz_task_ext_qs);extern void tick_nohz_enter_kernel (void );extern void tick_nohz_exit_kernel (void );extern void tick_nohz_enter_exception (struct pt_regs *regs);extern void tick_nohz_exit_exception (struct pt_regs *regs);extern int tick_nohz_adaptive_mode (void ); +extern void tick_nohz_cpu_exit_qs (void );extern bool tick_nohz_account_tick (void );extern void tick_nohz_flush_current_times (bool restart_tick);#else /* !CPUSETS_NO_HZ */ kernel/sched.c +1
-0@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)__get_cpu_var (task_nohz_mode) =0 ;tick_nohz_restart_sched_tick ();clear_thread_flag (TIF_NOHZ ); +tick_nohz_cpu_exit_qs (); }void cpuset_update_nohz (void )kernel/time/tick-sched.c +1
-0@@ -757,6 +757,7 @@ void tick_check_idle(int cpu) }#ifdef CONFIG_CPUSETS_NO_HZ +DEFINE_PER_CPU (int, nohz_task_ext_qs);void tick_nohz_exit_kernel (void ) {kernel/time/tick-sched.c void tick_nohz_exit_kernel(void) · line 788
Quoted 5 lines
kernel/time/tick-sched.c +2
-0@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void) ts->saved_jiffies = jiffies; ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz ();OK, I was wondering how this was going to work if RCU didn't know about kernel entry/exit. Whew!!! ;-)
Quoted 34 lines
kernel/time/tick-sched.c +16
-0+local_irq_restore (flags); }@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)return ; } +if (__get_cpu_var (nohz_task_ext_qs) ==1 ) { +__get_cpu_var (nohz_task_ext_qs) =0 ; +rcu_exit_nohz (); + } + ts = &__get_cpu_var (tick_cpu_sched);WARN_ON_ONCE (ts->saved_jiffies_whence ==JIFFIES_SAVED_SYS );@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)local_irq_restore (flags); } +void tick_nohz_cpu_exit_qs (void ) +{ +struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); + +if (__get_cpu_var (nohz_task_ext_qs)) { +rcu_exit_nohz (); +__get_cpu_var (nohz_task_ext_qs) =0 ; + } +} +void tick_nohz_enter_exception (struct pt_regs *regs) {if (user_mode (regs))kernel/time/tick-sched.c static void tick_nohz_cpuset_stop_tick(int user) · line 881
Quoted 5 lines
kernel/time/tick-sched.c +2
-0@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)if (user) { ts->saved_jiffies_whence =JIFFIES_SAVED_USER ; ts->saved_jiffies = jiffies; +__get_cpu_var (nohz_task_ext_qs) =1 ; +rcu_enter_nohz ();When entering an exception, shouldn't we call
rcu_exit_nohz()rather thanrcu_exit_nohz()? Or is this a "didn't really mean an exception" code path?Quoted 11 lines
kernel/time/tick-sched.c +0
-0}else if (!current->mm) { ts->saved_jiffies_whence =JIFFIES_SAVED_SYS ; ts->saved_jiffies = jiffies;Signature · 7 lines
1.7.5.4 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
PATCH 32/32 nohz/cpuset: Disable under some configs
In reply to Frederic WeisbeckerThis shows the various things that are not yet handled by the nohz cpusets: perf events, irq work, irq time accounting.
But there are further things that have yet to be handled:
sched clock tick, runqueue clock, sched_class::task_tick(), rq clock, cpu load, complete handling of cputimes, ...
--- init/Kconfig | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
init/Kconfig +1 -1
@@ -626,7 +626,7 @@ config PROC_PID_CPUSET config CPUSETS_NO_HZ bool "Tickless cpusets" - depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS + depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS && !PERF_EVENTS && !PROFILING && !IRQ_WORK && !IRQ_TIME_ACCOUNTING help This options let you apply a nohz property to a cpuset such that the periodic timer tick tries to be avoided when possible on
Signature · 1 line
1.7.5.4
Quoted 8 lines · Frederic Weisbecker
= What's the interface =
We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.
Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Frederic Weisbecker
= What's the interface =
We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.
Quoted 2 lines
Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.
Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...
But perhaps if one day it is proven to behave better in most cases then we can make it enabled by default on cpusets?
Quoted 6 lines · Frederic Weisbecker
Quoted 3 lines
Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.
Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...
I agree with Avi. I'd be inclined to investigate further to see if there are any important workloads on which it's not a win - and then add the extra complexity to control it from cpusets if necessary. Unless there's really a good reason to make it configurable, it's simpler to make it unconditional.
Paul
Quoted 5 lines · Paul Menage
Quoted 4 lines · Frederic Weisbecker
Quoted 3 lines
Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.
Quoted 8 lines
Quoted 2 lines
Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...
I agree with Avi. I'd be inclined to investigate further to see if there are any important workloads on which it's not a win - and then add the extra complexity to control it from cpusets if necessary. Unless there's really a good reason to make it configurable, it's simpler to make it unconditional.
There is another things. We still need to have a CPU with the periodic tick to maintain the jiffies and walltime progression.
Also on some workloads like HPC (I mean, that's just a personal guess), it may make sense to also migrate the peripherals interrupts affinity to that same CPU that keeps the periodic tick, so that we have only one CPU handling those backrgound things and all the others can be fully dedicated to their main task.
So with these factors combined, we need to be able to precisely choose at least a CPU that doesn't run in adaptive nohz mode. Having that nohz cpuset solves that issue (although I haven't forced to keep one non-nohz cpu but I should).
Perhaps having a simple interface that defines a fixed CPU to handle jiffies/walltime would be enough but the cpuset offers more flexibility. I just can't test every possible workload to know if it has no downside somewhere.
May be a global toggle in sysfs for a global adaptive nohz would be enough? I have no idea.
Is anybody aware of any worload that involve very frequent syscalls? Like more than 100 Hz ?
Hi,
Quoted 5 lines · Frederic Weisbecker
For those who want to play:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
nohz/cpuset-v1
You caught me in playful mood, so I took it for a spin... :-)
I know this is far from being production ready, but I hope you'll find the feedback useful.
First a short description of my testing setup is in order, I believe:
I've set up a small x86 VM with 4 CPUs running your git tree and a minimal buildroot system. I've created 2 cpusets: sys and nohz, and then assigned every task I could to the sys cpuset and set adaptive_nohz on the nohz set.
To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.
I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using /proc/interrupts
Now, for the things I've noticed -
1. Before I turn adaptive_nohz to 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turn adaptive_nohz to 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.
2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for rcu_pending() in cpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.
3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when nr_running
Quoted 1 line
1. In my case, I got a kernel panic. Since it happened with some
debug code I added and with aforementioned experimental removal of rcu_pending check, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.
Cheers,
Gilad
Signature · 9 lines
Gilad Ben-Yossef Chief Coffee Drinker [email protected] Israel Cell: +972-52-8260388 US Cell: +1-973-8260388 http://benyossef.com "Dance like no one is watching, love like you'll never be hurt, sing like no one is listening... but for BEEP sake you better code like you're going to maintain it for years!"
Quoted 22 lines · Gilad Ben-Yossef
Hi,
Quoted 5 lines · Frederic Weisbecker
For those who want to play:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
nohz/cpuset-v1You caught me in playful mood, so I took it for a spin... :-)
I know this is far from being production ready, but I hope you'll find the feedback useful.
First a short description of my testing setup is in order, I believe:
I've set up a small x86 VM with 4 CPUs running your git tree and a minimal buildroot system. I've created 2 cpusets: sys and nohz, and then assigned every task I could to the sys cpuset and set
adaptive_nohzon the nohz set.
Quoted 4 lines
To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
Quoted 5 lines
I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using/proc/interrupts
Quoted 8 lines
Now, for the things I've noticed -
1. Before I turn
adaptive_nohzto 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turnadaptive_nohzto 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.
Yeah I believe this is due to RCU that tries to wake up our nohz CPU. I need to have a deeper look there.
Quoted 10 lines
2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for
rcu_pending()incpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.
Are you sure the tick never goes off?
But yeah may be there is something that constantly requires RCU grace periods to complete in your system. I should drop the rcu_pending() check as long as we want to stop the tick from userspace because there we are off the RCU state machine.
Quoted 8 lines
3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when
nr_runningQuoted 1 line
1. In my case, I got a kernel panic. Since it happened with some
debug code I added and with aforementioned experimental removal of
rcu_pendingcheck, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.
I got some panic too but haven't seen any for some time. I made a lot of changes since then though so I thought the condition to trigger it just went away.
IIRC, it was a locking inversion against the rq lock and some other lock. Very nice condition for a cool lockup ;)
Quoted 2 lines · Frederic Weisbecker
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
It is? Is it being replaced by something? (I find it useful)
-Mike
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
It is? Is it being replaced by something? (I find it useful)
cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.
[cpuisol seems to already be broken for SCHED_FIFO tasks, since they share the def_root_domain]
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
It is? Is it being replaced by something? (I find it useful)
cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
-Mike
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
It is? Is it being replaced by something? (I find it useful)
cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
You need to play with the sched_load_balance file, no idea what this cset utility is though, never encountered it before.
Quoted 1 line
Quoted 1 line
Quoted 1 line
cpusets? Afaict all it does is not include a number of cpus in the sched
I think you meant to write cpuisol here :-)
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
You need to play with the
sched_load_balancefile, no idea what this cset utility is though, never encountered it before.
It's just a python wrapper around the /cgroup file system.
Gilad
Signature · 9 lines
Gilad Ben-Yossef Chief Coffee Drinker [email protected] Israel Cell: +972-52-8260388 US Cell: +1-973-8260388 http://benyossef.com "Dance like no one is watching, love like you'll never be hurt, sing like no one is listening... but for BEEP sake you better code like you're going to maintain it for years!"
Quoted 3 lines · Gilad Ben-Yossef
Quoted 1 line
Quoted 1 line
Quoted 1 line
cpusets? Afaict all it does is not include a number of cpus in the sched
I think you meant to write cpuisol here :-)
Uhh, yeah ;-)
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
You need to play with the
sched_load_balancefile, no idea what this cset utility is though, never encountered it before.It's just a python wrapper around the /cgroup file system.
ok, just never encountered it.
Quoted 2 lines
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
Try adding:
# cset shield --kthread on
It will try to move non bound kernel threads to the system set from the root set.
The results seems to vary from introducing system instability, via doing nothing at all, all the way to actually achieving better isolation, depending on the specific kernel threads you have running and your kernel version. Sometime more then one of the results at the same time. It's really like Russian roulette in a way, only way more geekier...
Signature · 9 lines
Gilad Ben-Yossef Chief Coffee Drinker [email protected] Israel Cell: +972-52-8260388 US Cell: +1-973-8260388 http://benyossef.com "Dance like no one is watching, love like you'll never be hurt, sing like no one is listening... but for BEEP sake you better code like you're going to maintain it for years!"
Quoted 6 lines · Gilad Ben-Yossef
Quoted 2 lines
Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.
Try adding:
# cset shield --kthread on
Yeah, did that.
Killing load balancing in the cpuset should eliminate the problem. Well, for the test app I was running anyway, which has everything pinned, and pure RT.
-Mike
Quoted 10 lines · Frederic Weisbecker
Quoted 6 lines
To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.
Ah I haven't tested with that isolcpus especially as it's headed toward removal.
I had the cpuisol option in the boot loader config but I did set up a proper cpuset as well, so I believe it made no difference.
I added the cpuisol option after noticing how many tasks I was unable to move from the root cpuset to the system cpuset due to them being bound per CPU and hoped (for no good reason, I admit) that cpuisol will somehow help with the task isolation I wanted to get to test the nohz task.
That's a different obstacle to workload of the kind where nohz cpuset would be useful, but should probably be discussed in another thread :-)
Quoted 15 lines
Quoted 6 lines
I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using/proc/interruptsQuoted 8 lines
Now, for the things I've noticed -
1. Before I turn
adaptive_nohzto 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turnadaptive_nohzto 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.
Quoted 2 lines
Yeah I believe this is due to RCU that tries to wake up our nohz CPU. I need to have a deeper look there.
I believe you are right with the reason for the IPI.
Before setting adaptive_nohz for the cpuset I did not get the IPI on an idle CPU. After setting it I started getting the IPI regularly even when the CPU was idle.
Quoted 11 lines
Quoted 10 lines
2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for
rcu_pending()incpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.
Quoted 1 line
Are you sure the tick never goes off?
Yes, I put debug code in the cpuset_nohz_can_stop_tick(). Every time the function was called for that CPU rcu_pending() returned 1.
Quoted 4 lines
But yeah may be there is something that constantly requires RCU grace periods to complete in your system. I should drop the
rcu_pending()check as long as we want to stop the tick from userspace because there we are off the RCU state machine.
I added debug code to rcu_pending() and noticed that the rcu_bh was the one pending each time.
I found that odd - my VM didn't even have a netowrk interface configured (except maybe for lo), let alone saw any network traffic. I thought rcu_bh was mostly used for networking code (Paul?)
Quoted 9 lines
Quoted 8 lines
3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when
nr_runningQuoted 1 line
1. In my case, I got a kernel panic. Since it happened with some
debug code I added and with aforementioned experimental removal of
rcu_pendingcheck, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.
Quoted 6 lines
I got some panic too but haven't seen any for some time. I made a lot of changes since then though so I thought the condition to trigger it just went away.
IIRC, it was a locking inversion against the rq lock and some other lock. Very nice condition for a cool lockup ;)
It certainly sounds exciting :-)
Let me know if I an help test anything else.
Thanks,
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"Dance like no one is watching, love like you'll never be hurt, sing like no one is listening... but for BEEP sake you better code like you're going to maintain it for years!"
Quoted 3 lines · Gilad Ben-Yossef
I added the cpuisol option after noticing how many tasks I was unable to move from the root cpuset to the system cpuset due to them being bound per CPU
Right, so ideally those tasks should be idle and not interfere. Where this is not so, we should make it so.
When userspace didn't ask for anything to happen, nothing should happen. When it did ask for it, well then it shouldn't complain it does :-)
Furthermore things like:
linux-2.6# git grep on_each_cpu mm/
mm/page_alloc.c: on_each_cpu(drain_local_pages, NULL, 1); mm/slab.c: on_each_cpu(do_drain, cachep, 1); mm/slab.c: on_each_cpu(do_ccupdate_local, (void *)new, 1); mm/slub.c: on_each_cpu(flush_cpu_slab, s, 1); mm/swap.c: return schedule_on_each_cpu(lru_add_drain_per_cpu);
Should be converted to smp_call_function_many() and for each we should keep a cpumask of cpus where there's work to do, avoiding disturbing cpus that have been quiet.