Skip to contentSkip to conversation

Nohz cpusets (was: Nohz Tasks)

RFC PATCH In review Frederic Weisbecker 9 participants 139 messages Aug 15, 2011 – Sep 9, 2011

Subsystems (17)
Arm (32-bit) x86 PowerPC MIPS +13 more subsystems
lore
Share & tools
lore
#0 lore raw

cover Nohz cpusets (was: Nohz Tasks)

So it's still in draft stage. It's far from covering everything the periodic timer does but it has made some progress since last posting. So I think it's time now for another early release.

= What's that? =

On the mainline kernel we have a feature (CONFIG_NO_HZ) that is able to turn off the periodic scheduler tick when the CPU has nothing to do, namely when it's running the idle task.

The scheduler tick handles many things like RCU and scheduler internal state, jiffies accouting, wall time accounting, load accounting, cputime accounting, timer wheel, posix cpu timers, etc...

However by the time we run idle and the CPU is going to sleep, none of these things are useful for the CPU. We can then shut it down.

The benefit of this is for energy saving purposes. We avoid to wake up the CPU needlessly with these useless interrupts.

What this patchset do is to extend that feature to non idle cases, implementing some new kind of "adaptive nohz". But the purpose is different and the implementation too.

= How does that work =

It tries to handle all the things that the timer tick usually handle but using different tricks. Sometimes we can't really afford to avoid the periodic tick, but sometimes we can and if we do, we need to take some special care.

- We can't shutdown the tick if we have more than one task running, due to the need for the tick for preemption. But I believe that one day we can avoid the periodic tick for that and rather anticipate when the scheduler really needs the tick.

- We can't shutdown the tick if RCU needs to complete a grace period from the current CPU, or if it has callbacks to handle.

- We can't shutdown the tick if we have a posix cpu timer queued. Similarly to the preemption case, we should be able to anticipate that with a precise timer and avoid a periodic check based on HZ.

- Restart the tick when more than one non-idle task are in the runqueue.

- We need to handle process accounting, RCU, rq clock, task tick, etc...

And that patchset for now only handles a part of the whole needs.

= What's the interface =

We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.

= Why do we need that? =

There are at least two potential users of this feature:

* High performance computing: To optimize the throughput, some workloads involve running one task per CPU that mostly run in userspace. These tasks don't want and don't need to suffer from the overhead of the timer interrupt. It consumes CPU time and it trashes the CPU cache.

* Real time: Minimizing timer interrupts means less interrupts and thus less critical sections that usually induce latency.

= What's missing? =

Many things like handling of perf events, irq work, sched clock tick, runqueue clock, sched_class::task_tick(), rq clock, cpu load, ...

The handling of cputimes is also incomplete as there are other places that use the utime/stime. Process time accounting is globally incomplete.

But anyway the thing is moving forward. An early posting was just very needed at that step.

For those who want to play:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
nohz/cpuset-v1

Frederic Weisbecker (32):
nohz: Drop useless call in tick_nohz_start_idle()
nohz: Drop ts->idle_active
nohz: Drop useless ts->inidle check before rearming the tick
nohz: Separate idle sleeping time accounting from nohz switching
nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs
nohz: Move idle ticks stats tracking out of nohz handlers
nohz: Rename ts->idle_tick to ts->last_tick
nohz: Move nohz load balancer selection into idle logic
nohz: Move ts->idle_calls into strict idle logic
nohz: Move next idle expiring time record into idle logic area
cpuset: Set up interface for nohz flag
nohz: Try not to give the timekeeping duty to a cpuset nohz cpu
nohz: Adaptive tick stop and restart on nohz cpuset nohz/cpuset: Don't turn off the tick if rcu needs it nohz/cpuset: Restart tick when switching to idle task nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued
x86: New cpuset nohz irq vector
nohz/cpuset: Don't stop the tick if posix cpu timers are running nohz/cpuset: Restart tick when nohz flag is cleared on cpuset nohz/cpuset: Restart the tick if printk needs it
rcu: Restart the tick on non-responding adaptive nohz CPUs
rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU nohz/cpuset: Account user and system times in adaptive nohz mode nohz/cpuset: Handle kernel entry/exit to account cputime nohz/cpuset: New API to flush cputimes on nohz cpusets nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader nohz/cpuset: Flush cputimes on procfs stat file read nohz/cpuset: Flush cputimes for getrusage() and times() syscalls
x86: Syscall hooks for nohz cpusets
x86: Exception hooks for nohz cpusets
rcu: Switch to extended quiescent state in userspace from nohz cpuset nohz/cpuset: Disable under some configs

 arch/Kconfig                           |    3 +
 arch/arm/kernel/process.c              |    4 +-
 arch/avr32/kernel/process.c            |    4 +-
 arch/blackfin/kernel/process.c         |    4 +-
 arch/microblaze/kernel/process.c       |    4 +-
 arch/mips/kernel/process.c             |    4 +-
 arch/powerpc/kernel/idle.c             |    4 +-
 arch/powerpc/platforms/iseries/setup.c |    8 +-
 arch/s390/kernel/process.c             |    4 +-
 arch/sh/kernel/idle.c                  |    4 +-
 arch/sparc/kernel/process_64.c         |    4 +-
 arch/tile/kernel/process.c             |    4 +-
 arch/um/kernel/process.c               |    4 +-
 arch/unicore32/kernel/process.c        |    4 +-
 arch/x86/Kconfig                       |    1 +
 arch/x86/include/asm/entry_arch.h      |    3 +
 arch/x86/include/asm/hw_irq.h          |    6 +
 arch/x86/include/asm/irq_vectors.h     |    2 +
 arch/x86/include/asm/smp.h             |   11 +
 arch/x86/include/asm/thread_info.h     |   10 +-
 arch/x86/kernel/entry_64.S             |    4 +
 arch/x86/kernel/irqinit.c              |    4 +
 arch/x86/kernel/process_32.c           |    4 +-
 arch/x86/kernel/process_64.c           |    5 +-
 arch/x86/kernel/ptrace.c               |   10 +
 arch/x86/kernel/smp.c                  |   26 ++
 arch/x86/kernel/traps.c                |   22 +-
 arch/x86/mm/fault.c                    |   13 +-
 fs/proc/array.c                        |    2 +
 include/linux/cpuset.h                 |   29 ++
 include/linux/kernel_stat.h            |    2 +
 include/linux/posix-timers.h           |    1 +
 include/linux/rcupdate.h               |    1 +
 include/linux/sched.h                  |   10 +-
 include/linux/tick.h                   |   50 +++-
 init/Kconfig                           |    8 +
 kernel/cpuset.c                        |  105 +++++++
 kernel/exit.c                          |    2 +
 kernel/posix-cpu-timers.c              |   12 +
 kernel/printk.c                        |   17 +-
 kernel/rcutree.c                       |   28 ++-
 kernel/sched.c                         |  132 +++++++++-
 kernel/softirq.c                       |    6 +-
 kernel/sys.c                           |    6 +
 kernel/time/tick-sched.c               |  479 ++++++++++++++++++++++++--------
 kernel/time/timer_list.c               |    4 +-
 kernel/timer.c                         |    8 +-
 47 files changed, 897 insertions(+), 185 deletions(-)
Signature · 1 line
1.7.5.4
#1 lore raw

PATCH RESEND 1/32 nohz: Drop useless call in tick_nohz_start_idle()

The call to update_ts_time_stats() there is useless. All we need is to save the idle entry_time.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)
kernel/time/tick-sched.c +0 -3
index d5097c4..58e1a96 100644
@@ -185,9 +185,6 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts)
 	ktime_t now;
 
 	now = ktime_get();
-
-	update_ts_time_stats(cpu, ts, now, NULL);
-
 	ts->idle_entrytime = now;
 	ts->idle_active = 1;
 	sched_clock_idle_sleep_event();
Signature · 1 line
1.7.5.4
#2 lore raw

Re: [PATCH 01/32 RESEND] nohz: Drop useless call in tick_nohz_start_idle()

Quoted 4 lines · Frederic Weisbecker

The call to update_ts_time_stats() there is useless. All we need is to save the idle entry_time.

Would have been clearer if you just said the call was a NOP. The whole second sentence distracts and confuses as its irrelevant to the change at hand.

If you want to expand you can explain that its a NOP because update_ts_time_stats() requires either ts->idle_active and/or @last_update_time and our callsite has neither.

Although this assumes its never called when ts->idle_active is already set, is this so (likely)? Do we want a WARN_ON_ONCE() testing that assumption?

#3 lore raw

Re: [PATCH 01/32 RESEND] nohz: Drop useless call in tick_nohz_start_idle()

Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

The call to update_ts_time_stats() there is useless. All we need is to save the idle entry_time.

Quoted 7 lines

Would have been clearer if you just said the call was a NOP. The whole second sentence distracts and confuses as its irrelevant to the change at hand.

If you want to expand you can explain that its a NOP because update_ts_time_stats() requires either ts->idle_active and/or @last_update_time and our callsite has neither.

Right, will fix the changelog.

Quoted 4 lines

Although this assumes its never called when ts->idle_active is already set, is this so (likely)? Do we want a WARN_ON_ONCE() testing that assumption?

Not sure. Looking at the ondemand cpufreq governor, it calls get_cpu_idle_time_us() from an initcall.

#4 lore raw

PATCH RESEND 2/32 nohz: Drop ts->idle_active

In reply to Frederic Weisbecker

ts->idle_active is used to know if we want to account the idle sleep time. But ts->inidle is enough to check that.

So drop that field and use inidle instead. This simplifies the code.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    1 -
 kernel/time/tick-sched.c |   14 ++++++--------
 2 files changed, 6 insertions(+), 9 deletions(-)
include/linux/tick.h +0 -1
index b232ccc..532e650 100644
@@ -56,7 +56,6 @@ struct tick_sched {
 	unsigned long			idle_jiffies;
 	unsigned long			idle_calls;
 	unsigned long			idle_sleeps;
-	int				idle_active;
 	ktime_t				idle_entrytime;
 	ktime_t				idle_waketime;
 	ktime_t				idle_exittime;
kernel/time/tick-sched.c +6 -8
index 58e1a96..c4d7113 100644
@@ -157,7 +157,7 @@ update_ts_time_stats(int cpu, struct tick_sched *ts, ktime_t now, u64 *last_upda
 {
 	ktime_t delta;
 
-	if (ts->idle_active) {
+	if (ts->inidle) {
 		delta = ktime_sub(now, ts->idle_entrytime);
 		ts->idle_sleeptime = ktime_add(ts->idle_sleeptime, delta);
 		if (nr_iowait_cpu(cpu) > 0)
@@ -175,7 +175,6 @@ static void tick_nohz_stop_idle(int cpu, ktime_t now)
 	struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu);
 
 	update_ts_time_stats(cpu, ts, now, NULL);
-	ts->idle_active = 0;
 
 	sched_clock_idle_wakeup_event(0);
 }
@@ -186,7 +185,6 @@ static ktime_t tick_nohz_start_idle(int cpu, struct tick_sched *ts)
 
 	now = ktime_get();
 	ts->idle_entrytime = now;
-	ts->idle_active = 1;
 	sched_clock_idle_sleep_event();
 	return now;
 }
@@ -502,11 +500,11 @@ void tick_nohz_restart_sched_tick(void)
 	ktime_t now;
 
 	local_irq_disable();
-	if (ts->idle_active || (ts->inidle && ts->tick_stopped))
-		now = ktime_get();
 
-	if (ts->idle_active)
+	if (ts->inidle) {
+		now = ktime_get();
 		tick_nohz_stop_idle(cpu, now);
+	}
 
 	if (!ts->inidle || !ts->tick_stopped) {
 		ts->inidle = 0;
@@ -677,10 +675,10 @@ static inline void tick_check_nohz(int cpu)
 	struct tick_sched *ts = &per_cpu(tick_cpu_sched, cpu);
 	ktime_t now;
 
-	if (!ts->idle_active && !ts->tick_stopped)
+	if (!ts->inidle && !ts->tick_stopped)
 		return;
 	now = ktime_get();
-	if (ts->idle_active)
+	if (ts->inidle)
 		tick_nohz_stop_idle(cpu, now);
 	if (ts->tick_stopped) {
 		tick_nohz_update_jiffies(now);
Signature · 1 line
1.7.5.4
#5 lore raw

Re: [PATCH 02/32 RESEND] nohz: Drop ts->idle_active

Quoted 3 lines · Frederic Weisbecker

ts->idle_active is used to know if we want to account the idle sleep time. But ts->inidle is enough to check that.

While possibly true, its not immediately obvious and no hints are supplied. For example: tick_check_nohz() would disable ->idle_active.. where is this mirrored in the ->inidle state.

Also, tick_nohz_stop_sched_tick() has this comment:

	/*
	 * Set ts->inidle unconditionally. Even if the system did not
	 * switch to NOHZ mode the cpu frequency governers rely on the
	 * update of the idle time accounting in tick_nohz_start_idle().
	 */
	ts->inidle = 1;

Which suggest the ->inidle state doesn't accurately reflect things.

This is all rather hairy code, such changes really want more in terms of explanation.

#6 lore raw

Re: [PATCH 02/32 RESEND] nohz: Drop ts->idle_active

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

ts->idle_active is used to know if we want to account the idle sleep time. But ts->inidle is enough to check that.

While possibly true, its not immediately obvious and no hints are supplied. For example: tick_check_nohz() would disable ->idle_active.. where is this mirrored in the ->inidle state.

Hmm, you're right. By the time we call tick_check_nohz() (irq_enter()) and tick_nohz_stop_sched_tick() (irq_exit()) there may be a softirq and then another hard irq that could update the idle time spuriously.

So the mapping inidle - idle_active is wrong.

Let's drop that patch.

Quoted 16 lines

Also, tick_nohz_stop_sched_tick() has this comment:

	/*
	 * Set ts->inidle unconditionally. Even if the system did not
	 * switch to NOHZ mode the cpu frequency governers rely on the
	 * update of the idle time accounting in tick_nohz_start_idle().
	 */
	ts->inidle = 1;

Which suggest the ->inidle state doesn't accurately reflect things.

This is all rather hairy code, such changes really want more in terms of explanation.

#7 lore raw

PATCH RESEND 3/32 nohz: Drop useless ts->inidle check before rearming the tick

In reply to Frederic Weisbecker

We only need to check if we have ts->stopped to ensure the tick was stopped and we want to re-enable it. Checking ts->inidle there is useless.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)
kernel/time/tick-sched.c +2 -4
index c4d7113..5934aee 100644
@@ -504,16 +504,14 @@ void tick_nohz_restart_sched_tick(void)
 	if (ts->inidle) {
 		now = ktime_get();
 		tick_nohz_stop_idle(cpu, now);
+		ts->inidle = 0;
 	}
 
-	if (!ts->inidle || !ts->tick_stopped) {
-		ts->inidle = 0;
+	if (!ts->tick_stopped) {
 		local_irq_enable();
 		return;
 	}
 
-	ts->inidle = 0;
-
 	rcu_exit_nohz();
 
 	/* Update jiffies first */
Signature · 1 line
1.7.5.4
#8 lore raw

Re: [PATCH 03/32 RESEND] nohz: Drop useless ts->inidle check before rearming the tick

Quoted 3 lines · Frederic Weisbecker

We only need to check if we have ts->stopped to ensure the tick was stopped and we want to re-enable it. Checking ts->inidle there is useless.

/me goes la-la-la-la...

It would so help poor little me who hasn't stared at this code in detail for the past several days and is thus horridly confused if you'd expand your reasoning somewhat.

#9 lore raw

Re: [PATCH 03/32 RESEND] nohz: Drop useless ts->inidle check before rearming the tick

Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

We only need to check if we have ts->stopped to ensure the tick was stopped and we want to re-enable it. Checking ts->inidle there is useless.

/me goes la-la-la-la...

It would so help poor little me who hasn't stared at this code in detail for the past several days and is thus horridly confused if you'd expand your reasoning somewhat.

Sorry, I'm no big fan of writing changelogs and sometimes the lack it's unfortunately visible :)$

It needs to be refactored due to the previous patch beeing broken. But the rationale, indeed missing here, is that if you have ts->stopped then you have ts->inidle. Once you entered tick_nohz_stop_sched_tick() you have ts->inidle set and only once you reached that step the tick can be stopped, so the following check:

	if (!ts->inidle || !ts->tick_stopped)

can be summed up with:

	if (!ts->tick_stopped) {
#10 lore raw

PATCH 4/32 nohz: Separate idle sleeping time accounting from nohz switching

In reply to Frederic Weisbecker

To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.

This implies to implement some new API to call when we enter/exit idle.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 arch/arm/kernel/process.c              |    4 +-
 arch/avr32/kernel/process.c            |    4 +-
 arch/blackfin/kernel/process.c         |    4 +-
 arch/microblaze/kernel/process.c       |    4 +-
 arch/mips/kernel/process.c             |    4 +-
 arch/powerpc/kernel/idle.c             |    4 +-
 arch/powerpc/platforms/iseries/setup.c |    8 +-
 arch/s390/kernel/process.c             |    4 +-
 arch/sh/kernel/idle.c                  |    4 +-
 arch/sparc/kernel/process_64.c         |    4 +-
 arch/tile/kernel/process.c             |    4 +-
 arch/um/kernel/process.c               |    4 +-
 arch/unicore32/kernel/process.c        |    4 +-
 arch/x86/kernel/process_32.c           |    4 +-
 arch/x86/kernel/process_64.c           |    5 +-
 include/linux/tick.h                   |   10 ++-
 kernel/softirq.c                       |    2 +-
 kernel/time/tick-sched.c               |  102 ++++++++++++++++++-------------
 18 files changed, 98 insertions(+), 81 deletions(-)
arch/arm/kernel/process.c +2 -2
index 5e1e541..27b68b0 100644
@@ -182,7 +182,7 @@ void cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		leds_event(led_idle_start);
 		while (!need_resched()) {
 #ifdef CONFIG_HOTPLUG_CPU
@@ -208,7 +208,7 @@ void cpu_idle(void)
 			}
 		}
 		leds_event(led_idle_end);
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/avr32/kernel/process.c +2 -2
index ef5a2a0..e683a34 100644
@@ -34,10 +34,10 @@ void cpu_idle(void)
 {
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched())
 			cpu_idle_sleep();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/blackfin/kernel/process.c +2 -2
index 6a660fa..8082a8f 100644
@@ -88,10 +88,10 @@ void cpu_idle(void)
 #endif
 		if (!idle)
 			idle = default_idle;
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched())
 			idle();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/microblaze/kernel/process.c +2 -2
index 968648a..1b295b2 100644
@@ -103,10 +103,10 @@ void cpu_idle(void)
 		if (!idle)
 			idle = default_idle;
 
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched())
 			idle();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 
 		preempt_enable_no_resched();
 		schedule();
arch/mips/kernel/process.c +2 -2
index c28fbe6..3aa4020 100644
@@ -56,7 +56,7 @@ void __noreturn cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched() && cpu_online(cpu)) {
 #ifdef CONFIG_MIPS_MT_SMTC
 			extern void smtc_idle_loop_hook(void);
@@ -77,7 +77,7 @@ void __noreturn cpu_idle(void)
 		     system_state == SYSTEM_BOOTING))
 			play_dead();
 #endif
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/powerpc/kernel/idle.c +2 -2
index 39a2baa..1108260 100644
@@ -56,7 +56,7 @@ void cpu_idle(void)
 
 	set_thread_flag(TIF_POLLING_NRFLAG);
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched() && !cpu_should_die()) {
 			ppc64_runlatch_off();
 
@@ -93,7 +93,7 @@ void cpu_idle(void)
 
 		HMT_medium();
 		ppc64_runlatch_on();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		if (cpu_should_die())
 			cpu_die();
arch/powerpc/platforms/iseries/setup.c +4 -4
index c25a081..d40dcd9 100644
@@ -562,7 +562,7 @@ static void yield_shared_processor(void)
 static void iseries_shared_idle(void)
 {
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched() && !hvlpevent_is_pending()) {
 			local_irq_disable();
 			ppc64_runlatch_off();
@@ -576,7 +576,7 @@ static void iseries_shared_idle(void)
 		}
 
 		ppc64_runlatch_on();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 
 		if (hvlpevent_is_pending())
 			process_iSeries_events();
@@ -592,7 +592,7 @@ static void iseries_dedicated_idle(void)
 	set_thread_flag(TIF_POLLING_NRFLAG);
 
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		if (!need_resched()) {
 			while (!need_resched()) {
 				ppc64_runlatch_off();
@@ -609,7 +609,7 @@ static void iseries_dedicated_idle(void)
 		}
 
 		ppc64_runlatch_on();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/s390/kernel/process.c +2 -2
index 541a750..560cd94 100644
@@ -90,10 +90,10 @@ static void default_idle(void)
 void cpu_idle(void)
 {
 	for (;;) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched())
 			default_idle();
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/sh/kernel/idle.c +2 -2
index 425d604..b7ea6ff 100644
@@ -88,7 +88,7 @@ void cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 
 		while (!need_resched()) {
 			check_pgt_cache();
@@ -109,7 +109,7 @@ void cpu_idle(void)
 			start_critical_timings();
 		}
 
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/sparc/kernel/process_64.c +2 -2
index c158a95..5c36632 100644
@@ -95,12 +95,12 @@ void cpu_idle(void)
 	set_thread_flag(TIF_POLLING_NRFLAG);
 
 	while(1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 
 		while (!need_resched() && !cpu_is_offline(cpu))
 			sparc64_yield(cpu);
 
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 
 		preempt_enable_no_resched();
 
arch/tile/kernel/process.c +2 -2
index 9c45d8b..cc1bd4f 100644
@@ -85,7 +85,7 @@ void cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched()) {
 			if (cpu_is_offline(cpu))
 				BUG();  /* no HOTPLUG_CPU */
@@ -105,7 +105,7 @@ void cpu_idle(void)
 				local_irq_enable();
 			current_thread_info()->status |= TS_POLLING;
 		}
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/um/kernel/process.c +2 -2
index fab4371..f1b3864 100644
@@ -245,10 +245,10 @@ void default_idle(void)
 		if (need_resched())
 			schedule();
 
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		nsecs = disable_timer();
 		idle_sleep(nsecs);
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 	}
 }
 
arch/unicore32/kernel/process.c +2 -2
index ba401df..e2df91a 100644
@@ -55,7 +55,7 @@ void cpu_idle(void)
 {
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched()) {
 			local_irq_disable();
 			stop_critical_timings();
@@ -63,7 +63,7 @@ void cpu_idle(void)
 			local_irq_enable();
 			start_critical_timings();
 		}
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/x86/kernel/process_32.c +2 -2
index a3d0dc5..1d7e26c 100644
@@ -97,7 +97,7 @@ void cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched()) {
 
 			check_pgt_cache();
@@ -112,7 +112,7 @@ void cpu_idle(void)
 			pm_idle();
 			start_critical_timings();
 		}
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
arch/x86/kernel/process_64.c +2 -3
index ca6f7ab..5fce49b 100644
@@ -120,7 +120,7 @@ void cpu_idle(void)
 
 	/* endless idle loop with no priority at all */
 	while (1) {
-		tick_nohz_stop_sched_tick(1);
+		tick_nohz_enter_idle();
 		while (!need_resched()) {
 
 			rmb();
@@ -144,8 +144,7 @@ void cpu_idle(void)
 			   loops can be woken up without interrupt. */
 			__exit_idle();
 		}
-
-		tick_nohz_restart_sched_tick();
+		tick_nohz_exit_idle();
 		preempt_enable_no_resched();
 		schedule();
 		preempt_disable();
include/linux/tick.h +6 -4
index 532e650..04f6418 100644
@@ -120,14 +120,16 @@ static inline int tick_oneshot_mode_active(void) { return 0; }
 #endif /* !CONFIG_GENERIC_CLOCKEVENTS */
 
 # ifdef CONFIG_NO_HZ
-extern void tick_nohz_stop_sched_tick(int inidle);
-extern void tick_nohz_restart_sched_tick(void);
+extern void tick_nohz_enter_idle(void);
+extern void tick_nohz_exit_idle(void);
+extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 # else
-static inline void tick_nohz_stop_sched_tick(int inidle) { }
-static inline void tick_nohz_restart_sched_tick(void) { }
+static inline void tick_nohz_enter_idle(void) { }
+static inline void tick_nohz_exit_idle(void) { }
+
 static inline ktime_t tick_nohz_get_sleep_length(void)
 {
 	ktime_t len = { .tv64 = NSEC_PER_SEC/HZ };
kernel/softirq.c +1 -1
index 40cf63d..67a1401 100644
@@ -343,7 +343,7 @@ void irq_exit(void)
 #ifdef CONFIG_NO_HZ
 	/* Make sure that timer wheel updates are propagated */
 	if (idle_cpu(smp_processor_id()) && !in_interrupt() && !need_resched())
-		tick_nohz_stop_sched_tick(0);
+		tick_nohz_irq_exit();
 #endif
 	preempt_enable_no_resched();
 }
kernel/time/tick-sched.c +59 -43

161 lines not on this page · raw message

Signature · 1 line
1.7.5.4
#11 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

Quoted 5 lines · Frederic Weisbecker

To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.

This implies to implement some new API to call when we enter/exit idle.

I mean, I really love brevity, but you seem to just not state all the important bits ;-)

So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the tick_nohz_stop_sched_tick() call on going idle will simply not do anything.

If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.

#12 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.

This implies to implement some new API to call when we enter/exit idle.

I mean, I really love brevity, but you seem to just not state all the important bits ;-)

So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the tick_nohz_stop_sched_tick() call on going idle will simply not do anything.

Quoted 4 lines

If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.

That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...

I could do these things conditionally using a some idle_cpu() checks but the end result would not be very proper.

#13 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

Quoted 21 lines · Frederic Weisbecker
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.

This implies to implement some new API to call when we enter/exit idle.

I mean, I really love brevity, but you seem to just not state all the important bits ;-)

So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the tick_nohz_stop_sched_tick() call on going idle will simply not do anything.

Quoted 4 lines

If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.

Quoted 6 lines

That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...

I could do these things conditionally using a some idle_cpu() checks but the end result would not be very proper.

Right, but you didn't explain any of that in the changelog. So the reasoning is that because tick_nohz_stop_sched_tick() does more than just stop the tick, and this extra work needs to be isolated to just the idle case, therefore we need hooks specific for the idle loop.

#14 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Frederic Weisbecker
Quoted 15 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for having nohz mode switching independant from idle, pull the idle sleeping time accounting out of the tick stop API.

This implies to implement some new API to call when we enter/exit idle.

I mean, I really love brevity, but you seem to just not state all the important bits ;-)

So the goal is to disable the tick more often (say when running 1 userbound task), why does that need new hooks? If we already had the tick disabled, the tick_nohz_stop_sched_tick() call on going idle will simply not do anything.

Quoted 4 lines

If we go from idle to running something we want to enable the tick initially because doing the task wakeup involves RCU etc.. Once we find the task is indeed userbound and we've finished all our state we can disable the thing again.

That's because we are going to have two different sources of stop/restarting the tick: either idle or a random task. In the case of idle we have very specific things to handle like idle time accounting, idle stats, rcu, ...

Quoted 7 lines
Quoted 2 lines

I could do these things conditionally using a some idle_cpu() checks but the end result would not be very proper.

Right, but you didn't explain any of that in the changelog. So the reasoning is that because tick_nohz_stop_sched_tick() does more than just stop the tick, and this extra work needs to be isolated to just the idle case, therefore we need hooks specific for the idle loop.

Right, I'll update the changelog.

#15 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

In reply to Frederic Weisbecker

include/linux/tick.h static inline int tick_oneshot_mode_active(void) { return 0; } · line 125

Quoted 2 lines · Frederic Weisbecker
include/linux/tick.h +2 -0
+extern void tick_nohz_exit_idle(void);
+extern void tick_nohz_irq_exit(void); 

For consistencies sake, pick either tick_nohz_exit_*() or tick_nohz_*_exit() but don't go mix and match (I prefer the former).

Also, there isn't a matching irq_enter() callback..

#16 lore raw

Re: [PATCH 04/32] nohz: Separate idle sleeping time accounting from nohz switching

Quoted 6 lines · Peter Zijlstra

include/linux/tick.h static inline int tick_oneshot_mode_active(void) { return 0; } · line 125

Quoted 2 lines · Frederic Weisbecker
include/linux/tick.h +2 -0
+extern void tick_nohz_exit_idle(void);
+extern void tick_nohz_irq_exit(void); 

For consistencies sake, pick either tick_nohz_exit_*() or tick_nohz_*_exit() but don't go mix and match (I prefer the former).

Agreed, I will.

Quoted 1 line

Also, there isn't a matching irq_enter() callback..

Yeah, probably tick_check_nohz() should be renamed accordingly.

#17 lore raw

PATCH 5/32 nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

In reply to Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)
kernel/time/tick-sched.c +7 -4
index df6bb4c..8937d4a 100644
@@ -385,7 +385,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now)
 			ts->idle_tick = hrtimer_get_expires(&ts->sched_timer);
 			ts->tick_stopped = 1;
 			ts->idle_jiffies = last_jiffies;
-			rcu_enter_nohz();
 		}
 
 		ts->idle_sleeps++;
@@ -429,9 +428,13 @@ out:
 static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 {
 	ktime_t now;
+	int was_stopped = ts->tick_stopped;
 
 	now = tick_nohz_start_idle(cpu, ts);
 	tick_nohz_stop_sched_tick(now);
+
+	if (!was_stopped && ts->tick_stopped)
+		rcu_enter_nohz();
 }
 
 void tick_nohz_enter_idle(void)
@@ -499,8 +502,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)
 	unsigned long ticks;
 #endif
 
-	rcu_exit_nohz();
-
 	/* Update jiffies first */
 	select_nohz_load_balancer(0);
 	tick_do_update_jiffies64(now);
@@ -547,8 +548,10 @@ void tick_nohz_exit_idle(void)
 	tick_nohz_stop_idle(smp_processor_id(), now);
 	ts->inidle = 0;
 
-	if (ts->tick_stopped)
+	if (ts->tick_stopped) {
+		rcu_exit_nohz();
 		tick_nohz_restart_sched_tick(now, ts);
+	}
 
 	local_irq_enable();
 }
Signature · 1 line
1.7.5.4
#18 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 5 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

#19 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

But we can stop the tick from the kernel, not just userspace.

#20 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

But we can stop the tick from the kernel, not just userspace.

Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?

Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?

#21 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

Quoted 8 lines
Quoted 1 line

But we can stop the tick from the kernel, not just userspace.

Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?

Yeah. Either the interrupt returns to userspace and then we call rcu_enter_nohz() or we return to kernel space and then a further use of rcu will restart the tick.

Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick. But we need it as long as there is an RCU callback enqueued on some CPU.

Quoted 2 lines

Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?

This is why we have rcu_needs_cpu() and rcu_pending() checks before stopping the tick.

rcu_needs_cpu() checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.

rcu_pending() is there to know if another CPU started a grace period so we need the tick to complete it.

#22 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 23 lines · Frederic Weisbecker
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

Quoted 8 lines
Quoted 1 line

But we can stop the tick from the kernel, not just userspace.

Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?

Quoted 6 lines

Yeah. Either the interrupt returns to userspace and then we call rcu_enter_nohz() or we return to kernel space and then a further use of rcu will restart the tick.

Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick.

But but but, then how is it going to stop a grace period from happening? The grace period state is per-cpu and the whole state machine is tick driven.

Now some of the new RCU things go kick cpus with IPIs to push grace periods along, but I would expect you don't want that to happen either, the whole purpose here is to leave a cpu alone, unperturbed.

That means it has to be in an extended grace period when we stop the tick.

Quoted 2 lines

But we need it as long as there is an RCU callback enqueued on some CPU.

Well, no, only if there's one enqueued on this cpu because then we can't enter the extended grace period.

Quoted 11 lines
Quoted 2 lines

Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?

This is why we have rcu_needs_cpu() and rcu_pending() checks before stopping the tick.

rcu_needs_cpu() checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.

rcu_pending() is there to know if another CPU started a grace period so we need the tick to complete it.

Hence the extended grace period, so we don't need to complete grace periods.

#23 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 28 lines · Peter Zijlstra
Quoted 27 lines · Frederic Weisbecker
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker

To prepare for nohz / idle logic split, pull out the rcu dynticks idle mode switching to strict idle entry/exit areas.

So we make the dyntick mode possible without always involving rcu extended quiescent state.

Why is this a good thing? I would be thinking that if we're a userspace bound task and we disable the tick rcu would be finished on this cpu and thus the extended quiescent state is just what we want?

Quoted 8 lines
Quoted 1 line

But we can stop the tick from the kernel, not just userspace.

Humm!? I'm confused, I thought the idea was to only stop the tick when we're 'stuck' in a user bound task. Now I get that we have to stop the tick from kernel space (as in the interrupt will clearly run in kernel space), but assuming the normal return from interrupt path doesn't use rcu, and using rcu (as per a later patch) re-enables the tick again, it doesn't matter, right?

Yeah. Either the interrupt returns to userspace and then we call rcu_enter_nohz() or we return to kernel space and then a further use of rcu will restart the tick.

Quoted 6 lines
Quoted 2 lines

Now this is not any use of rcu. Uses of rcu read side critical section don't need the tick.

But but but, then how is it going to stop a grace period from happening? The grace period state is per-cpu and the whole state machine is tick driven.

But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.

OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.

Quoted 3 lines

Now some of the new RCU things go kick cpus with IPIs to push grace periods along, but I would expect you don't want that to happen either, the whole purpose here is to leave a cpu alone, unperturbed.

Sure we want the CPU to be unperturbed but not if that sacrifies correctness. As long as we run in the kernel we want to receive such IPIs to restart the tick as needed.

Quoted 2 lines

That means it has to be in an extended grace period when we stop the tick.

You mean extended quiescent state?

As a summary here is what we do:

- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.

- if we are in userspace we can run in extended quiescent state.

Quoted 6 lines
Quoted 3 lines

But we need it as long as there is an RCU callback enqueued on some CPU.

Well, no, only if there's one enqueued on this cpu because then we can't enter the extended grace period.

True if we are in userspace.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Also, RCU needs the tick to drive the state machine, so how can you stop the tick and not also stop the RCU state machine?

This is why we have rcu_needs_cpu() and rcu_pending() checks before stopping the tick.

Quoted 8 lines
Quoted 5 lines

rcu_needs_cpu() checks we have no local callback enqueued, in which case the local CPU is responsible of the RCU state machine.

rcu_pending() is there to know if another CPU started a grace period so we need the tick to complete it.

Hence the extended grace period, so we don't need to complete grace periods.

I hope the above explanations made things more clear.

#24 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 3 lines · Frederic Weisbecker

But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.

Note that with PREEMPT_RCU only the rcu_read_lock() is actually an rcu read side critical section, non of the others should be used as such. Relying on preempt_disable(), local_bh_disable() and similar is broken as per a long while ago.

#25 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.

Note that with PREEMPT_RCU only the rcu_read_lock() is actually an rcu read side critical section, non of the others should be used as such. Relying on preempt_disable(), local_bh_disable() and similar is broken as per a long while ago.

Sure yeah.

My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.

#26 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.

Note that with PREEMPT_RCU only the rcu_read_lock() is actually an rcu read side critical section, non of the others should be used as such. Relying on preempt_disable(), local_bh_disable() and similar is broken as per a long while ago.

Quoted 7 lines

Sure yeah.

My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.

But you wake up the tick for any callback - systemwide - right? That's a massive fail.

#27 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

But rcu read side critical sections (preemption disabled, rcu_read_lock(), softirq disabled) don't need the tick to enforce the critical section itself.

Note that with PREEMPT_RCU only the rcu_read_lock() is actually an rcu read side critical section, non of the others should be used as such. Relying on preempt_disable(), local_bh_disable() and similar is broken as per a long while ago.

Sure yeah.

Quoted 8 lines
Quoted 5 lines

My point was that the patchset doesn't care about all that anyway. Read side critical section still work as usual. What changes is the way we notice periods where we are *not* in rcu read side critical sections. This was previously partly made through the tick. Now it's still the case but we need to remotely wake up that tick first.

But you wake up the tick for any callback - systemwide - right? That's a massive fail.

Until we find a better solution yeah.

#28 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

In reply to Frederic Weisbecker
Quoted 4 lines · Frederic Weisbecker

OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.

Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.

Requiring it to participate in the grace period state machine is a fail, plain and simple.

#29 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.

Quoted 7 lines

Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.

Requiring it to participate in the grace period state machine is a fail, plain and simple.

We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.

Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.

We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:

- Having explicit hooks in preempt_disable() and local_irq_restore() to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.

- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.

#30 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 15 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.

Quoted 7 lines

Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.

Requiring it to participate in the grace period state machine is a fail, plain and simple.

Quoted 8 lines

We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.

Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.

Well the simple fact is that rcu, when considered system-wide, is pretty much always busy, voiding any and all benefit you might want to gain.

Quoted 5 lines

We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:

Quoted 8 lines

- Having explicit hooks in preempt_disable() and local_irq_restore() to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.

- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.

Right, so what we can do is keep a per-cpu context switch counter (I'm sure we have one someplace and we already have the rcu_note_context_switch() callback in case we need another) and have another cpu (outside of our extended nohz domain) drive our state machine.

But I'm sure Paul can say more sensible things than me here.

#31 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

OTOH it is needed to find non-critical sections when asked to cooperate in a grace period completion. But if no callback have been enqueued on the whole system we are fine.

Quoted 7 lines

Its that 'whole system' clause that I have a problem with. It would be perfectly fine to have a number of cpus very busy generating rcu callbacks, however this should not mean our adaptive nohz cpu should be bothered to complete grace periods.

Requiring it to participate in the grace period state machine is a fail, plain and simple.

We need those nohz CPUs to participate because they may use read side critical section themselves. So we need them to delay running grace period until the end of their running rcu read side critical sections, like any other CPUs. Otherwise their supposed rcu read side critical section wouldn't be effective.

Quoted 5 lines
Quoted 2 lines

Either that or we need to only stop the tick when we are in userspace. I'm not sure it would be a good idea.

Well the simple fact is that rcu, when considered system-wide, is pretty much always busy, voiding any and all benefit you might want to gain.

With my testcase, a stupid userspace loop on a single CPU among 4, I actually see only few RCU activity. Especially as any other CPU is pretty much idle. There are some cases where it's not so pointless.

Quoted 14 lines
Quoted 5 lines

We discussed this problem, I believe the problem mostly resides in rcu sched. Because finding quiescent states for rcu bh is easy, but rcu sched needs the tick or context switches. (For rcu preempt I have no idea.) So for now that's the sanest way we found amongst:

Quoted 8 lines

- Having explicit hooks in preempt_disable() and local_irq_restore() to notice end of rcu sched critical section. So that we don't need the tick anymore to find quiescent states. But that's going to be costly. And we may miss some more implicitly non-preemptable code path.

- Rely on context switches only. I believe in practice it should be fine. But in theory this delays the grace period completion for an unbounded amount of time.

Quoted 7 lines

Right, so what we can do is keep a per-cpu context switch counter (I'm sure we have one someplace and we already have the rcu_note_context_switch() callback in case we need another) and have another cpu (outside of our extended nohz domain) drive our state machine.

But I'm sure Paul can say more sensible things than me here.

Yeah I hope we can find some solution to minimize these IPIs.

#32 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

In reply to Frederic Weisbecker
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines

That means it has to be in an extended grace period when we stop the tick.

You mean extended quiescent state?

Yeah that :-)

Quoted 11 lines

As a summary here is what we do:

- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.

Quoted 1 line

- if we are in userspace we can run in extended quiescent state.

But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.

#33 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines

That means it has to be in an extended grace period when we stop the tick.

You mean extended quiescent state?

Yeah that :-)

Quoted 11 lines

As a summary here is what we do:

- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.

Quoted 5 lines
Quoted 1 line

- if we are in userspace we can run in extended quiescent state.

But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.

No, we have a specific stacking in the irq:

	rcu_irq_enter()

	disable tick...
	if (user)
		rcu_enter_nohz();

	rcu_irq_exit() <-- extended quiescent state entry effective only there

And by the time we call rcu_irq_exit() and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal with do_notify_resume() which I have yet to handle).

#34 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 36 lines · Frederic Weisbecker
Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines

That means it has to be in an extended grace period when we stop the tick.

You mean extended quiescent state?

Yeah that :-)

Quoted 11 lines

As a summary here is what we do:

- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.

Quoted 5 lines
Quoted 1 line

- if we are in userspace we can run in extended quiescent state.

But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.

No, we have a specific stacking in the irq:

	rcu_irq_enter()

	disable tick...
	if (user)
		rcu_enter_nohz();

	rcu_irq_exit() <-- extended quiescent state entry effective only there
Quoted 3 lines

And by the time we call rcu_irq_exit() and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal with do_notify_resume() which I have yet to handle).

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

#35 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Frederic Weisbecker
Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker
Quoted 2 lines

That means it has to be in an extended grace period when we stop the tick.

You mean extended quiescent state?

Yeah that :-)

Quoted 11 lines

As a summary here is what we do:

- if we are in the kernel, we can't run into extended quiescent state because we may make use of rcu anytime there. But if we run nohz we don't have the tick to notice quiescent states to the RCU machinery and help completing grace periods so as soon as we receive an rcu IPI from another CPU (due to the grace period beeing extended because our nohz CPU doesn't report quiescent states), we restart the tick. We are optimistic enough to consider that we may avoid a lot of ticks even if there are some risks to be disturbed in some random rates. So even with the IPI we consider it as an upside.

Quoted 5 lines
Quoted 1 line

- if we are in userspace we can run in extended quiescent state.

But you can only disable the tick/enter extended quiescent state while in kernel-space. Thus the second clause is precluded from ever being true.

No, we have a specific stacking in the irq:

	rcu_irq_enter()

	disable tick...
	if (user)
		rcu_enter_nohz();

	rcu_irq_exit() <-- extended quiescent state entry effective only there
Quoted 8 lines
Quoted 3 lines

And by the time we call rcu_irq_exit() and we resume to userspace, we are not supposed to have rcu read side critical section (minus the case of a signal with do_notify_resume() which I have yet to handle).

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

#36 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 6 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

#37 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 4 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Quoted 5 lines
Quoted 2 lines

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

At least if we are in userspace it does as long as we have no local rcu callbacks to handle. We have the rcu_pending() check but we can remove it for userspace.

Now for the kernel space I still think it's worth, for example when all other CPUs are idle, or when they don't queue that much RCU callbacks. In theory if we have a grace period to complete 100 times per second but HZ=1000 then we are going to avoid a lot of timer interrupts.
If we can't remove the timer, we can at least minimize it.

maintainer
#38 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

In reply to Frederic Weisbecker
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 4 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Quoted 5 lines
Quoted 2 lines

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

So how about something like:

Assuming we are in rcu_nohz state; on kernel enter we leave rcu_nohz but don't start the tick, instead we assign another cpu to run our state machine.

On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter rcu_nohz state.

If between that time we did restart the tick, we take back our rcu state and skip the donate and rcu_nohz enter on kernel exit.

I really should go read all those docs Paul send me to see how insane the above is.

#39 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 6 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Quoted 8 lines
Quoted 2 lines

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

So how about something like:

Assuming we are in rcu_nohz state; on kernel enter we leave rcu_nohz but don't start the tick, instead we assign another cpu to run our state machine.

The nohz CPU still has to notice its own quiescent states. Now it could be an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.

Quoted 3 lines

On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter rcu_nohz state.

That's already what does rcu_enter_nohz().

Quoted 2 lines

If between that time we did restart the tick, we take back our rcu state and skip the donate and rcu_nohz enter on kernel exit.

That's also what is done in this patchset. As soon as we re-enter the kernel or the tick had to be restarted before we re-enter the kernel, we call rcu_exit_nohz() that pulls back the CPU to the whole RCU machinery.

maintainer
#40 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 26 lines · Frederic Weisbecker
Quoted 25 lines · Peter Zijlstra
Quoted 23 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 6 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

Quoted 7 lines
Quoted 5 lines

So how about something like:

Assuming we are in rcu_nohz state; on kernel enter we leave rcu_nohz but don't start the tick, instead we assign another cpu to run our state machine.

The nohz CPU still has to notice its own quiescent states.

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

Quoted 4 lines

Now it could be
an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.

No, seriously, RCU should not, ever, need to re-enable the tick. Imagine a HPC workload where the system cores are also responsible for all IO and all the adaptive-nohz cores are simply crunching numbers. In that scenario you'll have a very high rcu usage because the system cores are all very busy arranging work for the computation cores.

Quoted 5 lines
Quoted 3 lines

On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter rcu_nohz state.

That's already what does rcu_enter_nohz().

Almost but not quite, it doesn't donate the callbacks for example (something it does do on hotplug -- and therefore any assumption the callback will in fact run on the cpu you submit it on is already broken).

Quoted 4 lines
Quoted 2 lines

If between that time we did restart the tick, we take back our rcu state and skip the donate and rcu_nohz enter on kernel exit.

That's also what is done in this patchset.

Its not, since you don't hand of the grace period detectoring you don't take it back now do you..

Quoted 2 lines

As soon as we re-enter the kernel
or the tick had to be restarted before we re-enter the kernel,

Another impossibility, you can only restart the tick from the kernel.

Quoted 2 lines

we call
rcu_exit_nohz() that pulls back the CPU to the whole RCU machinery.

But you then also start the tick again..

#41 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Frederic Weisbecker
Quoted 27 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines

See all that is still kernelspace ;-) I think I know what you mean to say though, but seeing as you note there is even now a known shortcoming I'm not very confident its a solid construction. What will help us find such holes?

This: https://lkml.org/lkml/2011/6/23/744

It's in one of Paul's branches and should make it for the next merge window. This should detect any of such holes. I made that on purpose for the nohz cpusets when I saw how much error prone that can be with rcu :)

OK, good ;-)

Quoted 4 lines
Quoted 3 lines

I would much rather we not rely on such fragile things too much.. this RCU stuff wants way more thought, as it stands your patch-set doesn't do anything useful IMO.

Quoted 5 lines
Quoted 2 lines

Not sure what you mean. Well that Rcu thing for sure is fragile but we have the tools ready to find the problems.

Right that thing you linked above does catch abuse, still your current proposal means that due to RCU it will basically never disable the tick.

So how about something like:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Assuming we are in rcu_nohz state; on kernel enter we leave rcu_nohz but don't start the tick, instead we assign another cpu to run our state machine.

The nohz CPU still has to notice its own quiescent states.

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.

Would be nice to hear about Paul's opinion on that.

Quoted 10 lines
Quoted 4 lines

Now it could be
an optimization to ask another CPU to handle all the rest once that quiescent state is found. That doesn't solve our main problem though which is to reliably report quiescent states when asked for.

No, seriously, RCU should not, ever, need to re-enable the tick. Imagine a HPC workload where the system cores are also responsible for all IO and all the adaptive-nohz cores are simply crunching numbers. In that scenario you'll have a very high rcu usage because the system cores are all very busy arranging work for the computation cores.

Of course if we find a better way than having to restart this tick I'm all for doing that way.

That said if it requires some significant changes this should be done outside this patchset, as an optimization afterward may be, the patchset is already big while still missing very important features for now that the timer handles.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

On kernel exit we 'donate' all our rcu state to a willing victim (the same that earlier was kind enough to drive our state) and undo our entire GP accounting and re-enter rcu_nohz state.

That's already what does rcu_enter_nohz().

Almost but not quite, it doesn't donate the callbacks for example (something it does do on hotplug -- and therefore any assumption the callback will in fact run on the cpu you submit it on is already broken).

Good to know, so that would avoid to restart the tick on call_rcu() ? Sounds good but again I think this should be done later.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

If between that time we did restart the tick, we take back our rcu state and skip the donate and rcu_nohz enter on kernel exit.

That's also what is done in this patchset.

Its not, since you don't hand of the grace period detectoring you don't take it back now do you..

So you are talking about grace period started locally due to local callbacks enqueued, right?

Quoted 4 lines
Quoted 2 lines

As soon as we re-enter the kernel
or the tick had to be restarted before we re-enter the kernel,

Another impossibility, you can only restart the tick from the kernel.

Ok I meant it can be restarted from an interrupt interrupting userspace. I was talking about kernel enter/exit considering the new hooks brought (syscalls and exceptions).

Quoted 4 lines
Quoted 2 lines

we call
rcu_exit_nohz() that pulls back the CPU to the whole RCU machinery.

But you then also start the tick again..

When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().

maintainer
#42 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.

But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.

Quoted 4 lines
Quoted 1 line

But you then also start the tick again..

When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().

So thinking more about all this:

rcu_exit_nohz() will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).

Then when we leave the kernel (or go idle) we re-enter rcu_nohz state, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.

I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.

So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.

Now I forgot where all you send IPIs from, and I'll go look at these patches once more.

As for call_rcu() for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.

But yeah, input from Paul would be nice...

#43 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 10 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.

But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.

Ah, but it really is written, among other things, by the OOM killer. ;-)

Quoted 5 lines
Quoted 4 lines
Quoted 1 line

But you then also start the tick again..

When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().

Quoted 7 lines

So thinking more about all this:

rcu_exit_nohz() will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).

Lacking a tick, the CPU also fails to respond to state updates from other CPUs.

Quoted 4 lines

Then when we leave the kernel (or go idle) we re-enter rcu_nohz state, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.

This is true.

Quoted 2 lines

I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.

All the ways I know of to make PREEMPT_RCU live without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.

Or are you saying that PREEMPT_RCU does not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.

Quoted 6 lines

So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.

Now I forgot where all you send IPIs from, and I'll go look at these patches once more.

Quoted 6 lines

As for call_rcu() for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.

But yeah, input from Paul would be nice...

In the call_rcu() case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)

The reason for doing this is that people are enabling CONFIG_RCU_FAST_NO_HZ on systems that have no business enabling it. Bad choice of names on my part.

Thanx, Paul

maintainer
#44 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 5 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
Quoted 2 lines

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.

But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.

Ah, but it really is written, among other things, by the OOM killer. ;-)

Well there is that of course :-) But I think the below argument relies on what we already have without requiring more.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

But you then also start the tick again..

When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().

So thinking more about all this:

Quoted 8 lines
Quoted 5 lines

rcu_exit_nohz() will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).

Lacking a tick, the CPU also fails to respond to state updates from other CPUs.

I'm sure I'll have to go re-read your documents, but does that matter? If we would have had a tick we still couldn't have progressed since we wouldn't have scheduled etc.. so we would hold up GP completion any way.

Quoted 6 lines
Quoted 4 lines

Then when we leave the kernel (or go idle) we re-enter rcu_nohz state, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.

This is true.

So suppose all other CPUs completed the GP and our CPU is the one holding things up, now I don't see rcu_enter_nohz() doing anything much at all, who is responsible for GP completion?

Quoted 9 lines
Quoted 2 lines

I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.

All the ways I know of to make PREEMPT_RCU live without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.

Quoted 3 lines

Or are you saying that PREEMPT_RCU does not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.

I'm not saying anything much about PREEMPT_RCU, I voiced an ill-considered suspicion :-)

So in the nr_running=[0,1] case we're in rcu_nohz state when idle or when in userspace. The only interesting part is being in kernel space where we cannot be in rcu_nohz state because we might actually use pointers and thus have to stop callbacks from destroying state etc..

The only PREEMPT_RCU implementation I can recall is the counting one, and that one does indeed want a tick, because even in kernel space it could move things forward if the 'old' index counter reaches 0.

Now we could possibly add magic to rcu_read_unlock_special() to restart the tick in that case.

Now clearly all that might be non-applicable to the current one, will have to wrap my head around the current PREEMPT_RCU implementation some more.

Quoted 11 lines
Quoted 11 lines

So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.

Now I forgot where all you send IPIs from, and I'll go look at these patches once more.

As for call_rcu() for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.

Quoted 6 lines
Quoted 1 line

But yeah, input from Paul would be nice...

In the call_rcu() case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)

Hand your callback to someone else is one solution, but I'm not overly worried about re-starting the tick if we do call_rcu().

Quoted 3 lines

The reason for doing this is that people are enabling CONFIG_RCU_FAST_NO_HZ on systems that have no business enabling it. Bad choice of names on my part.

hehe :-)

#45 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Paul E. McKenney
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Quoted 2 lines

Why? rcu-sched can use a context-switch counter, rcu-preempt doesn't even need that. Remote cpus can notice those just fine.

If that's fine to only rely on context switches, which don't happen in a bounded time in theory, then ok.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

But (!PREEMPT) rcu already depends on that, and suffers this lack of time-bounds. What it does to expedite matters is force context switches, but nowhere is it written the GP is bounded by anything sane.

Ah, but it really is written, among other things, by the OOM killer. ;-)

Well there is that of course :-) But I think the below argument relies on what we already have without requiring more.

Almost. ;-)

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 4 lines
Quoted 1 line

But you then also start the tick again..

When we enter kernel? (minus interrupts)
No we only call rcu_exit_nohz().

So thinking more about all this:

rcu_exit_nohz() will make remote cpus wait for us, this is exactly what is needed because we might have looked at pointers. Lacking a tick we don't progress our own state but that is fine, !PREEMPT RCU wouldn't have been able to progress our state anyway since we haven't scheduled (there's nothing to schedule to except idle, see below).

Quoted 6 lines
Quoted 2 lines

Lacking a tick, the CPU also fails to respond to state updates from other CPUs.

I'm sure I'll have to go re-read your documents, but does that matter? If we would have had a tick we still couldn't have progressed since we wouldn't have scheduled etc.. so we would hold up GP completion any way.

There are two phases to quiescent-state detection: (1) actually detecting the quiescent state and (2) reporting detection to the RCU core. If you turn off the tick at an inopportune time, you can have CPUs that have detected the tick, but not yet reported it.

You asked the follow-up question below, so please see below.

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

Then when we leave the kernel (or go idle) we re-enter rcu_nohz state, and the other cpus will ignore our contribution (since we have entered a QS and can't be holding any pointers) the other CPUs can continue and complete the GP and run the callbacks.

This is true.

So suppose all other CPUs completed the GP and our CPU is the one holding things up, now I don't see rcu_enter_nohz() doing anything much at all, who is responsible for GP completion?

Any CPU that has RCU callbacks queued that are waiting for the current or some susequent grace period to complete are responsible for pushing the current grace period forward, hence the checks in the non-RCU_FAST_NO_HZ variants of rcu_needs_cpu(). This is why CPUs with callbacks that are not yet done cannot currently disable the tick -- because we need at least one CPU to detect the fact that dyntick-idle CPUs are in fact in extended quiescent states.

Again, I believe that I can do better, hence the in-progress rewrite of RCU_FAST_NO_HZ. Either that or get most people to stop using it, and given its name, getting people to stop using it is likely an exercise in futility. "But it is FAST, and that is good, and it involve NO_HZ, which saves energy, which is also good. Therefore, I will enable it -everywhere-!!!"

Sigh. It will be much easier to rewrite it, ugly corner cases nonwithstanding. :-(

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

I haven't fully considered PREEMPT RCU quite yet, but I'm thinking we can get away with something similar.

All the ways I know of to make PREEMPT_RCU live without a scheduling clock tick while not in some form of dyntick-idle mode require either IPIs or read-side memory barriers. The special case where all CPUs are in dyntick-idle mode and something needs to happen also needs to be handled correctly.

Quoted 6 lines
Quoted 3 lines

Or are you saying that PREEMPT_RCU does not need a CPU to take scheduling-clock interrupts while that CPU is in dyntick-idle mode? That is true enough.

I'm not saying anything much about PREEMPT_RCU, I voiced an ill-considered suspicion :-)

;-)

Quoted 4 lines

So in the nr_running=[0,1] case we're in rcu_nohz state when idle or when in userspace. The only interesting part is being in kernel space where we cannot be in rcu_nohz state because we might actually use pointers and thus have to stop callbacks from destroying state etc..

Yep!

Quoted 6 lines

The only PREEMPT_RCU implementation I can recall is the counting one, and that one does indeed want a tick, because even in kernel space it could move things forward if the 'old' index counter reaches 0.

Now we could possibly add magic to rcu_read_unlock_special() to restart the tick in that case.

Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...

I don't currently have an opinion about the advisability of restarting the tick from hardIRQ handlers, but I do feel the need to point out the the possibility.

Quoted 3 lines

Now clearly all that might be non-applicable to the current one, will have to wrap my head around the current PREEMPT_RCU implementation some more.

Indeed, the documentation is going much more slowly than I would like...

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

So per the above we don't need the tick at all (for the case of nr_running=[0,1]), RCU will sort itself out.

Now I forgot where all you send IPIs from, and I'll go look at these patches once more.

As for call_rcu() for that we can indeed wake the tick (on leaving kernel space or entering idle, no need to IPI since we can't process anything before that anyway) or we could hand off our call list to a 'willing' victim.

But yeah, input from Paul would be nice...

Quoted 7 lines
Quoted 4 lines

In the call_rcu() case, I do have some code in preparation that allows CPUs to have non-empty callback queues and still be tickless. There are some tricky corner cases, but it does look possible. (Famous last words...)

Hand your callback to someone else is one solution, but I'm not overly worried about re-starting the tick if we do call_rcu().

As long as the handoff doesn't turn into a battery-killing game of RCU-callback hot potato. And I am seriously concerned about this possibility.

I will be updating the Dyntick-Idle doc to cover the new RCU_FAST_NO_HZ algorithm if and when I get it into human-readable form.

Quoted 5 lines
Quoted 3 lines

The reason for doing this is that people are enabling CONFIG_RCU_FAST_NO_HZ on systems that have no business enabling it. Bad choice of names on my part.

hehe :-)

Sigh! Scanning dyntick-idle state on a system with 256 CPUs. What could possibly go wrong? :-/

Thanx, Paul

maintainer
#46 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 5 lines · Paul E. McKenney
Quoted 2 lines

Now we could possibly add magic to rcu_read_unlock_special() to restart the tick in that case.

Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...

Uhm.. good point ;-) yes that will be somewhat detrimental to the overall OS health if we attempt such a thing.

#47 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 9 lines · Peter Zijlstra
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines

Now we could possibly add magic to rcu_read_unlock_special() to restart the tick in that case.

Not from NMI handlers we can't. Unless I am really confused about the code that restarts the tick. Which is not impossible, but ...

Uhm.. good point ;-) yes that will be somewhat detrimental to the overall OS health if we attempt such a thing.

But if you don't mind, I would very much like to use your question as a Quick Quiz. It should be a good one. ;-)

Thanx, Paul

maintainer
#48 lore raw

Re: [PATCH 05/32] nohz: Move rcu dynticks idle mode handling to idle enter/exit APIs

Quoted 3 lines · Paul E. McKenney

But if you don't mind, I would very much like to use your question as a Quick Quiz. It should be a good one. ;-)

No problem.. :-)

#49 lore raw

PATCH 6/32 nohz: Move idle ticks stats tracking out of nohz handlers

In reply to Frederic Weisbecker

Idle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |   34 +++++++++++++++++++---------------
 1 files changed, 19 insertions(+), 15 deletions(-)
kernel/time/tick-sched.c +19 -15
index 8937d4a..21b187c 100644
@@ -384,7 +384,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now)
 
 			ts->idle_tick = hrtimer_get_expires(&ts->sched_timer);
 			ts->tick_stopped = 1;
-			ts->idle_jiffies = last_jiffies;
 		}
 
 		ts->idle_sleeps++;
@@ -433,8 +432,10 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 	now = tick_nohz_start_idle(cpu, ts);
 	tick_nohz_stop_sched_tick(now);
 
-	if (!was_stopped && ts->tick_stopped)
+	if (!was_stopped && ts->tick_stopped) {
+		ts->idle_jiffies = ts->last_jiffies;
 		rcu_enter_nohz();
+	}
 }
 
 void tick_nohz_enter_idle(void)
@@ -498,16 +499,26 @@ static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
 static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)
 {
 	int cpu = smp_processor_id();
-#ifndef CONFIG_VIRT_CPU_ACCOUNTING
-	unsigned long ticks;
-#endif
 
 	/* Update jiffies first */
 	select_nohz_load_balancer(0);
 	tick_do_update_jiffies64(now);
 	cpumask_clear_cpu(cpu, nohz_cpu_mask);
 
+	touch_softlockup_watchdog();
+	/*
+	 * Cancel the scheduled timer and restore the tick
+	 */
+	ts->tick_stopped  = 0;
+	ts->idle_exittime = now;
+
+	tick_nohz_restart(ts, now);
+}
+
+static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
+{
 #ifndef CONFIG_VIRT_CPU_ACCOUNTING
+	unsigned long ticks;
 	/*
 	 * We stopped the tick in idle. Update process times would miss the
 	 * time we slept as update_process_times does only a 1 tick
@@ -520,15 +531,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)
 	if (ticks && ticks < LONG_MAX)
 		account_idle_ticks(ticks);
 #endif
-
-	touch_softlockup_watchdog();
-	/*
-	 * Cancel the scheduled timer and restore the tick
-	 */
-	ts->tick_stopped  = 0;
-	ts->idle_exittime = now;
-
-	tick_nohz_restart(ts, now);
 }
 
 void tick_nohz_exit_idle(void)
@@ -551,6 +553,7 @@ void tick_nohz_exit_idle(void)
 	if (ts->tick_stopped) {
 		rcu_exit_nohz();
 		tick_nohz_restart_sched_tick(now, ts);
+		tick_nohz_account_idle_ticks(ts);
 	}
 
 	local_irq_enable();
@@ -766,7 +769,8 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)
 		 */
 		if (ts->tick_stopped) {
 			touch_softlockup_watchdog();
-			ts->idle_jiffies++;
+			if (idle_cpu(cpu))
+				ts->idle_jiffies++;
 		}
 		update_process_times(user_mode(regs));
 		profile_tick(CPU_PROFILING);
Signature · 1 line
1.7.5.4
#50 lore raw

Re: [PATCH 06/32] nohz: Move idle ticks stats tracking out of nohz handlers

Quoted 3 lines · Frederic Weisbecker

Idle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.

Are you trying to say:

Currently idle time tracking is part of the nohz state tracking, separate this so that we can disable the tick while we're non-idle?

If so, how does idle time tracking work on a !NOHZ kernel, surely such things can be idle too..

#51 lore raw

Re: [PATCH 06/32] nohz: Move idle ticks stats tracking out of nohz handlers

Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

Idle ticks time tracking is merged into nohz stop/restart handlers. Pull it out into idle entry/exit handlers instead, so that nohz APIs is more idle independant.

Are you trying to say:

Currently idle time tracking is part of the nohz state tracking, separate this so that we can disable the tick while we're non-idle?

Right.

Quoted 3 lines

If so, how does idle time tracking work on a !NOHZ kernel, surely such things can be idle too..

In that case the tick calls account_process_tick() which delegates to account_idle_time() if we are idle on tick.

That's also used in NOHZ kernels in fact.

In every tick we call account_process_tick(), but when the tick is stopped we save its value in ts->idle_jiffies and when it is restarted we account jiffies - ts->idle_jiffies.

#52 lore raw

PATCH 7/32 nohz: Rename ts->idle_tick to ts->last_tick

In reply to Frederic Weisbecker

Now that idle and nohz logic are getting split, idle_tick becomes a misnomer when it takes a field name to save the last tick before switching to nohz.

Call it last_tick instead. This changes a bit the timer list stat export so we need to increase its version.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    8 ++++----
 kernel/time/tick-sched.c |    4 ++--
 kernel/time/timer_list.c |    4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)
include/linux/tick.h +4 -4
index 04f6418..849a0b2 100644
@@ -30,10 +30,10 @@ enum tick_nohz_mode {
  * struct tick_sched - sched tick emulation and no idle tick control/stats
  * @sched_timer:	hrtimer to schedule the periodic tick in high
  *			resolution mode
- * @idle_tick:		Store the last idle tick expiry time when the tick
- *			timer is modified for idle sleeps. This is necessary
+ * @last_tick:		Store the last tick expiry time when the tick
+ *			timer is modified for nohz sleeps. This is necessary
  *			to resume the tick timer operation in the timeline
- *			when the CPU returns from idle
+ *			when the CPU returns from nohz sleep.
  * @tick_stopped:	Indicator that the idle tick has been stopped
  * @idle_jiffies:	jiffies at the entry to idle for idle time accounting
  * @idle_calls:		Total number of idle calls
@@ -50,7 +50,7 @@ struct tick_sched {
 	struct hrtimer			sched_timer;
 	unsigned long			check_clocks;
 	enum tick_nohz_mode		nohz_mode;
-	ktime_t				idle_tick;
+	ktime_t				last_tick;
 	int				inidle;
 	int				tick_stopped;
 	unsigned long			idle_jiffies;
kernel/time/tick-sched.c +2 -2
index 21b187c..bca1519 100644
@@ -382,7 +382,7 @@ static void tick_nohz_stop_sched_tick(ktime_t now)
 		if (!ts->tick_stopped) {
 			select_nohz_load_balancer(1);
 
-			ts->idle_tick = hrtimer_get_expires(&ts->sched_timer);
+			ts->last_tick = hrtimer_get_expires(&ts->sched_timer);
 			ts->tick_stopped = 1;
 		}
 
@@ -468,7 +468,7 @@ ktime_t tick_nohz_get_sleep_length(void)
 static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
 {
 	hrtimer_cancel(&ts->sched_timer);
-	hrtimer_set_expires(&ts->sched_timer, ts->idle_tick);
+	hrtimer_set_expires(&ts->sched_timer, ts->last_tick);
 
 	while (1) {
 		/* Forward the time to expire in the future */
kernel/time/timer_list.c +2 -2
index 3258455..af5a7e9 100644
@@ -167,7 +167,7 @@ static void print_cpu(struct seq_file *m, int cpu, u64 now)
 	{
 		struct tick_sched *ts = tick_get_tick_sched(cpu);
 		P(nohz_mode);
-		P_ns(idle_tick);
+		P_ns(last_tick);
 		P(tick_stopped);
 		P(idle_jiffies);
 		P(idle_calls);
@@ -259,7 +259,7 @@ static int timer_list_show(struct seq_file *m, void *v)
 	u64 now = ktime_to_ns(ktime_get());
 	int cpu;
 
-	SEQ_printf(m, "Timer List Version: v0.6\n");
+	SEQ_printf(m, "Timer List Version: v0.7\n");
 	SEQ_printf(m, "HRTIMER_MAX_CLOCK_BASES: %d\n", HRTIMER_MAX_CLOCK_BASES);
 	SEQ_printf(m, "now at %Ld nsecs\n", (unsigned long long)now);
 
Signature · 1 line
1.7.5.4
#53 lore raw

PATCH 8/32 nohz: Move nohz load balancer selection into idle logic

In reply to Frederic Weisbecker

We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |    5 ++---
 1 files changed, 2 insertions(+), 3 deletions(-)
kernel/time/tick-sched.c +2 -3
index bca1519..de1b629 100644
@@ -380,8 +380,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now)
 		 * the scheduler tick in nohz_restart_sched_tick.
 		 */
 		if (!ts->tick_stopped) {
-			select_nohz_load_balancer(1);
-
 			ts->last_tick = hrtimer_get_expires(&ts->sched_timer);
 			ts->tick_stopped = 1;
 		}
@@ -434,6 +432,7 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 
 	if (!was_stopped && ts->tick_stopped) {
 		ts->idle_jiffies = ts->last_jiffies;
+		select_nohz_load_balancer(1);
 		rcu_enter_nohz();
 	}
 }
@@ -501,7 +500,6 @@ static void tick_nohz_restart_sched_tick(ktime_t now, struct tick_sched *ts)
 	int cpu = smp_processor_id();
 
 	/* Update jiffies first */
-	select_nohz_load_balancer(0);
 	tick_do_update_jiffies64(now);
 	cpumask_clear_cpu(cpu, nohz_cpu_mask);
 
@@ -552,6 +550,7 @@ void tick_nohz_exit_idle(void)
 
 	if (ts->tick_stopped) {
 		rcu_exit_nohz();
+		select_nohz_load_balancer(0);
 		tick_nohz_restart_sched_tick(now, ts);
 		tick_nohz_account_idle_ticks(ts);
 	}
Signature · 1 line
1.7.5.4
#54 lore raw

Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic

Quoted 2 lines · Frederic Weisbecker

We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.

Again, the important part is missing, why is this correct?

I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.

AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.

#55 lore raw

Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic

Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker

We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.

Again, the important part is missing, why is this correct?

Quoted 8 lines

I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.

AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.

Right, we have the support for trigger_load_balance() in scheduler_tick() that is still missing.

What about using that CPU that has to stay awake with a periodic tick to handle jiffies? We could force that CPU to be the idle load balancer. The problem is perhaps to find the right frequency for doing that because we have all the rq to handle.

#56 lore raw

Re: [PATCH 08/32] nohz: Move nohz load balancer selection into idle logic

Quoted 16 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Frederic Weisbecker

We want the nohz load balancer to be an idle CPU, thus move that selection to strict dyntick idle logic.

Again, the important part is missing, why is this correct?

Quoted 8 lines

I'm not at all convinced this is correct, suppose all your cpus (except the system CPU, which we'll assume has many tasks) are busy running 1 task. Then two of them get an extra task, now if those two happen to be SMT siblings you want the load-balancer to pull on task out from the SMT pair, however nobody is pulling since nobody is idle.

AFAICT this breaks stuff and the ILB needs some serious attention in order to fix this.

Quoted 7 lines

Right, we have the support for trigger_load_balance() in scheduler_tick() that is still missing.

What about using that CPU that has to stay awake with a periodic tick to handle jiffies? We could force that CPU to be the idle load balancer. The problem is perhaps to find the right frequency for doing that because we have all the rq to handle.

If this CPU can also be the RCU grace-period advancer of last resort, that would make it easier to arrive at an improved RCU_FAST_NO_HZ.

Thanx, Paul

#57 lore raw

PATCH 9/32 nohz: Move ts->idle_calls into strict idle logic

In reply to Frederic Weisbecker

Split the nohz switch in two parts, a first that checks if we can really stop the tick, and another that actually stop it. This way we can pull out idle_calls stat incrementation into strict idle logic.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |   87 ++++++++++++++++++++++++---------------------
 1 files changed, 46 insertions(+), 41 deletions(-)
kernel/time/tick-sched.c +46 -41
index de1b629..2794150 100644
@@ -249,48 +249,14 @@ EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);
  * Called either from the idle loop or from irq_exit() when an idle period was
  * just interrupted by an interrupt which did not cause a reschedule.
  */
-static void tick_nohz_stop_sched_tick(ktime_t now)
+static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *ts)
 {
 	unsigned long seq, last_jiffies, next_jiffies, delta_jiffies;
-	struct tick_sched *ts;
 	ktime_t last_update, expires;
 	struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
 	u64 time_delta;
-	int cpu;
-
-	cpu = smp_processor_id();
-	ts = &per_cpu(tick_cpu_sched, cpu);
-
-	/*
-	 * If this cpu is offline and it is the one which updates
-	 * jiffies, then give up the assignment and let it be taken by
-	 * the cpu which runs the tick timer next. If we don't drop
-	 * this here the jiffies might be stale and do_timer() never
-	 * invoked.
-	 */
-	if (unlikely(!cpu_online(cpu))) {
-		if (cpu == tick_do_timer_cpu)
-			tick_do_timer_cpu = TICK_DO_TIMER_NONE;
-	}
 
-	if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
-		return;
-
-	if (need_resched())
-		return;
-
-	if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
-		static int ratelimit;
-
-		if (ratelimit < 10) {
-			printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
-			       (unsigned int) local_softirq_pending());
-			ratelimit++;
-		}
-		return;
-	}
 
-	ts->idle_calls++;
 	/* Read jiffies and the time when jiffies were updated last */
 	do {
 		seq = read_seqbegin(&xtime_lock);
@@ -422,18 +388,57 @@ out:
 	ts->sleep_length = ktime_sub(dev->next_event, now);
 }
 
+static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
+{
+	/*
+	 * If this cpu is offline and it is the one which updates
+	 * jiffies, then give up the assignment and let it be taken by
+	 * the cpu which runs the tick timer next. If we don't drop
+	 * this here the jiffies might be stale and do_timer() never
+	 * invoked.
+	 */
+	if (unlikely(!cpu_online(cpu))) {
+		if (cpu == tick_do_timer_cpu)
+			tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+	}
+
+	if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
+		return false;
+
+	if (need_resched())
+		return false;
+
+	if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+		static int ratelimit;
+
+		if (ratelimit < 10) {
+			printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+			       (unsigned int) local_softirq_pending());
+			ratelimit++;
+		}
+		return false;
+	}
+
+	return true;
+}
+
 static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 {
 	ktime_t now;
-	int was_stopped = ts->tick_stopped;
 
 	now = tick_nohz_start_idle(cpu, ts);
-	tick_nohz_stop_sched_tick(now);
 
-	if (!was_stopped && ts->tick_stopped) {
-		ts->idle_jiffies = ts->last_jiffies;
-		select_nohz_load_balancer(1);
-		rcu_enter_nohz();
+	if (tick_nohz_can_stop_tick(cpu, ts)) {
+		int was_stopped = ts->tick_stopped;
+
+		ts->idle_calls++;
+		tick_nohz_stop_sched_tick(now, cpu, ts);
+
+		if (!was_stopped && ts->tick_stopped) {
+			ts->idle_jiffies = ts->last_jiffies;
+			select_nohz_load_balancer(1);
+			rcu_enter_nohz();
+		}
 	}
 }
 
Signature · 1 line
1.7.5.4
#58 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23 -0
+static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
+{
+       /*
+        * If this cpu is offline and it is the one which updates
+        * jiffies, then give up the assignment and let it be taken by
+        * the cpu which runs the tick timer next. If we don't drop
+        * this here the jiffies might be stale and do_timer() never
+        * invoked.
+        */
+       if (unlikely(!cpu_online(cpu))) {
+               if (cpu == tick_do_timer_cpu)
+                       tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+       }
+
+       if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
+               return false;
+
+       if (need_resched())
+               return false;
+
+       if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+               static int ratelimit;
+

kernel/time/tick-sched.c out: · line 423

Quoted 10 lines
kernel/time/tick-sched.c +10 -0
+               if (ratelimit < 10) {
+                       printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+                              (unsigned int) local_softirq_pending());
+                       ratelimit++;
+               }
+               return false;
+       }
+
+       return true;
+} 

Why aren't rcu_needs_cpu(), printk_needs_cpu() and arch_needs_cpu() not in there?

That are typical 'can we go sleep now?' functions.

#59 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23 -0
+static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
+{
+       /*
+        * If this cpu is offline and it is the one which updates
+        * jiffies, then give up the assignment and let it be taken by
+        * the cpu which runs the tick timer next. If we don't drop
+        * this here the jiffies might be stale and do_timer() never
+        * invoked.
+        */
+       if (unlikely(!cpu_online(cpu))) {
+               if (cpu == tick_do_timer_cpu)
+                       tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+       }
+
+       if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
+               return false;
+
+       if (need_resched())
+               return false;
+
+       if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+               static int ratelimit;
+

kernel/time/tick-sched.c out: · line 423

Quoted 10 lines
kernel/time/tick-sched.c +10 -0
+               if (ratelimit < 10) {
+                       printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+                              (unsigned int) local_softirq_pending());
+                       ratelimit++;
+               }
+               return false;
+       }
+
+       return true;
+} 

Why aren't rcu_needs_cpu(), printk_needs_cpu() and arch_needs_cpu() not in there?

Quoted 1 line

That are typical 'can we go sleep now?' functions.

Because when one of these functions are positive, the ts->next_jiffies and ts->last_jiffies stats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.

We still want tick_nohz_stop_sched_tick() to have the *needs_cpu() checks so that they can restore a HZ periodic behaviour on interrupt return if needed.

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

#60 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 39 lines · Frederic Weisbecker
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23 -0
+static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
+{
+       /*
+        * If this cpu is offline and it is the one which updates
+        * jiffies, then give up the assignment and let it be taken by
+        * the cpu which runs the tick timer next. If we don't drop
+        * this here the jiffies might be stale and do_timer() never
+        * invoked.
+        */
+       if (unlikely(!cpu_online(cpu))) {
+               if (cpu == tick_do_timer_cpu)
+                       tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+       }
+
+       if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
+               return false;
+
+       if (need_resched())
+               return false;
+
+       if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+               static int ratelimit;
+

kernel/time/tick-sched.c out: · line 423

Quoted 10 lines
kernel/time/tick-sched.c +10 -0
+               if (ratelimit < 10) {
+                       printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+                              (unsigned int) local_softirq_pending());
+                       ratelimit++;
+               }
+               return false;
+       }
+
+       return true;
+} 

Why aren't rcu_needs_cpu(), printk_needs_cpu() and arch_needs_cpu() not in there?

Quoted 5 lines
Quoted 1 line

That are typical 'can we go sleep now?' functions.

Because when one of these functions are positive, the ts->next_jiffies and ts->last_jiffies stats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.

Then call it tick_nohz_can_stop_tick_idle() or so, and create tick_nohz_can_stop_tick() to deal with all stuff.

Quoted 3 lines

We still want tick_nohz_stop_sched_tick() to have the *needs_cpu() checks so that they can restore a HZ periodic behaviour on interrupt return if needed.

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

#61 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 40 lines · Peter Zijlstra
Quoted 39 lines · Frederic Weisbecker
Quoted 38 lines · Peter Zijlstra
Quoted 23 lines · Frederic Weisbecker
kernel/time/tick-sched.c +23 -0
+static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
+{
+       /*
+        * If this cpu is offline and it is the one which updates
+        * jiffies, then give up the assignment and let it be taken by
+        * the cpu which runs the tick timer next. If we don't drop
+        * this here the jiffies might be stale and do_timer() never
+        * invoked.
+        */
+       if (unlikely(!cpu_online(cpu))) {
+               if (cpu == tick_do_timer_cpu)
+                       tick_do_timer_cpu = TICK_DO_TIMER_NONE;
+       }
+
+       if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE))
+               return false;
+
+       if (need_resched())
+               return false;
+
+       if (unlikely(local_softirq_pending() && cpu_online(cpu))) {
+               static int ratelimit;
+

kernel/time/tick-sched.c out: · line 423

Quoted 10 lines
kernel/time/tick-sched.c +10 -0
+               if (ratelimit < 10) {
+                       printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+                              (unsigned int) local_softirq_pending());
+                       ratelimit++;
+               }
+               return false;
+       }
+
+       return true;
+} 

Why aren't rcu_needs_cpu(), printk_needs_cpu() and arch_needs_cpu() not in there?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

That are typical 'can we go sleep now?' functions.

Because when one of these functions are positive, the ts->next_jiffies and ts->last_jiffies stats are updated. Not with the above.
Also I start to think the above checks are only useful in the idle case.

Then call it tick_nohz_can_stop_tick_idle() or so, and create tick_nohz_can_stop_tick() to deal with all stuff.

Yeah I need to have a deeper look into these checks.

Quoted 8 lines
Quoted 4 lines

We still want tick_nohz_stop_sched_tick() to have the *needs_cpu() checks so that they can restore a HZ periodic behaviour on interrupt return if needed.

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.

I don't see a good reason to duplicate that logic with a pure restart from the IPI.

Quoted 7 lines
Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...

#62 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.

Quoted 2 lines

I don't see a good reason to duplicate that logic with a pure restart from the IPI.

That sounds like an optimization, and should thus be done later.

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...

softirqs shouldn't be pending when you go into nohz mode..

That is, I'm really not seeing what's wrong with the very simple:

if (tick_nohz_can_stop_tick())

	tick_nohz_stop_tick();

and relying on everybody who invalidates tick_nohz_can_stop_tick(), to do:

  tick_nohz_start_tick();

I'm also not quite sure why you always IPI, is that to avoid lock inversions?

#63 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.

Quoted 4 lines
Quoted 2 lines

I don't see a good reason to duplicate that logic with a pure restart from the IPI.

That sounds like an optimization, and should thus be done later.

The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

Quoted 5 lines
Quoted 3 lines

Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...

softirqs shouldn't be pending when you go into nohz mode..

You mean it can't happen or we don't want that to happen?

Quoted 4 lines

That is, I'm really not seeing what's wrong with the very simple:

Quoted 8 lines

if (tick_nohz_can_stop_tick())

	tick_nohz_stop_tick();

and relying on everybody who invalidates tick_nohz_can_stop_tick(), to do:

  tick_nohz_start_tick();

May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.

Quoted 3 lines

I'm also not quite sure why you always IPI, is that to avoid lock inversions?

Exactly! I think I wrote that to some changelog but I'm not sure. I'll check that.

#64 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 15 lines · Frederic Weisbecker
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Frederic Weisbecker
Quoted 3 lines

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't see a good reason to duplicate that logic with a pure restart from the IPI.

That sounds like an optimization, and should thus be done later.

The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.

care to point me to the relevant code, because I can't remember a half-assed nohz state.. then again, maybe I didn't look hard enough.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...

softirqs shouldn't be pending when you go into nohz mode..

You mean it can't happen or we don't want that to happen?

We don't want that.. going into nohz with pending softirqs means the softirqs will be delayed for an unknown amount of time, this should not occur.

tick_nohz_stop_sched_tick() has:

        if (unlikely(local_softirq_pending() && cpu_online(cpu))) {                          
                static int ratelimit;                                                        
                                                                                             
                if (ratelimit < 10) {                                                        
                        printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",                
                               (unsigned int) local_softirq_pending());                      
                        ratelimit++;                                                         
                }                                                                            
                goto end;                                                                    
        }

which should warn us if this ever was to occur.

Quoted 8 lines
Quoted 8 lines

That is, I'm really not seeing what's wrong with the very simple:

if (tick_nohz_can_stop_tick())

	tick_nohz_stop_tick();
Quoted 7 lines
Quoted 4 lines

and relying on everybody who invalidates tick_nohz_can_stop_tick(), to do:

  tick_nohz_start_tick();

May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.

How exactly do idle and non-idle differ? Its about stopping the tick, regardless of if we're idle or not if someone needs the thing we need to start it again.

#65 lore raw

Re: [PATCH 09/32] nohz: Move ts->idle_calls into strict idle logic

Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
Quoted 17 lines · Peter Zijlstra
Quoted 14 lines · Frederic Weisbecker
Quoted 3 lines

Well, no, on interrupt return you shouldn't do anything. If you've stopped the tick it stays stopped until you do something that needs it, then that action will re-enable it.

Sure, when something needs the tick in this mode, we usually receive an IPI and restart the tick from there but then tick_nohz_stop_sched_tick() handles the cases with *needs_cpu() very well on interrupt return (our IPI return) by doing a kind of "light" HZ mode by logically switching to nohz mode but with the next timer happening in HZ, assuming it's a matter of one tick and we will switch to a real nohz behaviour soon.

I don't see a good reason to duplicate that logic with a pure restart from the IPI.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

That sounds like an optimization, and should thus be done later.

The optimization is already there upstream. I can split the logic for non-idle case but I'm not sure about the point of that.

care to point me to the relevant code, because I can't remember a half-assed nohz state.. then again, maybe I didn't look hard enough.

It's in tick_nohz_stop_sched_tick():

	if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
	    arch_needs_cpu(cpu)) {
		next_jiffies = last_jiffies + 1;
		delta_jiffies = 1;
	} else {
		/* Get the next timer wheel timer */
		next_jiffies = get_next_timer_interrupt(last_jiffies);
		delta_jiffies = next_jiffies - last_jiffies;
	}

There are two cases:

- tick_nohz_stop_sched_tick() is called from idle. If one of the *_needs_cpu() is true, then the tick is not stopped. It doesn't even enter the nohz logic.

- tick_nohz_stop_sched_tick() is called from an interrupt exit while we are idle. If one of the *_needs_cpu() is true, then if the tick was previously stopped, we program the next timer to be in 1 jiffy but without exiting nohz mode. Otherwise if the tick was disabled, then we don't stop the tick.

In fact those *_needs_cpu() are treated like timer list timer or hrtimers that happen in one jiffy from now.

When we are in such a scheme:

	tick_nohz_stop_sched_tick();
	while (!need_resched())
		hlt();
	tick_nohz_restart_sched_tick();

It avoids to go back and forth between hz/nohz logic from inside the loop.

Quoted 33 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

That said I wonder if some of the above conditions should restore a periodic behaviour on interrupt return...

I would expect the tick not to be stopped when tick_nohz_can_stop_tick() returns false. If it returns true, then I expect anything that needs it to re-enable it.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Yeah. In the case of need_resched() in idle I believe the CPU doesn't really go to sleep later so it should be fine. But for the case of softirq pending or nohz_mode, I'm not sure...

softirqs shouldn't be pending when you go into nohz mode..

You mean it can't happen or we don't want that to happen?

We don't want that.. going into nohz with pending softirqs means the softirqs will be delayed for an unknown amount of time, this should not occur.

tick_nohz_stop_sched_tick() has:

        if (unlikely(local_softirq_pending() && cpu_online(cpu))) {                          
                static int ratelimit;                                                        
                                                                                             
                if (ratelimit < 10) {                                                        
                        printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",                
                               (unsigned int) local_softirq_pending());                      
                        ratelimit++;                                                         
                }                                                                            
                goto end;                                                                    
        }
Quoted 1 line

which should warn us if this ever was to occur.

Yeah. Moreover that check is weird because if softirqs are pending at that time, then the softirqd thread should be woken up. Because either we come from idle and we are not in an interrupt (softirq raised would use softirqd in not interrupt exit) or we come from an interrupt and the softirqs have been executed. If there were too much of them, softirqd would have been woken up as well.

Thus the need_resched() condition that comes before should have exited the function already.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

That is, I'm really not seeing what's wrong with the very simple:

if (tick_nohz_can_stop_tick())

	tick_nohz_stop_tick();

and relying on everybody who invalidates tick_nohz_can_stop_tick(), to do:

Quoted 8 lines
Quoted 4 lines
Quoted 1 line
  tick_nohz_start_tick();

May be for the non-idle case. But for the idle case I need to ensure this is necessary somewhere.

How exactly do idle and non-idle differ? Its about stopping the tick, regardless of if we're idle or not if someone needs the thing we need to start it again.

It may differ because of the way the idle loop is made, considering some things as 1 jiffy timer to avoid a hard whole tick restart.

#66 lore raw

PATCH 10/32 nohz: Move next idle expiring time record into idle logic area

In reply to Frederic Weisbecker

Move next idle expiry time record and idle sleeps tracking into idle entry functions as they are not generic nohz stats.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |   21 ++++++++++++---------
 1 files changed, 12 insertions(+), 9 deletions(-)
kernel/time/tick-sched.c +12 -9
index 2794150..f5e12da 100644
@@ -249,10 +249,10 @@ EXPORT_SYMBOL_GPL(get_cpu_iowait_time_us);
  * Called either from the idle loop or from irq_exit() when an idle period was
  * just interrupted by an interrupt which did not cause a reschedule.
  */
-static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *ts)
+static ktime_t tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *ts)
 {
 	unsigned long seq, last_jiffies, next_jiffies, delta_jiffies;
-	ktime_t last_update, expires;
+	ktime_t last_update, expires, ret = { .tv64 = 0 };
 	struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
 	u64 time_delta;
 
@@ -338,6 +338,8 @@ static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *t
 		if (ts->tick_stopped && ktime_equal(expires, dev->next_event))
 			goto out;
 
+		ret = expires;
+
 		/*
 		 * nohz_stop_sched_tick can be called several times before
 		 * the nohz_restart_sched_tick is called. This happens when
@@ -350,11 +352,6 @@ static void tick_nohz_stop_sched_tick(ktime_t now, int cpu, struct tick_sched *t
 			ts->tick_stopped = 1;
 		}
 
-		ts->idle_sleeps++;
-
-		/* Mark expires */
-		ts->idle_expires = expires;
-
 		/*
 		 * If the expiration time == KTIME_MAX, then
 		 * in this case we simply stop the tick timer.
@@ -386,6 +383,8 @@ out:
 	ts->next_jiffies = next_jiffies;
 	ts->last_jiffies = last_jiffies;
 	ts->sleep_length = ktime_sub(dev->next_event, now);
+
+	return ret;
 }
 
 static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
@@ -424,7 +423,7 @@ static bool tick_nohz_can_stop_tick(int cpu, struct tick_sched *ts)
 
 static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 {
-	ktime_t now;
+	ktime_t now, expires;
 
 	now = tick_nohz_start_idle(cpu, ts);
 
@@ -432,7 +431,11 @@ static void __tick_nohz_enter_idle(struct tick_sched *ts, int cpu)
 		int was_stopped = ts->tick_stopped;
 
 		ts->idle_calls++;
-		tick_nohz_stop_sched_tick(now, cpu, ts);
+		expires = tick_nohz_stop_sched_tick(now, cpu, ts);
+		if (expires.tv64 > 0LL) {
+			ts->idle_sleeps++;
+			ts->idle_expires = expires;
+		}
 
 		if (!was_stopped && ts->tick_stopped) {
 			ts->idle_jiffies = ts->last_jiffies;
Signature · 1 line
1.7.5.4
#67 lore raw

PATCH 11/32 cpuset: Set up interface for nohz flag

In reply to Frederic Weisbecker

Prepare the interface to implement the nohz cpuset flag. This flag, once set, will tell the system to try to shutdown the periodic timer tick when possible.

We use here a per cpu refcounter. As long as a CPU is contained into at least one cpuset that has the nohz flag set, it is part of the set of CPUs that run into adaptive nohz mode.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 arch/Kconfig           |    3 ++
 include/linux/cpuset.h |   22 ++++++++++++++++++++
 init/Kconfig           |    8 +++++++
 kernel/cpuset.c        |   52 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 85 insertions(+), 0 deletions(-)
arch/Kconfig +3 -0
index 341ac95..5fe21c4 100644
@@ -172,6 +172,9 @@ config HAVE_ARCH_JUMP_LABEL
 	bool
 
 config HAVE_ARCH_MUTEX_CPU_RELAX
+       bool
+
+config HAVE_CPUSETS_NO_HZ
 	bool
 
 config HAVE_RCU_TABLE_FREE
include/linux/cpuset.h +22 -0
index e9eaec5..62e5d5a 100644
@@ -244,4 +244,26 @@ static inline void put_mems_allowed(void)
 
 #endif /* !CONFIG_CPUSETS */
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+
+DECLARE_PER_CPU(int, cpu_adaptive_nohz_ref);
+
+static inline bool cpuset_cpu_adaptive_nohz(int cpu)
+{
+	if (per_cpu(cpu_adaptive_nohz_ref, cpu) > 0)
+		return true;
+
+	return false;
+}
+
+static inline bool cpuset_adaptive_nohz(void)
+{
+	if (__get_cpu_var(cpu_adaptive_nohz_ref) > 0)
+		return true;
+
+	return false;
+}
+
+#endif /* CONFIG_CPUSETS_NO_HZ */
+
 #endif /* _LINUX_CPUSET_H */
init/Kconfig +8 -0
index 3cf7855..0cb591a 100644
@@ -624,6 +624,14 @@ config PROC_PID_CPUSET
 	depends on CPUSETS
 	default y
 
+config CPUSETS_NO_HZ
+       bool "Tickless cpusets"
+       depends on CPUSETS && HAVE_CPUSETS_NO_HZ
+       help
+         This options let you apply a nohz property to a cpuset such
+	 that the periodic timer tick tries to be avoided when possible on
+	 the concerned CPUs.
+
 config CGROUP_CPUACCT
 	bool "Simple CPU accounting cgroup subsystem"
 	help
kernel/cpuset.c +52 -0

100 lines not on this page · raw message

Signature · 1 line
1.7.5.4
#68 lore raw

PATCH 12/32 nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

In reply to Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/time/tick-sched.c |   52 ++++++++++++++++++++++++++++++++++++---------
 1 files changed, 41 insertions(+), 11 deletions(-)
kernel/time/tick-sched.c +41 -11
index f5e12da..5f41ef7 100644
@@ -20,6 +20,7 @@
 #include <linux/profile.h>
 #include <linux/sched.h>
 #include <linux/module.h>
+#include <linux/cpuset.h>
 
 #include <asm/irq_regs.h>
 
@@ -729,6 +730,45 @@ void tick_check_idle(int cpu)
 	tick_check_nohz(cpu);
 }
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+
+/*
+ * Take the timer duty if nobody is taking care of it.
+ * If a CPU already does and and it's in a nohz cpuset,
+ * then take the charge so that it can switch to nohz mode.
+ */
+static void tick_do_timer_check_handler(int cpu)
+{
+	int handler = tick_do_timer_cpu;
+
+	if (unlikely(handler == TICK_DO_TIMER_NONE)) {
+		tick_do_timer_cpu = cpu;
+	} else {
+		if (!cpuset_adaptive_nohz() &&
+		    cpuset_cpu_adaptive_nohz(handler))
+			tick_do_timer_cpu = cpu;
+	}
+}
+
+#else
+
+static void tick_do_timer_check_handler(int cpu)
+{
+#ifdef CONFIG_NO_HZ
+	/*
+	 * Check if the do_timer duty was dropped. We don't care about
+	 * concurrency: This happens only when the cpu in charge went
+	 * into a long sleep. If two cpus happen to assign themself to
+	 * this duty, then the jiffies update is still serialized by
+	 * xtime_lock.
+	 */
+	if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE))
+		tick_do_timer_cpu = cpu;
+#endif
+}
+
+#endif /* CONFIG_CPUSETS_NO_HZ */
+
 /*
  * High resolution timer specific code
  */
@@ -745,17 +785,7 @@ static enum hrtimer_restart tick_sched_timer(struct hrtimer *timer)
 	ktime_t now = ktime_get();
 	int cpu = smp_processor_id();
 
-#ifdef CONFIG_NO_HZ
-	/*
-	 * Check if the do_timer duty was dropped. We don't care about
-	 * concurrency: This happens only when the cpu in charge went
-	 * into a long sleep. If two cpus happen to assign themself to
-	 * this duty, then the jiffies update is still serialized by
-	 * xtime_lock.
-	 */
-	if (unlikely(tick_do_timer_cpu == TICK_DO_TIMER_NONE))
-		tick_do_timer_cpu = cpu;
-#endif
+	tick_do_timer_check_handler(cpu);
 
 	/* Check, if the jiffies need an update */
 	if (tick_do_timer_cpu == cpu)
Signature · 1 line
1.7.5.4
#69 lore raw

Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

Quoted 4 lines · Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

You and Dmitiri might want to get together:

lkml.kernel.org/r/[email protected]

#70 lore raw

Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

You and Dmitiri might want to get together:

lkml.kernel.org/r/[email protected]

Right!

There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the do_timer duty and adaptive nohz CPUs run with a stale jiffies and walltime.

I need to handle that.

#71 lore raw

Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

You and Dmitiri might want to get together:

lkml.kernel.org/r/[email protected]

Quoted 7 lines

Right!

There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the do_timer duty and adaptive nohz CPUs run with a stale jiffies and walltime.

I need to handle that.

Frederic,

Please note that the patch that Peter references (lkml.kernel.org/r/[email protected]) supports only the nohz=off case.

#72 lore raw

Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

In reply to Frederic Weisbecker
Quoted 9 lines · Frederic Weisbecker
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

You and Dmitiri might want to get together:

Quoted 7 lines
Quoted 1 line

lkml.kernel.org/r/[email protected]

Right!

There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the do_timer duty and adaptive nohz CPUs run with a stale jiffies and walltime.

Doesn't nohz already deal with the case of all cpus being idle? In that case the cpu that wakes up first gets to play catch up on irq_enter() or so.

#73 lore raw

Re: [PATCH 12/32] nohz: Try not to give the timekeeping duty to a cpuset nohz cpu

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Frederic Weisbecker
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

Try to give the timekeeing duty to a CPU that doesn't belong to any nohz cpuset when possible, so that we increase the chance for these nohz cpusets to run their CPUs out of periodic tick mode.

You and Dmitiri might want to get together:

lkml.kernel.org/r/[email protected]

Right!

Quoted 7 lines
Quoted 3 lines

There is another missing piece in my patchset. If every non adaptive-nohz CPUs are sleeping, then none is handling the do_timer duty and adaptive nohz CPUs run with a stale jiffies and walltime.

Doesn't nohz already deal with the case of all cpus being idle? In that case the cpu that wakes up first gets to play catch up on irq_enter() or so.

Sure and that works for the nohz idle case. But that's not enough anymore in the case of adaptive nohz CPUs. They can run for a while without the tick and if nobody else maintains a tick either then jiffies and walltime are not maintained anymore.

#74 lore raw

PATCH 13/32 nohz: Adaptive tick stop and restart on nohz cpuset

In reply to Frederic Weisbecker

When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.

Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.

This assumes we are using TTWU_QUEUE sched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.

I also need to look at the ARCH_WANT_INTERRUPTS_ON_CTXW case and the remote wakeups.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/cpuset.h   |    4 +++
 include/linux/sched.h    |    6 ++++
 include/linux/tick.h     |   12 ++++++++-
 init/Kconfig             |    2 +-
 kernel/sched.c           |   35 +++++++++++++++++++++++++
 kernel/softirq.c         |    4 +-
 kernel/time/tick-sched.c |   63 ++++++++++++++++++++++++++++++++++++++-------
 7 files changed, 112 insertions(+), 14 deletions(-)
include/linux/cpuset.h +4 -0
index 62e5d5a..799b9a4 100644
@@ -264,6 +264,10 @@ static inline bool cpuset_adaptive_nohz(void)
 	return false;
 }
 
+extern void cpuset_update_nohz(void);
+#else
+static inline void cpuset_update_nohz(void) { }
+
 #endif /* CONFIG_CPUSETS_NO_HZ */
 
 #endif /* _LINUX_CPUSET_H */
include/linux/sched.h +6 -0
index dbe021a..53a95b5 100644
@@ -2652,6 +2652,12 @@ static inline void inc_syscw(struct task_struct *tsk)
 #define TASK_SIZE_OF(tsk)	TASK_SIZE
 #endif
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+extern bool cpuset_nohz_can_stop_tick(void);
+#else
+static inline bool cpuset_nohz_can_stop_tick(void) { return false; }
+#endif
+
 #ifdef CONFIG_MM_OWNER
 extern void mm_update_next_owner(struct mm_struct *mm);
 extern void mm_init_owner(struct mm_struct *mm, struct task_struct *p);
include/linux/tick.h +11 -1
index 849a0b2..cc4880e 100644
@@ -122,11 +122,21 @@ static inline int tick_oneshot_mode_active(void) { return 0; }
 # ifdef CONFIG_NO_HZ
 extern void tick_nohz_enter_idle(void);
 extern void tick_nohz_exit_idle(void);
+extern void tick_nohz_restart_sched_tick(void);
 extern void tick_nohz_irq_exit(void);
 extern ktime_t tick_nohz_get_sleep_length(void);
 extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
 extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
-# else
+
+#ifdef CONFIG_CPUSETS_NO_HZ
+DECLARE_PER_CPU(int, task_nohz_mode);
+
+extern int tick_nohz_adaptive_mode(void);
+#else /* !CPUSETS_NO_HZ */
+static inline int tick_nohz_adaptive_mode(void) { return 0; }
+#endif /* CPUSETS_NO_HZ */
+
+# else /* !NO_HZ */
 static inline void tick_nohz_enter_idle(void) { }
 static inline void tick_nohz_exit_idle(void) { }
 
init/Kconfig +1 -1
index 0cb591a..7a144ad 100644
@@ -626,7 +626,7 @@ config PROC_PID_CPUSET
 
 config CPUSETS_NO_HZ
        bool "Tickless cpusets"
-       depends on CPUSETS && HAVE_CPUSETS_NO_HZ
+       depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS
        help
          This options let you apply a nohz property to a cpuset such
 	 that the periodic timer tick tries to be avoided when possible on
kernel/sched.c +35 -0
index 609a867..0e1aa4e 100644
@@ -2433,6 +2433,38 @@ static void update_avg(u64 *avg, u64 sample)
 }
 #endif
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+DEFINE_PER_CPU(int, task_nohz_mode);
+
+bool cpuset_nohz_can_stop_tick(void)
+{
+	struct rq *rq;
+
+	rq = this_rq();
+
+	/* More than one running task need preemption */
+	if (rq->nr_running > 1)
+		return false;
+
+	return true;
+}
+
+static void cpuset_nohz_restart_tick(void)
+{
+	__get_cpu_var(task_nohz_mode) = 0;
+	tick_nohz_restart_sched_tick();
+}
+
+void cpuset_update_nohz(void)
+{
+	if (!tick_nohz_adaptive_mode())
+		return;
+
+	if (!cpuset_nohz_can_stop_tick())
+		cpuset_nohz_restart_tick();
+}
+#endif
+
 static void
 ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
 {
@@ -2560,6 +2592,8 @@ static void sched_ttwu_pending(void)
 		ttwu_do_activate(rq, p, 0);
 	}
 
+	cpuset_update_nohz();
+
 	raw_spin_unlock(&rq->lock);
 }
 
@@ -2620,6 +2654,7 @@ static void ttwu_queue(struct task_struct *p, int cpu)
 
 	raw_spin_lock(&rq->lock);
 	ttwu_do_activate(rq, p, 0);
+	cpuset_update_nohz();
 	raw_spin_unlock(&rq->lock);
 }
 
kernel/softirq.c +2 -2
index 67a1401..2dbeeb9 100644
@@ -297,7 +297,7 @@ void irq_enter(void)
 	int cpu = smp_processor_id();
 
 	rcu_irq_enter();
-	if (idle_cpu(cpu) && !in_interrupt()) {
+	if ((idle_cpu(cpu) || tick_nohz_adaptive_mode()) && !in_interrupt()) {
 		/*
 		 * Prevent raise_softirq from needlessly waking up ksoftirqd
 		 * here, as softirq will be serviced on return from interrupt.
@@ -342,7 +342,7 @@ void irq_exit(void)
 	rcu_irq_exit();
 #ifdef CONFIG_NO_HZ
 	/* Make sure that timer wheel updates are propagated */
-	if (idle_cpu(smp_processor_id()) && !in_interrupt() && !need_resched())
+	if (!in_interrupt())
 		tick_nohz_irq_exit();
 #endif
 	preempt_enable_no_resched();
kernel/time/tick-sched.c +53 -10

113 lines not on this page · raw message

Signature · 1 line
1.7.5.4
#75 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 7 lines · Frederic Weisbecker

When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.

Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.

Shouldn't you first put the syscall hooks in place before allowing the tick to be switched off? It seems this patch is somewhat too early in the series.

Quoted 5 lines

This assumes we are using TTWU_QUEUE sched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.

We could certainly remove that feature flag and always use it, it was mostly a transition debug switch in case something didn't work or performance issues were found due to this.

Quoted 2 lines

I also need to look at the ARCH_WANT_INTERRUPTS_ON_CTXW case and the remote wakeups.

Well, ideally we'd simply get rid of that, rmk has some preliminary patches in that direction.

#76 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

When a CPU is included in a nohz cpuset, try to switch it to nohz mode from the timer interrupt if it is the only non-idle task running.

Quoted 7 lines
Quoted 3 lines

Then restart the tick if necessary from the wakeup path if we are enqueuing a second task while the timer is stopped, so that the scheduler tick is rearmed.

Shouldn't you first put the syscall hooks in place before allowing the tick to be switched off? It seems this patch is somewhat too early in the series.

I don't think it's necessary, that part doesn't depend on userspace hooks. The whole thing is enabled very late anyway.

Quoted 9 lines
Quoted 5 lines

This assumes we are using TTWU_QUEUE sched feature so I need to handle the off case (or actually not handle it but properly), because we need the adaptive tick restart and what will come along in further patches to be done locally and before the new task ever gets scheduled.

We could certainly remove that feature flag and always use it, it was mostly a transition debug switch in case something didn't work or performance issues were found due to this.

Ok, good.

Quoted 5 lines
Quoted 2 lines

I also need to look at the ARCH_WANT_INTERRUPTS_ON_CTXW case and the remote wakeups.

Well, ideally we'd simply get rid of that, rmk has some preliminary patches in that direction.

Great!

Thanks.

#77 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

In reply to Frederic Weisbecker

kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450

Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12 -0
+bool cpuset_nohz_can_stop_tick(void)
+{
+       struct rq *rq;
+
+       rq = this_rq();
+
+       /* More than one running task need preemption */
+       if (rq->nr_running > 1)
+               return false;
+
+       return true;
+} 

int sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.

#78 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 16 lines · Peter Zijlstra

kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450

Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12 -0
+bool cpuset_nohz_can_stop_tick(void)
+{
+       struct rq *rq;
+
+       rq = this_rq();
+
+       /* More than one running task need preemption */
+       if (rq->nr_running > 1)
+               return false;
+
+       return true;
+} 

int sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.

tick_nohz_stop_sched_tick() already handles that by keeping a periodic behaviour if one of these conditions are met.

It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.

#79 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 18 lines · Frederic Weisbecker
Quoted 16 lines · Peter Zijlstra

kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450

Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12 -0
+bool cpuset_nohz_can_stop_tick(void)
+{
+       struct rq *rq;
+
+       rq = this_rq();
+
+       /* More than one running task need preemption */
+       if (rq->nr_running > 1)
+               return false;
+
+       return true;
+} 

int sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.

Quoted 2 lines

tick_nohz_stop_sched_tick() already handles that by keeping a periodic behaviour if one of these conditions are met.

What? tick_nohz_stop_sched_tick() most surely cannot access struct rq, so it cannot do the nr_running test.

Quoted 2 lines

It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.

Again, what?

#80 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Frederic Weisbecker
Quoted 16 lines · Peter Zijlstra

kernel/sched.c static void update_avg(u64 *avg, u64 sample) · line 2450

Quoted 12 lines · Frederic Weisbecker
kernel/sched.c +12 -0
+bool cpuset_nohz_can_stop_tick(void)
+{
+       struct rq *rq;
+
+       rq = this_rq();
+
+       /* More than one running task need preemption */
+       if (rq->nr_running > 1)
+               return false;
+
+       return true;
+} 

int sched_needs_cpu(int cpu), seems the right name, matches the existing {rcu,printk,arch}_needs_cpu() functions.

Quoted 5 lines
Quoted 2 lines

tick_nohz_stop_sched_tick() already handles that by keeping a periodic behaviour if one of these conditions are met.

What? tick_nohz_stop_sched_tick() most surely cannot access struct rq, so it cannot do the nr_running test.

I was talking about {rcu,printk,arch}_needs_cpu() functions.

Quoted 5 lines
Quoted 3 lines

It has also the upside to restore the periodic behaviour if needed from irq return if the tick was stopped.

Again, what?

If the tick is stopped then an irq fires and something calls printk() or call_rcu() then on interrupt return, tick_nohz_stop_sched_tick() checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.

#81 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 3 lines · Frederic Weisbecker

tick_nohz_stop_sched_tick() checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.

tick_nohz_stop_sched_tick() should not restore stuff, it should at worst fail to stop but never enable that's just weird.

#82 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

tick_nohz_stop_sched_tick() checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.

tick_nohz_stop_sched_tick() should not restore stuff, it should at worst fail to stop but never enable that's just weird.

It's not really enablement, it's just nohz behaviour but the next tick is in HZ :o)

Like you said before it's an optimization. I can do the things differently for idle and non-idle cases there but I'm not sure it's really a good thing.

#83 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

Quoted 6 lines · Frederic Weisbecker
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker

tick_nohz_stop_sched_tick() checks that
with {rcu,printk,arch}_needs_cpu() and restores a periodic behaviour until nobody else needs the CPU.

Quoted 7 lines
Quoted 2 lines

tick_nohz_stop_sched_tick() should not restore stuff, it should at worst fail to stop but never enable that's just weird.

It's not really enablement, it's just nohz behaviour but the next tick is in HZ :o)

Like you said before it's an optimization.

I'm still not feeling very confident about all that..

Quoted 2 lines

I can do the things differently
for idle and non-idle cases there but I'm not sure it's really a good thing.

its not, but I'm very sure I've lost you on why that should be the case.

#84 lore raw

Re: [PATCH 13/32] nohz: Adaptive tick stop and restart on nohz cpuset

In reply to Frederic Weisbecker
Quoted 29 lines · Frederic Weisbecker
kernel/sched.c +18 -0
+static void cpuset_nohz_restart_tick(void)
+{
+       __get_cpu_var(task_nohz_mode) = 0;
+       tick_nohz_restart_sched_tick();
+}
+
+void cpuset_update_nohz(void)
+{
+       if (!tick_nohz_adaptive_mode())
+               return;
+
+       if (!cpuset_nohz_can_stop_tick())
+               cpuset_nohz_restart_tick();
+}
+#endif
+
 static void
 ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
 {
@@ -2560,6 +2592,8 @@ static void sched_ttwu_pending(void)
                ttwu_do_activate(rq, p, 0);
        }
 
+       cpuset_update_nohz();
+
        raw_spin_unlock(&rq->lock);
 }
 

kernel/sched.c static void ttwu_queue(struct task_struct *p, int cpu) · line 2659

Quoted 6 lines
kernel/sched.c +1 -0
@@ -2620,6 +2654,7 @@ static void ttwu_queue(struct task_struct *p, int cpu)
 
        raw_spin_lock(&rq->lock);
        ttwu_do_activate(rq, p, 0);
+       cpuset_update_nohz();
        raw_spin_unlock(&rq->lock);
 }

That really has nothing to do with cpusets, why doesn't that live in the tick_nohz_* namespace? Something like tick_nohz_sched_wakeup() or so.

#85 lore raw

PATCH 14/32 nohz/cpuset: Don't turn off the tick if rcu needs it

In reply to Frederic Weisbecker

If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/rcupdate.h |    1 +
 kernel/rcutree.c         |    3 +--
 kernel/sched.c           |   14 ++++++++++++++
 3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/rcupdate.h +1 -0
index 99f9aa7..55a482a 100644
@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
 extern void rcu_check_callbacks(int cpu, int user);
+extern int rcu_pending(int cpu);
 struct notifier_block;
 
 #ifdef CONFIG_NO_HZ
kernel/rcutree.c +1 -2
index ba06207..0009bfc 100644
@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;
 module_param(rcu_cpu_stall_suppress, int, 0644);
 
 static void force_quiescent_state(struct rcu_state *rsp, int relaxed);
-static int rcu_pending(int cpu);
 
 /*
  * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
  * by the current CPU, returning 1 if so.  This function is part of the
  * RCU implementation; it is -not- an exported member of the RCU API.
  */
-static int rcu_pending(int cpu)
+int rcu_pending(int cpu)
 {
 	return __rcu_pending(&rcu_sched_state, &per_cpu(rcu_sched_data, cpu)) ||
 	       __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
kernel/sched.c +14 -0
index 0e1aa4e..353a66f 100644
@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);
 bool cpuset_nohz_can_stop_tick(void)
 {
 	struct rq *rq;
+	int cpu;
 
 	rq = this_rq();
 
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rq->nr_running > 1)
 		return false;
 
+	cpu = smp_processor_id();
+
+	/*
+	 * FIXME: will probably be removed soon as it's
+	 * already checked from tick_nohz_stop_sched_tick()
+	 */
+	if (rcu_needs_cpu(cpu))
+		return false;
+
+	/* Is there a grace period to complete ? */
+	if (rcu_pending(cpu))
+		return false;
+
 	return true;
 }
 
Signature · 1 line
1.7.5.4
#86 lore raw

Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it

Quoted 2 lines · Frederic Weisbecker

If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we

s/dynctik/dyntick/ ;-)

Quoted 3 lines

are not necessarily going to enter into rcu extended quiescent state, so we may need to keep the tick to note current CPU's quiescent states.

One question below...

Thanx, Paul

Quoted 68 lines
Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/rcupdate.h |    1 +
 kernel/rcutree.c         |    3 +--
 kernel/sched.c           |   14 ++++++++++++++
 3 files changed, 16 insertions(+), 2 deletions(-)
include/linux/rcupdate.h +1 -0
index 99f9aa7..55a482a 100644
@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
 extern void rcu_check_callbacks(int cpu, int user);
+extern int rcu_pending(int cpu);
 struct notifier_block;
 
 #ifdef CONFIG_NO_HZ
kernel/rcutree.c +1 -2
index ba06207..0009bfc 100644
@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;
 module_param(rcu_cpu_stall_suppress, int, 0644);
 
 static void force_quiescent_state(struct rcu_state *rsp, int relaxed);
-static int rcu_pending(int cpu);
 
 /*
  * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
  * by the current CPU, returning 1 if so.  This function is part of the
  * RCU implementation; it is -not- an exported member of the RCU API.
  */
-static int rcu_pending(int cpu)
+int rcu_pending(int cpu)
 {
 	return __rcu_pending(&rcu_sched_state, &per_cpu(rcu_sched_data, cpu)) ||
 	       __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
kernel/sched.c +2 -0
index 0e1aa4e..353a66f 100644
@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);
 bool cpuset_nohz_can_stop_tick(void)
 {
 	struct rq *rq;
+	int cpu;
 
 	rq = this_rq();
 
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rq->nr_running > 1)
 		return false;
 
+	cpu = smp_processor_id();

kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460

Quoted 10 lines
kernel/sched.c +10 -0
+
+	/*
+	 * FIXME: will probably be removed soon as it's
+	 * already checked from tick_nohz_stop_sched_tick()
+	 */
+	if (rcu_needs_cpu(cpu))
+		return false;
+
+	/* Is there a grace period to complete ? */
+	if (rcu_pending(cpu))

This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)

But force_quiescent_state() will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?

Quoted 8 lines
kernel/sched.c +2 -0
+		return false;
+
 	return true;
 }
 
Signature · 1 line
1.7.5.4
#87 lore raw

Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it

Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Frederic Weisbecker

If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we

s/dynctik/dyntick/ ;-)

Heh! :)

Quoted 49 lines
Quoted 49 lines
include/linux/rcupdate.h +1 -0
index 99f9aa7..55a482a 100644
@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
 extern void rcu_check_callbacks(int cpu, int user);
+extern int rcu_pending(int cpu);
 struct notifier_block;
 
 #ifdef CONFIG_NO_HZ
kernel/rcutree.c +1 -2
index ba06207..0009bfc 100644
@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;
 module_param(rcu_cpu_stall_suppress, int, 0644);
 
 static void force_quiescent_state(struct rcu_state *rsp, int relaxed);
-static int rcu_pending(int cpu);
 
 /*
  * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
  * by the current CPU, returning 1 if so.  This function is part of the
  * RCU implementation; it is -not- an exported member of the RCU API.
  */
-static int rcu_pending(int cpu)
+int rcu_pending(int cpu)
 {
 	return __rcu_pending(&rcu_sched_state, &per_cpu(rcu_sched_data, cpu)) ||
 	       __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
kernel/sched.c +1 -0
index 0e1aa4e..353a66f 100644
@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);
 bool cpuset_nohz_can_stop_tick(void)
 {
 	struct rq *rq;
+	int cpu;
 
 	rq = this_rq();
 
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rq->nr_running > 1)
 		return false;
 
Quoted 15 lines

kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460

Quoted 11 lines
kernel/sched.c +11 -0
+	cpu = smp_processor_id();
+
+	/*
+	 * FIXME: will probably be removed soon as it's
+	 * already checked from tick_nohz_stop_sched_tick()
+	 */
+	if (rcu_needs_cpu(cpu))
+		return false;
+
+	/* Is there a grace period to complete ? */
+	if (rcu_pending(cpu))

This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)

Yeah this can happen. This is called from the timer interrupt or from an IPI. We can be in any kind of rcu critical section.

Quoted 3 lines

But force_quiescent_state() will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?

Yeah we should receive IPIs from CPUs that need us. This can be an optimization though. No need to run into a cycle of on timers shutdown/restart if we can complete something right away.

#88 lore raw

Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it

Quoted 77 lines · Frederic Weisbecker
Quoted 5 lines · Paul E. McKenney
Quoted 2 lines · Frederic Weisbecker

If RCU is waiting for the current CPU to complete a grace period, don't turn off the tick. Unlike dynctik-idle, we

s/dynctik/dyntick/ ;-)

Heh! :)

Quoted 49 lines
Quoted 49 lines
include/linux/rcupdate.h +1 -0
index 99f9aa7..55a482a 100644
@@ -133,6 +133,7 @@ static inline int rcu_preempt_depth(void)
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
 extern void rcu_check_callbacks(int cpu, int user);
+extern int rcu_pending(int cpu);
 struct notifier_block;
 
 #ifdef CONFIG_NO_HZ
kernel/rcutree.c +1 -2
index ba06207..0009bfc 100644
@@ -205,7 +205,6 @@ int rcu_cpu_stall_suppress __read_mostly;
 module_param(rcu_cpu_stall_suppress, int, 0644);
 
 static void force_quiescent_state(struct rcu_state *rsp, int relaxed);
-static int rcu_pending(int cpu);
 
 /*
  * Return the number of RCU-sched batches processed thus far for debug & stats.
@@ -1729,7 +1728,7 @@ static int __rcu_pending(struct rcu_state *rsp, struct rcu_data *rdp)
  * by the current CPU, returning 1 if so.  This function is part of the
  * RCU implementation; it is -not- an exported member of the RCU API.
  */
-static int rcu_pending(int cpu)
+int rcu_pending(int cpu)
 {
 	return __rcu_pending(&rcu_sched_state, &per_cpu(rcu_sched_data, cpu)) ||
 	       __rcu_pending(&rcu_bh_state, &per_cpu(rcu_bh_data, cpu)) ||
kernel/sched.c +1 -0
index 0e1aa4e..353a66f 100644
@@ -2439,6 +2439,7 @@ DEFINE_PER_CPU(int, task_nohz_mode);
 bool cpuset_nohz_can_stop_tick(void)
 {
 	struct rq *rq;
+	int cpu;
 
 	rq = this_rq();
 
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rq->nr_running > 1)
 		return false;
 
Quoted 15 lines

kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2460

Quoted 11 lines
kernel/sched.c +11 -0
+	cpu = smp_processor_id();
+
+	/*
+	 * FIXME: will probably be removed soon as it's
+	 * already checked from tick_nohz_stop_sched_tick()
+	 */
+	if (rcu_needs_cpu(cpu))
+		return false;
+
+	/* Is there a grace period to complete ? */
+	if (rcu_pending(cpu))

This is from a quiescent state for both RCU and RCU-bh, right? Or can their be RCU or RCU-bh read-side critical sections held across here? (It would be mildly bad if so.)

Yeah this can happen. This is called from the timer interrupt or from an IPI. We can be in any kind of rcu critical section.

Quoted 7 lines
Quoted 2 lines

But force_quiescent_state() will catch cases where RCU needs quiescent states from CPUs, so is this check really needed?

Yeah we should receive IPIs from CPUs that need us. This can be an optimization though. No need to run into a cycle of on timers shutdown/restart if we can complete something right away.

Never mind... I was confusing this with rcu_needs_cpu(), which should not be called from within an RCU-sched or RCU-bh read-side critical section.

It is plenty fine to call rcu_pending() from within RCU read-side critical sections.

Thanx, Paul

#89 lore raw

Re: [PATCH 14/32] nohz/cpuset: Don't turn off the tick if rcu needs it

In reply to Frederic Weisbecker
Quoted 9 lines · Frederic Weisbecker
kernel/sched.c +5 -0
@@ -2446,6 +2447,19 @@ bool cpuset_nohz_can_stop_tick(void)
        if (rq->nr_running > 1)
                return false;
 
+       cpu = smp_processor_id();
+
+       /*
+        * FIXME: will probably be removed soon as it's
+        * already checked from tick_nohz_stop_sched_tick()

kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2464

Quoted 10 lines
kernel/sched.c +8 -0
+        */
+       if (rcu_needs_cpu(cpu))
+               return false;
+
+       /* Is there a grace period to complete ? */
+       if (rcu_pending(cpu))
+               return false;
+
        return true;
 } 

This really shouldn't live in sched.c, also I would have expected tick_nohz_can_stop_tick() to do this, its named like it does this.

#90 lore raw

PATCH 15/32 nohz/cpuset: Restart tick when switching to idle task

In reply to Frederic Weisbecker

Ideally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.

That optimization requires some minor tweaks here and there though, lets handle that later.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/sched.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
kernel/sched.c +13 -0
index 353a66f..9b6b8eb 100644
@@ -2477,6 +2477,18 @@ void cpuset_update_nohz(void)
 	if (!cpuset_nohz_can_stop_tick())
 		cpuset_nohz_restart_tick();
 }
+
+static void cpuset_nohz_task_switch(struct task_struct *next)
+{
+	int cpu = smp_processor_id();
+
+	if (tick_nohz_adaptive_mode() && next == idle_task(cpu))
+		cpuset_nohz_restart_tick();
+}
+#else
+static void cpuset_nohz_task_switch(struct task_struct *next)
+{
+}
 #endif
 
 static void
@@ -3023,6 +3035,7 @@ static inline void
 prepare_task_switch(struct rq *rq, struct task_struct *prev,
 		    struct task_struct *next)
 {
+	cpuset_nohz_task_switch(next);
 	sched_info_switch(prev, next);
 	perf_event_task_sched_out(prev, next);
 	fire_sched_out_preempt_notifiers(prev, next);
Signature · 1 line
1.7.5.4
#91 lore raw

Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task

Quoted 6 lines · Frederic Weisbecker

Ideally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.

That optimization requires some minor tweaks here and there though, lets handle that later.

You have a knack for confusing changelogs.. so basically you say:

Restart the tick when we switch to idle.

Now all that needs is an explanation of why..

Also, please drop the whole cpuset_nohz_ stuff, this really isn't about cpusets, cpusets simly provide the interface, the functionality lives in the tick_nohz_ namespace.

#92 lore raw

Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task

Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Frederic Weisbecker

Ideally if we are in adaptive nohz mode and we switch to the the idle task, we shouldn't restart the tick since it's going to stop the tick soon anyway.

Quoted 8 lines
Quoted 2 lines

That optimization requires some minor tweaks here and there though, lets handle that later.

You have a knack for confusing changelogs.. so basically you say:

Restart the tick when we switch to idle.

Now all that needs is an explanation of why..

In the end we don't want to restart the tick when we switch to idle. But to support that I'll need to tweak various things on idle time accounting, otherwise the [cpu] time spent in idle is going to be accounted as system time.

That's definetly headed to be temporary.

Quoted 3 lines

Also, please drop the whole cpuset_nohz_ stuff, this really isn't about cpusets, cpusets simly provide the interface, the functionality lives in the tick_nohz_ namespace.

Agreed this sucks.

#93 lore raw

Re: [PATCH 15/32] nohz/cpuset: Restart tick when switching to idle task

Quoted 10 lines · Frederic Weisbecker
Quoted 5 lines

... so basically you say:

Restart the tick when we switch to idle.

Now all that needs is an explanation of why..

In the end we don't want to restart the tick when we switch to idle. But to support that I'll need to tweak various things on idle time accounting, otherwise the [cpu] time spent in idle is going to be accounted as system time.

Ok, so then the changelog should say so, preferably with a little bit more detail on how exactly the idle time accounting would go tits-up.

#94 lore raw

PATCH 16/32 nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued

In reply to Frederic Weisbecker

Wake up a CPU when a timer list timer is enqueued there and the CPU is in adaptive nohz mode. Sending an IPI to it makes it reconsidering the next timer to program on top of recent updates.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/sched.h    |    4 ++--
 kernel/sched.c           |   33 ++++++++++++++++++++++++++++++++-
 kernel/time/tick-sched.c |    5 ++++-
 kernel/timer.c           |    2 +-
 4 files changed, 39 insertions(+), 5 deletions(-)
include/linux/sched.h +2 -2
index 53a95b5..5ff0764 100644
@@ -1947,9 +1947,9 @@ static inline void idle_task_exit(void) {}
 #endif
 
 #if defined(CONFIG_NO_HZ) && defined(CONFIG_SMP)
-extern void wake_up_idle_cpu(int cpu);
+extern void wake_up_nohz_cpu(int cpu);
 #else
-static inline void wake_up_idle_cpu(int cpu) { }
+static inline void wake_up_nohz_cpu(int cpu) { }
 #endif
 
 extern unsigned int sysctl_sched_latency;
kernel/sched.c +32 -1
index 9b6b8eb..8bf8280 100644
@@ -1234,7 +1234,7 @@ unlock:
  * account when the CPU goes back to idle and evaluates the timer
  * wheel for the next timer event.
  */
-void wake_up_idle_cpu(int cpu)
+static void wake_up_idle_cpu(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
 
@@ -1264,6 +1264,37 @@ void wake_up_idle_cpu(int cpu)
 		smp_send_reschedule(cpu);
 }
 
+
+static bool wake_up_cpuset_nohz_cpu(int cpu)
+{
+#ifdef CONFIG_CPUSETS_NO_HZ
+	/* Ensure task_nohz_mode update is visible */
+	smp_rmb();
+	/*
+	 * Even if task_nohz_mode is set concurrently, what
+	 * matters is that by the time we do that check, we know
+	 * that the CPU has not reached tick_nohz_stop_sched_tick().
+	 * As we are holding the base->lock and that lock needs
+	 * to be taken by tick_nohz_stop_sched_tick() we know
+	 * we are preceding it and it will see our update
+	 * synchronously. Thus we know we don't need to send an
+	 * IPI to that CPU.
+	 */
+	if (per_cpu(task_nohz_mode, cpu)) {
+		smp_cpuset_update_nohz(cpu);
+		return true;
+	}
+#endif
+
+	return false;
+}
+
+void wake_up_nohz_cpu(int cpu)
+{
+	if (!wake_up_cpuset_nohz_cpu(cpu))
+		wake_up_idle_cpu(cpu);
+}
+
 #endif /* CONFIG_NO_HZ */
 
 static u64 sched_avg_period(void)
kernel/time/tick-sched.c +4 -1
index fb97cd0..9e450d8 100644
@@ -767,8 +767,11 @@ static void tick_nohz_cpuset_stop_tick(int user)
 	if (!cpuset_adaptive_nohz() || tick_nohz_adaptive_mode())
 		return;
 
-	if (cpuset_nohz_can_stop_tick())
+	if (cpuset_nohz_can_stop_tick()) {
 		__get_cpu_var(task_nohz_mode) = 1;
+		/* Nohz mode must be visible to wake_up_nohz_cpu() */
+		smp_wmb();
+	}
 }
 
 /*
kernel/timer.c +1 -1
index 8cff361..8cdbd48 100644
@@ -880,7 +880,7 @@ void add_timer_on(struct timer_list *timer, int cpu)
 	 * makes sure that a CPU on the way to idle can not evaluate
 	 * the timer wheel.
 	 */
-	wake_up_idle_cpu(cpu);
+	wake_up_nohz_cpu(cpu);
 	spin_unlock_irqrestore(&base->lock, flags);
 }
 EXPORT_SYMBOL_GPL(add_timer_on);
Signature · 1 line
1.7.5.4
#95 lore raw

Re: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued

Quoted 11 lines · Frederic Weisbecker

Wake up a CPU when a timer list timer is enqueued there and the CPU is in adaptive nohz mode. Sending an IPI to it makes it reconsidering the next timer to program on top of recent updates.

 include/linux/sched.h    |    4 ++--
 kernel/sched.c           |   33 ++++++++++++++++++++++++++++++++-
 kernel/time/tick-sched.c |    5 ++++-
 kernel/timer.c           |    2 +-
 4 files changed, 39 insertions(+), 5 deletions(-) 

So here I would have expected timer_needs_cpu() and an addition to tick_nohz_can_stop_tick(). Why does sched.c get touched at all?

Also, all the delta_jiffies stuff in the current tick_nohz_stop_sched_tick() deals with this, why duplicate the logic?

maintainer
#96 lore raw

Re: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued

In reply to Frederic Weisbecker
Quoted 4 lines · Peter Zijlstra

Also, all the delta_jiffies stuff in the current tick_nohz_stop_sched_tick() deals with this, why duplicate the logic?

Damn, n/m this is about new timers.. tick_nohz_new_timer() then.

#97 lore raw

Re: [PATCH 16/32] nohz/cpuset: Wake up adaptive nohz CPU when a timer gets enqueued

Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra

Also, all the delta_jiffies stuff in the current tick_nohz_stop_sched_tick() deals with this, why duplicate the logic?

Damn, n/m this is about new timers.. tick_nohz_new_timer() then.

I can call that tick_nohz_new_timer() if you prefer yeah. I personally don't mind either name.

#98 lore raw

PATCH 17/32 x86: New cpuset nohz irq vector

In reply to Frederic Weisbecker

We need a way to send an irq IPI (or local) in any case and asynchronously in order to restart the tick for CPUs in nohz adaptive mode

Generic smp operations don't fit into this because they need interrupts to be enabled and they try to execute the functions in place if the dest CPU is the current one. But we always need the function to be executed in irq context so it happens quickly and restarting the tick doesn't mess up with random lock scenarios in place.

In fact this is a temporary solution, what we really need is an irq work subsystem that supports remote enqueuing.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 arch/x86/include/asm/entry_arch.h  |    3 +++
 arch/x86/include/asm/hw_irq.h      |    6 ++++++
 arch/x86/include/asm/irq_vectors.h |    2 ++
 arch/x86/include/asm/smp.h         |   11 +++++++++++
 arch/x86/kernel/entry_64.S         |    4 ++++
 arch/x86/kernel/irqinit.c          |    4 ++++
 arch/x86/kernel/smp.c              |   26 ++++++++++++++++++++++++++
 7 files changed, 56 insertions(+), 0 deletions(-)
arch/x86/include/asm/entry_arch.h +3 -0
index 1cd6d26..019cf29 100644
@@ -10,6 +10,9 @@
  * through the ICC by us (IPIs)
  */
 #ifdef CONFIG_SMP
+#ifdef CONFIG_CPUSETS_NO_HZ
+BUILD_INTERRUPT(cpuset_update_nohz_interrupt,CPUSET_UPDATE_NOHZ_VECTOR)
+#endif
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
 BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR)
arch/x86/include/asm/hw_irq.h +6 -0
index bb9efe8..1978050 100644
@@ -34,6 +34,9 @@ extern void irq_work_interrupt(void);
 extern void spurious_interrupt(void);
 extern void thermal_interrupt(void);
 extern void reschedule_interrupt(void);
+#ifdef CONFIG_CPUSETS_NO_HZ
+extern void cpuset_update_nohz_interrupt(void);
+#endif
 extern void mce_self_interrupt(void);
 
 extern void invalidate_interrupt(void);
@@ -153,6 +156,9 @@ extern asmlinkage void smp_irq_move_cleanup_interrupt(void);
 #endif
 #ifdef CONFIG_SMP
 extern void smp_reschedule_interrupt(struct pt_regs *);
+#ifdef CONFIG_CPUSETS_NO_HZ
+extern void smp_cpuset_update_nohz_interrupt(struct pt_regs *);
+#endif
 extern void smp_call_function_interrupt(struct pt_regs *);
 extern void smp_call_function_single_interrupt(struct pt_regs *);
 #ifdef CONFIG_X86_32
arch/x86/include/asm/irq_vectors.h +2 -0
index 6e976ee..5e33fec 100644
@@ -117,6 +117,8 @@
 /* Xen vector callback to receive events in a HVM domain */
 #define XEN_HVM_EVTCHN_CALLBACK		0xf3
 
+#define CPUSET_UPDATE_NOHZ_VECTOR	0xf2
+
 /*
  * Local APIC timer IRQ vector is on a different priority level,
  * to work around the 'lost local interrupt if more than 2 IRQ
arch/x86/include/asm/smp.h +11 -0
index 73b11bc..66dc629 100644
@@ -70,6 +70,10 @@ struct smp_ops {
 	void (*stop_other_cpus)(int wait);
 	void (*smp_send_reschedule)(int cpu);
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+	void (*smp_cpuset_update_nohz)(int cpu);
+#endif
+
 	int (*cpu_up)(unsigned cpu);
 	int (*cpu_disable)(void);
 	void (*cpu_die)(unsigned int cpu);
@@ -138,6 +142,13 @@ static inline void smp_send_reschedule(int cpu)
 	smp_ops.smp_send_reschedule(cpu);
 }
 
+static inline void smp_cpuset_update_nohz(int cpu)
+{
+#ifdef CONFIG_CPUSETS_NO_HZ
+	smp_ops.smp_cpuset_update_nohz(cpu);
+#endif
+}
+
 static inline void arch_send_call_function_single_ipi(int cpu)
 {
 	smp_ops.send_call_func_single_ipi(cpu);
arch/x86/kernel/entry_64.S +4 -0
index d656f68..06d79c2 100644
@@ -996,6 +996,10 @@ apicinterrupt CALL_FUNCTION_VECTOR \
 	call_function_interrupt smp_call_function_interrupt
 apicinterrupt RESCHEDULE_VECTOR \
 	reschedule_interrupt smp_reschedule_interrupt
+#ifdef CONFIG_CPUSETS_NO_HZ
+apicinterrupt CPUSET_UPDATE_NOHZ_VECTOR \
+	cpuset_update_nohz_interrupt smp_cpuset_update_nohz_interrupt
+#endif
 #endif
 
 apicinterrupt ERROR_APIC_VECTOR \
arch/x86/kernel/irqinit.c +4 -0
index f470e4e..ba5665c 100644
@@ -172,6 +172,10 @@ static void __init smp_intr_init(void)
 	 */
 	alloc_intr_gate(RESCHEDULE_VECTOR, reschedule_interrupt);
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+	alloc_intr_gate(CPUSET_UPDATE_NOHZ_VECTOR, cpuset_update_nohz_interrupt);
+#endif
+
 	/* IPIs for invalidation */
 #define ALLOC_INVTLB_VEC(NR) \
 	alloc_intr_gate(INVALIDATE_TLB_VECTOR_START+NR, \
arch/x86/kernel/smp.c +26 -0
index 013e7eb..7c3e399 100644
@@ -22,6 +22,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/gfp.h>
+#include <linux/cpuset.h>
 
 #include <asm/mtrr.h>
 #include <asm/tlbflush.h>
@@ -121,6 +122,17 @@ static void native_smp_send_reschedule(int cpu)
 	apic->send_IPI_mask(cpumask_of(cpu), RESCHEDULE_VECTOR);
 }
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+static void native_smp_cpuset_update_nohz(int cpu)
+{
+	if (unlikely(cpu_is_offline(cpu))) {
+		WARN_ON(1);
+		return;
+	}
+	apic->send_IPI_mask(cpumask_of(cpu), CPUSET_UPDATE_NOHZ_VECTOR);
+}
+#endif
+
 void native_send_call_func_single_ipi(int cpu)
 {
 	apic->send_IPI_mask(cpumask_of(cpu), CALL_FUNCTION_SINGLE_VECTOR);
@@ -206,6 +218,17 @@ void smp_reschedule_interrupt(struct pt_regs *regs)
 	 */
 }
 
+#ifdef CONFIG_CPUSETS_NO_HZ
+void smp_cpuset_update_nohz_interrupt(struct pt_regs *regs)
+{
+	ack_APIC_irq();
+	irq_enter();
+	cpuset_update_nohz();
+	inc_irq_stat(irq_call_count);
+	irq_exit();
+}
+#endif
+
 void smp_call_function_interrupt(struct pt_regs *regs)
 {
 	ack_APIC_irq();
@@ -231,6 +254,9 @@ struct smp_ops smp_ops = {
 
 	.stop_other_cpus	= native_stop_other_cpus,
 	.smp_send_reschedule	= native_smp_send_reschedule,
+#ifdef CONFIG_CPUSETS_NO_HZ
+	.smp_cpuset_update_nohz = native_smp_cpuset_update_nohz,
+#endif
 
 	.cpu_up			= native_cpu_up,
 	.cpu_die		= native_cpu_die,
Signature · 1 line
1.7.5.4
#99 lore raw

PATCH 18/32 nohz/cpuset: Don't stop the tick if posix cpu timers are running

In reply to Frederic Weisbecker

If either a per thread or a per process posix cpu timer is running, don't stop the tick.

TODO: restart the tick if it is stopped and a posix cpu timer is enqueued. Check we probably need a memory barrier for the per process posix timer that can be enqueued from another task of the group.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/posix-timers.h |    1 +
 kernel/posix-cpu-timers.c    |   12 ++++++++++++
 kernel/sched.c               |    4 ++++
 3 files changed, 17 insertions(+), 0 deletions(-)
include/linux/posix-timers.h +1 -0
index 959c141..5092cfd 100644
@@ -116,6 +116,7 @@ int posix_timer_event(struct k_itimer *timr, int si_private);
 void posix_cpu_timer_schedule(struct k_itimer *timer);
 
 void run_posix_cpu_timers(struct task_struct *task);
+bool posix_cpu_timers_running(struct task_struct *tsk);
 void posix_cpu_timers_exit(struct task_struct *task);
 void posix_cpu_timers_exit_group(struct task_struct *task);
 
kernel/posix-cpu-timers.c +12 -0
index 58f405b..f284fa4 100644
@@ -6,6 +6,7 @@
 #include <linux/posix-timers.h>
 #include <linux/errno.h>
 #include <linux/math64.h>
+#include <linux/cpuset.h>
 #include <asm/uaccess.h>
 #include <linux/kernel_stat.h>
 #include <trace/events/timer.h>
@@ -1300,6 +1301,17 @@ static inline int fastpath_timer_check(struct task_struct *tsk)
 	return 0;
 }
 
+bool posix_cpu_timers_running(struct task_struct *tsk)
+{
+	if (!task_cputime_zero(&tsk->cputime_expires))
+		return true;
+
+	if (tsk->signal->cputimer.running)
+		return true;
+
+	return false;
+}
+
 /*
  * This is called from the timer interrupt handler.  The irq handler has
  * already updated our counts.  We need to check if any timers fire now.
kernel/sched.c +4 -0
index 8bf8280..78ea0a5 100644
@@ -71,6 +71,7 @@
 #include <linux/ctype.h>
 #include <linux/ftrace.h>
 #include <linux/slab.h>
+#include <linux/posix-timers.h>
 
 #include <asm/tlb.h>
 #include <asm/irq_regs.h>
@@ -2491,6 +2492,9 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rcu_pending(cpu))
 		return false;
 
+	if (posix_cpu_timers_running(current))
+		return false;
+
 	return true;
 }
 
Signature · 1 line
1.7.5.4
#100 lore raw

Re: [PATCH 18/32] nohz/cpuset: Don't stop the tick if posix cpu timers are running

Quoted 7 lines · Frederic Weisbecker

If either a per thread or a per process posix cpu timer is running, don't stop the tick.

TODO: restart the tick if it is stopped and a posix cpu timer is enqueued. Check we probably need a memory barrier for the per process posix timer that can be enqueued from another task of the group.

it would

Quoted 10 lines

+++ b/kernel/sched.c

kernel/sched.c +1 -0
@@ -71,6 +71,7 @@
 #include <linux/ctype.h>
 #include <linux/ftrace.h>
 #include <linux/slab.h>
+#include <linux/posix-timers.h>
 
 #include <asm/tlb.h>
 #include <asm/irq_regs.h>

kernel/sched.c bool cpuset_nohz_can_stop_tick(void) · line 2499

Quoted 8 lines
kernel/sched.c +3 -0
@@ -2491,6 +2492,9 @@ bool cpuset_nohz_can_stop_tick(void)
 	if (rcu_pending(cpu))
 		return false;
 
+	if (posix_cpu_timers_running(current))
+		return false;
+
 	return true;
 }

Doesn't belong here, go poke at tick_nohz_can_stop_tick().

#101 lore raw

PATCH 19/32 nohz/cpuset: Restart tick when nohz flag is cleared on cpuset

In reply to Frederic Weisbecker

Issue an IPI to restart the tick on a CPU that belongs to a cpuset when its nohz flag gets cleared.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/cpuset.h |    1 +
 kernel/cpuset.c        |   21 +++++++++++++++++++++
 kernel/sched.c         |    8 ++++++++
 3 files changed, 30 insertions(+), 0 deletions(-)
include/linux/cpuset.h +1 -0
index 799b9a4..7f9d78d 100644
@@ -265,6 +265,7 @@ static inline bool cpuset_adaptive_nohz(void)
 }
 
 extern void cpuset_update_nohz(void);
+extern void cpuset_exit_nohz_interrupt(void *unused);
 #else
 static inline void cpuset_update_nohz(void) { }
 
kernel/cpuset.c +21 -0
index 3135096..ee3b0d0 100644
@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,
 
 DEFINE_PER_CPU(int, cpu_adaptive_nohz_ref);
 
+static void cpu_exit_nohz(int cpu)
+{
+	preempt_disable();
+	smp_call_function_single(cpu, cpuset_exit_nohz_interrupt,
+				 NULL, true);
+	preempt_enable();
+}
+
 static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 {
 	int cpu;
@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 			per_cpu(cpu_adaptive_nohz_ref, cpu) += 1;
 		else
 			per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1;
+
+		val = per_cpu(cpu_adaptive_nohz_ref, cpu);
+
+		if (!val) {
+			/*
+			 * The update to cpu_adaptive_nohz_ref must be
+			 * visible right away. So that once we restart the tick
+			 * from the IPI, it won't be stopped again due to cache
+			 * update lag.
+			 */
+			smp_mb();
+			cpu_exit_nohz(cpu);
+		}
 	}
 }
 #else
kernel/sched.c +8 -0
index 78ea0a5..75378be 100644
@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)
 		cpuset_nohz_restart_tick();
 }
 
+void cpuset_exit_nohz_interrupt(void *unused)
+{
+	if (!tick_nohz_adaptive_mode())
+		return;
+
+	cpuset_nohz_restart_tick();
+}
+
 static void cpuset_nohz_task_switch(struct task_struct *next)
 {
 	int cpu = smp_processor_id();
Signature · 1 line
1.7.5.4
#102 lore raw

Re: [PATCH 19/32] nohz/cpuset: Restart tick when nohz flag is cleared on cpuset

Quoted 41 lines · Frederic Weisbecker

+++ b/kernel/cpuset.c

kernel/cpuset.c +21 -0
@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,
 
 DEFINE_PER_CPU(int, cpu_adaptive_nohz_ref);
 
+static void cpu_exit_nohz(int cpu)
+{
+       preempt_disable();
+       smp_call_function_single(cpu, cpuset_exit_nohz_interrupt,
+                                NULL, true);
+       preempt_enable();
+}
+
 static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 {
        int cpu;
@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
                        per_cpu(cpu_adaptive_nohz_ref, cpu) += 1;
                else
                        per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1;
+
+               val = per_cpu(cpu_adaptive_nohz_ref, cpu);
+
+               if (!val) {
+                       /*
+                        * The update to cpu_adaptive_nohz_ref must be
+                        * visible right away. So that once we restart the tick
+                        * from the IPI, it won't be stopped again due to cache
+                        * update lag.
+                        */
+                       smp_mb();
+                       cpu_exit_nohz(cpu);
+               }
        }
 }
 #else
kernel/sched.c +0 -0
index 78ea0a5..75378be 100644

kernel/sched.c void cpuset_update_nohz(void) · line 2522

Quoted 10 lines
kernel/sched.c +7 -0
@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)
                cpuset_nohz_restart_tick();
 }
 
+void cpuset_exit_nohz_interrupt(void *unused)
+{
+       if (!tick_nohz_adaptive_mode())
+               return;
+
+       cpuset_nohz_restart_tick();
+} 

You do this just to annoy me, right? Why doesn't it live in cpuset.c where you use it?

#103 lore raw

Re: [PATCH 19/32] nohz/cpuset: Restart tick when nohz flag is cleared on cpuset

Quoted 45 lines · Peter Zijlstra
Quoted 44 lines · Frederic Weisbecker

+++ b/kernel/cpuset.c

kernel/cpuset.c +21 -0
@@ -1199,6 +1199,14 @@ static void cpuset_change_flag(struct task_struct *tsk,
 
 DEFINE_PER_CPU(int, cpu_adaptive_nohz_ref);
 
+static void cpu_exit_nohz(int cpu)
+{
+       preempt_disable();
+       smp_call_function_single(cpu, cpuset_exit_nohz_interrupt,
+                                NULL, true);
+       preempt_enable();
+}
+
 static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 {
        int cpu;
@@ -1212,6 +1220,19 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
                        per_cpu(cpu_adaptive_nohz_ref, cpu) += 1;
                else
                        per_cpu(cpu_adaptive_nohz_ref, cpu) -= 1;
+
+               val = per_cpu(cpu_adaptive_nohz_ref, cpu);
+
+               if (!val) {
+                       /*
+                        * The update to cpu_adaptive_nohz_ref must be
+                        * visible right away. So that once we restart the tick
+                        * from the IPI, it won't be stopped again due to cache
+                        * update lag.
+                        */
+                       smp_mb();
+                       cpu_exit_nohz(cpu);
+               }
        }
 }
 #else
kernel/sched.c +0 -0
index 78ea0a5..75378be 100644
@@ -2513,6 +2513,14 @@ void cpuset_update_nohz(void)
                cpuset_nohz_restart_tick();
 }
 
Quoted 10 lines

kernel/sched.c void cpuset_update_nohz(void) · line 2522

Quoted 7 lines
kernel/sched.c +7 -0
+void cpuset_exit_nohz_interrupt(void *unused)
+{
+       if (!tick_nohz_adaptive_mode())
+               return;
+
+       cpuset_nohz_restart_tick();
+} 

You do this just to annoy me, right? Why doesn't it live in cpuset.c where you use it?

Right, I'll move all those things and fix the naming / namespace.

#104 lore raw

PATCH 20/32 nohz/cpuset: Restart the tick if printk needs it

In reply to Frederic Weisbecker

If we are in nohz adaptive mode and printk is called, the tick is missing to wake up the logger. We need to restart the tick when that happens.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/printk.c |   17 ++++++++++++++++-
 1 files changed, 16 insertions(+), 1 deletions(-)
kernel/printk.c +16 -1
index 3518539..aff07f0 100644
@@ -41,6 +41,7 @@
 #include <linux/cpu.h>
 #include <linux/notifier.h>
 #include <linux/rculist.h>
+#include <linux/tick.h>
 
 #include <asm/uaccess.h>
 
@@ -1220,8 +1221,22 @@ int printk_needs_cpu(int cpu)
 
 void wake_up_klogd(void)
 {
-	if (waitqueue_active(&log_wait))
+	unsigned long flags;
+
+	if (waitqueue_active(&log_wait)) {
 		this_cpu_write(printk_pending, 1);
+		/* Make it visible from any interrupt from now */
+		barrier();
+		/*
+		 * It's safe to check that even if interrupts are not disabled.
+		 * If we enable nohz adaptive mode concurrently, we'll the
+		 * printk_pending value and thus keep a periodic tick behaviour.
+		 * Unless it's possible tick_nohz_adaptive_mode() reads its value
+		 * before the barrier() ?
+		 */
+		if (tick_nohz_adaptive_mode())
+			smp_cpuset_update_nohz(smp_processor_id());
+	}
 }
 
 /**
Signature · 1 line
1.7.5.4
#105 lore raw

PATCH 21/32 rcu: Restart the tick on non-responding adaptive nohz CPUs

In reply to Frederic Weisbecker

When a CPU in adaptive nohz mode doesn't respond to complete a grace period, issue it a specific IPI so that it restarts the tick and chases a quiescent state.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/rcutree.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
kernel/rcutree.c +17 -0
index 0009bfc..d496c70 100644
@@ -50,6 +50,7 @@
 #include <linux/wait.h>
 #include <linux/kthread.h>
 #include <linux/prefetch.h>
+#include <linux/cpuset.h>
 
 #include "rcutree.h"
 
@@ -295,6 +296,20 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp)
 
 #ifdef CONFIG_SMP
 
+static void cpuset_update_rcu_cpu(int cpu)
+{
+#ifdef CONFIG_CPUSETS_NO_HZ
+	unsigned long flags;
+
+	local_irq_save(flags);
+
+	if (cpuset_cpu_adaptive_nohz(cpu))
+		smp_cpuset_update_nohz(cpu);
+
+	local_irq_restore(flags);
+#endif
+}
+
 /*
  * If the specified CPU is offline, tell the caller that it is in
  * a quiescent state.  Otherwise, whack it with a reschedule IPI.
@@ -317,6 +332,8 @@ static int rcu_implicit_offline_qs(struct rcu_data *rdp)
 		return 1;
 	}
 
+	cpuset_update_rcu_cpu(rdp->cpu);
+
 	/* If preemptible RCU, no point in sending reschedule IPI. */
 	if (rdp->preemptible)
 		return 0;
Signature · 1 line
1.7.5.4
#106 lore raw

PATCH 22/32 rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU

In reply to Frederic Weisbecker

If we enqueue an rcu callback, we need the CPU tick to stay alive until we take care of those by completing the appropriate grace period.

Thus, when we call_rcu(), send a self IPI that checks rcu_needs_cpu() so that we restore a periodic tick behaviour that can take care of everything.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/rcutree.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
kernel/rcutree.c +8 -0
index d496c70..b5643ce2 100644
@@ -51,6 +51,7 @@
 #include <linux/kthread.h>
 #include <linux/prefetch.h>
 #include <linux/cpuset.h>
+#include <linux/tick.h>
 
 #include "rcutree.h"
 
@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
 	rdp->qlen++;
 
+	/* Restart the timer if needed to handle the callbacks */
+	if (tick_nohz_adaptive_mode()) {
+		/* Make updates on nxtlist visible to self IPI */
+		barrier();
+		smp_cpuset_update_nohz(smp_processor_id());
+	}
+
 	/* If interrupts were disabled, don't dive into RCU core. */
 	if (irqs_disabled_flags(flags)) {
 		local_irq_restore(flags);
Signature · 1 line
1.7.5.4
#107 lore raw

Re: [PATCH 22/32] rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU

Quoted 7 lines · Frederic Weisbecker

If we enqueue an rcu callback, we need the CPU tick to stay alive until we take care of those by completing the appropriate grace period.

Thus, when we call_rcu(), send a self IPI that checks rcu_needs_cpu() so that we restore a periodic tick behaviour that can take care of everything.

One question below.

Thanx, Paul

Quoted 29 lines
Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/rcutree.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
kernel/rcutree.c +1 -0
index d496c70..b5643ce2 100644
@@ -51,6 +51,7 @@
 #include <linux/kthread.h>
 #include <linux/prefetch.h>
 #include <linux/cpuset.h>
+#include <linux/tick.h>
 
 #include "rcutree.h"
 

kernel/rcutree.c __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · line 1556

Quoted 10 lines
kernel/rcutree.c +7 -0
@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
 	rdp->qlen++;
 
+	/* Restart the timer if needed to handle the callbacks */
+	if (tick_nohz_adaptive_mode()) {
+		/* Make updates on nxtlist visible to self IPI */
+		barrier();
+		smp_cpuset_update_nohz(smp_processor_id());
+	}
+

But this must be happening in a system call or interrupt handler, right? If so, won't we get a chance to check things on exit from the system call or interrupt? Or are you hooking only into syscall entry?

Quoted 6 lines
kernel/rcutree.c +0 -0
 	/* If interrupts were disabled, don't dive into RCU core. */
 	if (irqs_disabled_flags(flags)) {
 		local_irq_restore(flags);
Signature · 1 line
1.7.5.4
#108 lore raw

Re: [PATCH 22/32] rcu: Restart tick if we enqueue a callback in a nohz/cpuset CPU

Quoted 4 lines · Paul E. McKenney
Quoted 4 lines
kernel/rcutree.c +0 -0
@@ -1546,6 +1547,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
 	rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
 	rdp->qlen++;
 
Quoted 11 lines

kernel/rcutree.c __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · line 1556

Quoted 7 lines
kernel/rcutree.c +7 -0
+	/* Restart the timer if needed to handle the callbacks */
+	if (tick_nohz_adaptive_mode()) {
+		/* Make updates on nxtlist visible to self IPI */
+		barrier();
+		smp_cpuset_update_nohz(smp_processor_id());
+	}
+

But this must be happening in a system call or interrupt handler, right? If so, won't we get a chance to check things on exit from the system call or interrupt? Or are you hooking only into syscall entry?

Sure in theory when we call call_rcu() we are in the kernel and thus not very far from a syscall/exception/irq exit into userspace or a scheduling out.

We could rely on that assumption and I bet it won't ever be a problem in practice. I just wanted to ensure we don't spend too much time before handling that callback.

Quoted 6 lines
Quoted 6 lines
kernel/rcutree.c +0 -0
 	/* If interrupts were disabled, don't dive into RCU core. */
 	if (irqs_disabled_flags(flags)) {
 		local_irq_restore(flags);
Signature · 1 line
1.7.5.4
#109 lore raw

PATCH 23/32 nohz/cpuset: Account user and system times in adaptive nohz mode

In reply to Frederic Weisbecker

If we are not running the tick, we are not anymore regularly counting the cputime at every jiffies.

Lay the ground to count that cputime from the points that require it. Start by catching up from timer interrupts and when we schedule out a process. We record the last jiffies and from which ring we saved it and compute the difference later when we can catch up.

For now it assumes we haven't switched to another ring while we were running nohz.

TODO: wrap operation on jiffies?

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/kernel_stat.h |    2 ++
 include/linux/tick.h        |   11 +++++++++++
 kernel/sched.c              |   23 +++++++++++++++++++++++
 kernel/time/tick-sched.c    |   39 +++++++++++++++++++++++++++++++++++++++
 kernel/timer.c              |    6 ++++--
 5 files changed, 79 insertions(+), 2 deletions(-)
include/linux/kernel_stat.h +2 -0
index 0cce2db..14cfce4 100644
@@ -114,7 +114,9 @@ static inline unsigned int kstat_cpu_irqs_sum(unsigned int cpu)
 extern unsigned long long task_delta_exec(struct task_struct *);
 
 extern void account_user_time(struct task_struct *, cputime_t, cputime_t);
+extern void account_user_jiffies(struct task_struct *, unsigned long);
 extern void account_system_time(struct task_struct *, int, cputime_t, cputime_t);
+extern void account_system_jiffies(struct task_struct *, unsigned long);
 extern void account_steal_time(cputime_t);
 extern void account_idle_time(cputime_t);
 
include/linux/tick.h +11 -0
index cc4880e..ea6dfb7 100644
@@ -26,6 +26,12 @@ enum tick_nohz_mode {
 	NOHZ_MODE_HIGHRES,
 };
 
+enum tick_saved_jiffies {
+	JIFFIES_SAVED_NONE,
+	JIFFIES_SAVED_USER,
+	JIFFIES_SAVED_SYS,
+};
+
 /**
  * struct tick_sched - sched tick emulation and no idle tick control/stats
  * @sched_timer:	hrtimer to schedule the periodic tick in high
@@ -60,6 +66,8 @@ struct tick_sched {
 	ktime_t				idle_waketime;
 	ktime_t				idle_exittime;
 	ktime_t				idle_sleeptime;
+	enum tick_saved_jiffies		saved_jiffies_whence;
+	unsigned long			saved_jiffies;
 	ktime_t				iowait_sleeptime;
 	ktime_t				sleep_length;
 	unsigned long			last_jiffies;
@@ -132,8 +140,11 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 DECLARE_PER_CPU(int, task_nohz_mode);
 
 extern int tick_nohz_adaptive_mode(void);
+extern bool tick_nohz_account_tick(void);
+extern void tick_nohz_flush_current_times(void);
 #else /* !CPUSETS_NO_HZ */
 static inline int tick_nohz_adaptive_mode(void) { return 0; }
+static inline bool tick_nohz_account_tick(void) { return false; }
 #endif /* CPUSETS_NO_HZ */
 
 # else /* !NO_HZ */
kernel/sched.c +23 -0
index 75378be..a58f993 100644
@@ -2500,6 +2500,7 @@ bool cpuset_nohz_can_stop_tick(void)
 
 static void cpuset_nohz_restart_tick(void)
 {
+	tick_nohz_flush_current_times();
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
 }
@@ -3838,6 +3839,17 @@ void account_user_time(struct task_struct *p, cputime_t cputime,
 	acct_update_integrals(p);
 }
 
+void account_user_jiffies(struct task_struct *p, unsigned long count)
+{
+	cputime_t delta_cputime, delta_scaled;
+
+	if (count) {
+		delta_cputime = jiffies_to_cputime(count);
+		delta_scaled = cputime_to_scaled(count);
+		account_user_time(p, delta_cputime, delta_scaled);
+	}
+}
+
 /*
  * Account guest cpu time to a process.
  * @p: the process that the cpu time gets accounted to
@@ -3922,6 +3934,17 @@ void account_system_time(struct task_struct *p, int hardirq_offset,
 	__account_system_time(p, cputime, cputime_scaled, target_cputime64);
 }
 
+void account_system_jiffies(struct task_struct *p, unsigned long count)
+{
+	cputime_t delta_cputime, delta_scaled;
+
+	if (count) {
+		delta_cputime = jiffies_to_cputime(count);
+		delta_scaled = cputime_to_scaled(count);
+		account_system_time(p, 0, delta_cputime, delta_scaled);
+	}
+}
+
 /*
  * Account for involuntary wait time.
  * @cputime: the cpu time spent in involuntary wait
kernel/time/tick-sched.c +39 -0
index 9e450d8..c3a8f26 100644
@@ -764,6 +764,8 @@ int tick_nohz_adaptive_mode(void)
 
 static void tick_nohz_cpuset_stop_tick(int user)
 {
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+
 	if (!cpuset_adaptive_nohz() || tick_nohz_adaptive_mode())
 		return;
 
@@ -771,6 +773,13 @@ static void tick_nohz_cpuset_stop_tick(int user)
 		__get_cpu_var(task_nohz_mode) = 1;
 		/* Nohz mode must be visible to wake_up_nohz_cpu() */
 		smp_wmb();
+
+		WARN_ON_ONCE(ts->saved_jiffies_whence != JIFFIES_SAVED_NONE);
+		ts->saved_jiffies = jiffies;
+		if (user)
+			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
+		else
+			ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
 	}
 }
 
@@ -792,6 +801,36 @@ static void tick_do_timer_check_handler(int cpu)
 	}
 }
 
+bool tick_nohz_account_tick(void)
+{
+	struct tick_sched *ts;
+	unsigned long delta_jiffies;
+
+	if (!tick_nohz_adaptive_mode())
+		return false;
+
+	ts = &__get_cpu_var(tick_cpu_sched);
+
+	delta_jiffies = jiffies - ts->saved_jiffies;
+	if (ts->saved_jiffies_whence == JIFFIES_SAVED_SYS)
+		account_system_jiffies(current, delta_jiffies);
+	else
+		account_user_jiffies(current, delta_jiffies);
+
+	ts->saved_jiffies = jiffies;
+
+	return true;
+}
+
+void tick_nohz_flush_current_times(void)
+{
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+
+	tick_nohz_account_tick();
+
+	ts->saved_jiffies_whence = JIFFIES_SAVED_NONE;
+}
+
 #else
 
 static void tick_nohz_cpuset_stop_tick(int user) { }
kernel/timer.c +4 -2
index 8cdbd48..db984ff 100644
@@ -1288,8 +1288,10 @@ void update_process_times(int user_tick)
 	struct task_struct *p = current;
 	int cpu = smp_processor_id();
 
-	/* Note: this timer irq context must be accounted for as well. */
-	account_process_tick(p, user_tick);
+	if (!tick_nohz_account_tick()) {
+		/* Note: this timer irq context must be accounted for as well. */
+		account_process_tick(p, user_tick);
+	}
 	run_local_timers();
 	rcu_check_callbacks(cpu, user_tick);
 	printk_tick();
Signature · 1 line
1.7.5.4
#110 lore raw

PATCH 24/32 nohz/cpuset: Handle kernel entry/exit to account cputime

In reply to Frederic Weisbecker

Provide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.

The new APIs are:

- tick_nohz_enter_kernel() (called when we enter a syscall)
- tick_nohz_exit_kernel() (called when we exit a syscall)
- tick_nohz_enter_exception() (called when we enter any exception, trap, faults...but not irqs)
- tick_nohz_exit_exception() (called when we exit any exception)

Hooks into syscalls are typically driven by the TIF_NOHZ thread flag.

In addition, we use the value returned by user_mode(regs) from the timer interrupt to know where we are. Nonetheless, we can rely on user_mode(regs) != 0 to know we are in userspace, but we can't rely on user_mode(regs) == 0 to know we are in the system.

Consider the following scenario: we stop the tick after syscall return, so we set TIF_NOHZ but the syscall exit hook is behind us. If we haven't yet returned to userspace, then we have user_mode(regs) == 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.

The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with user_mode(regs) == 1 fires. Kernel threads don't have this worry.

This sucks but for now I have no better solution. Let's hope we can find better.

TODO: wrap operation on jiffies?

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    8 +++
 kernel/sched.c           |    1 +
 kernel/time/tick-sched.c |  114 ++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 109 insertions(+), 14 deletions(-)
include/linux/tick.h +8 -0
index ea6dfb7..3ad649f 100644
@@ -139,10 +139,18 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #ifdef CONFIG_CPUSETS_NO_HZ
 DECLARE_PER_CPU(int, task_nohz_mode);
 
+extern void tick_nohz_enter_kernel(void);
+extern void tick_nohz_exit_kernel(void);
+extern void tick_nohz_enter_exception(struct pt_regs *regs);
+extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
 extern bool tick_nohz_account_tick(void);
 extern void tick_nohz_flush_current_times(void);
 #else /* !CPUSETS_NO_HZ */
+static inline void tick_nohz_enter_kernel(void) { }
+static inline void tick_nohz_exit_kernel(void) { }
+static inline void tick_nohz_enter_exception(struct pt_regs *regs) { }
+static inline void tick_nohz_exit_exception(struct pt_regs *regs) { }
 static inline int tick_nohz_adaptive_mode(void) { return 0; }
 static inline bool tick_nohz_account_tick(void) { return false; }
 #endif /* CPUSETS_NO_HZ */
kernel/sched.c +1 -0
index a58f993..c49c1b1 100644
@@ -2503,6 +2503,7 @@ static void cpuset_nohz_restart_tick(void)
 	tick_nohz_flush_current_times();
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
+	clear_thread_flag(TIF_NOHZ);
 }
 
 void cpuset_update_nohz(void)
kernel/time/tick-sched.c +100 -14

164 lines not on this page · raw message

Signature · 1 line
1.7.5.4
#111 lore raw

Re: [PATCH 24/32] nohz/cpuset: Handle kernel entry/exit to account cputime

Quoted 35 lines · Frederic Weisbecker

Provide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.

The new APIs are:

- tick_nohz_enter_kernel() (called when we enter a syscall)
- tick_nohz_exit_kernel() (called when we exit a syscall)
- tick_nohz_enter_exception() (called when we enter any exception, trap, faults...but not irqs)
- tick_nohz_exit_exception() (called when we exit any exception)

Hooks into syscalls are typically driven by the TIF_NOHZ thread flag.

In addition, we use the value returned by user_mode(regs) from the timer interrupt to know where we are. Nonetheless, we can rely on user_mode(regs) != 0 to know we are in userspace, but we can't rely on user_mode(regs) == 0 to know we are in the system.

Consider the following scenario: we stop the tick after syscall return, so we set TIF_NOHZ but the syscall exit hook is behind us. If we haven't yet returned to userspace, then we have user_mode(regs) == 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.

The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with user_mode(regs) == 1 fires. Kernel threads don't have this worry.

Quoted 4 lines

This sucks but for now I have no better solution. Let's hope we can find better.

TODO: wrap operation on jiffies?

Hmmm... Does the RCU dyntick-idle code need to know about exception entry and exit?

Thanx, Paul

Quoted 230 lines
Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    8 +++
 kernel/sched.c           |    1 +
 kernel/time/tick-sched.c |  114 ++++++++++++++++++++++++++++++++++++++++------
 3 files changed, 109 insertions(+), 14 deletions(-)
include/linux/tick.h +8 -0
index ea6dfb7..3ad649f 100644
@@ -139,10 +139,18 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #ifdef CONFIG_CPUSETS_NO_HZ
 DECLARE_PER_CPU(int, task_nohz_mode);
 
+extern void tick_nohz_enter_kernel(void);
+extern void tick_nohz_exit_kernel(void);
+extern void tick_nohz_enter_exception(struct pt_regs *regs);
+extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
 extern bool tick_nohz_account_tick(void);
 extern void tick_nohz_flush_current_times(void);
 #else /* !CPUSETS_NO_HZ */
+static inline void tick_nohz_enter_kernel(void) { }
+static inline void tick_nohz_exit_kernel(void) { }
+static inline void tick_nohz_enter_exception(struct pt_regs *regs) { }
+static inline void tick_nohz_exit_exception(struct pt_regs *regs) { }
 static inline int tick_nohz_adaptive_mode(void) { return 0; }
 static inline bool tick_nohz_account_tick(void) { return false; }
 #endif /* CPUSETS_NO_HZ */
kernel/sched.c +1 -0
index a58f993..c49c1b1 100644
@@ -2503,6 +2503,7 @@ static void cpuset_nohz_restart_tick(void)
 	tick_nohz_flush_current_times();
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
+	clear_thread_flag(TIF_NOHZ);
 }
 
 void cpuset_update_nohz(void)
kernel/time/tick-sched.c +100 -14
index c3a8f26..d8f01b8 100644
@@ -595,8 +595,9 @@ void tick_nohz_irq_exit(void)
 	if (ts->inidle && !need_resched())
 		__tick_nohz_enter_idle(ts, cpu);
 	else if (tick_nohz_adaptive_mode() && !idle_cpu(cpu)) {
-		if (tick_nohz_can_stop_tick(cpu, ts))
-			tick_nohz_stop_sched_tick(ktime_get(), cpu, ts);
+		if (ts->saved_jiffies_whence != JIFFIES_SAVED_NONE
+		    && tick_nohz_can_stop_tick(cpu, ts))
+				tick_nohz_stop_sched_tick(ktime_get(), cpu, ts);
 	}
 }
 
@@ -757,6 +758,74 @@ void tick_check_idle(int cpu)
 
 #ifdef CONFIG_CPUSETS_NO_HZ
 
+void tick_nohz_exit_kernel(void)
+{
+	unsigned long flags;
+	struct tick_sched *ts;
+	unsigned long delta_jiffies;
+
+	local_irq_save(flags);
+
+	if (!tick_nohz_adaptive_mode()) {
+		local_irq_restore(flags);
+		return;
+	}
+
+	ts = &__get_cpu_var(tick_cpu_sched);
+
+	WARN_ON_ONCE(ts->saved_jiffies_whence == JIFFIES_SAVED_USER);
+
+	if (ts->saved_jiffies_whence == JIFFIES_SAVED_SYS) {
+		delta_jiffies = jiffies - ts->saved_jiffies;
+		account_system_jiffies(current, delta_jiffies);
+	}
+
+	ts->saved_jiffies = jiffies;
+	ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
+
+	local_irq_restore(flags);
+}
+
+void tick_nohz_enter_kernel(void)
+{
+	unsigned long flags;
+	struct tick_sched *ts;
+	unsigned long delta_jiffies;
+
+	local_irq_save(flags);
+
+	if (!tick_nohz_adaptive_mode()) {
+		local_irq_restore(flags);
+		return;
+	}
+
+	ts = &__get_cpu_var(tick_cpu_sched);
+
+	WARN_ON_ONCE(ts->saved_jiffies_whence == JIFFIES_SAVED_SYS);
+
+	if (ts->saved_jiffies_whence == JIFFIES_SAVED_USER) {
+		delta_jiffies = jiffies - ts->saved_jiffies;
+		account_user_jiffies(current, delta_jiffies);
+	}
+
+	ts->saved_jiffies = jiffies;
+	ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
+
+	local_irq_restore(flags);
+}
+
+void tick_nohz_enter_exception(struct pt_regs *regs)
+{
+	if (user_mode(regs))
+		tick_nohz_enter_kernel();
+}
+
+void tick_nohz_exit_exception(struct pt_regs *regs)
+{
+	if (user_mode(regs))
+		tick_nohz_exit_kernel();
+}
+
 int tick_nohz_adaptive_mode(void)
 {
 	return __get_cpu_var(task_nohz_mode);
@@ -766,20 +835,33 @@ static void tick_nohz_cpuset_stop_tick(int user)
 {
 	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
 
-	if (!cpuset_adaptive_nohz() || tick_nohz_adaptive_mode())
+	if (!cpuset_adaptive_nohz())
 		return;
 
+	if (tick_nohz_adaptive_mode()) {
+		if (user && ts->saved_jiffies_whence == JIFFIES_SAVED_NONE) {
+			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
+			ts->saved_jiffies = jiffies;
+		}
+
+		return;
+	}
+
 	if (cpuset_nohz_can_stop_tick()) {
 		__get_cpu_var(task_nohz_mode) = 1;
 		/* Nohz mode must be visible to wake_up_nohz_cpu() */
 		smp_wmb();
 
+		set_thread_flag(TIF_NOHZ);
 		WARN_ON_ONCE(ts->saved_jiffies_whence != JIFFIES_SAVED_NONE);
-		ts->saved_jiffies = jiffies;
-		if (user)
+
+		if (user) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
-		else
+			ts->saved_jiffies = jiffies;
+		} else if (!current->mm) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
+			ts->saved_jiffies = jiffies;
+		}
 	}
 }
 
@@ -803,7 +885,7 @@ static void tick_do_timer_check_handler(int cpu)
 
 bool tick_nohz_account_tick(void)
 {
-	struct tick_sched *ts;
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
 	unsigned long delta_jiffies;
 
 	if (!tick_nohz_adaptive_mode())
@@ -811,11 +893,15 @@ bool tick_nohz_account_tick(void)
 
 	ts = &__get_cpu_var(tick_cpu_sched);
 
+	if (ts->saved_jiffies_whence == JIFFIES_SAVED_NONE)
+		return false;
+
 	delta_jiffies = jiffies - ts->saved_jiffies;
-	if (ts->saved_jiffies_whence == JIFFIES_SAVED_SYS)
-		account_system_jiffies(current, delta_jiffies);
-	else
+
+	if (ts->saved_jiffies_whence == JIFFIES_SAVED_USER)
 		account_user_jiffies(current, delta_jiffies);
+	else
+		account_system_jiffies(current, delta_jiffies);
 
 	ts->saved_jiffies = jiffies;
 
@@ -825,12 +911,12 @@ bool tick_nohz_account_tick(void)
 void tick_nohz_flush_current_times(void)
 {
 	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+	unsigned long delta_jiffies;
+	struct pt_regs *regs;
 
-	tick_nohz_account_tick();
-
-	ts->saved_jiffies_whence = JIFFIES_SAVED_NONE;
+	if (tick_nohz_account_tick())
+		ts->saved_jiffies_whence = JIFFIES_SAVED_NONE;
 }
-
 #else
 
 static void tick_nohz_cpuset_stop_tick(int user) { }
Signature · 1 line
1.7.5.4
#112 lore raw

Re: [PATCH 24/32] nohz/cpuset: Handle kernel entry/exit to account cputime

Quoted 39 lines · Paul E. McKenney
Quoted 38 lines · Frederic Weisbecker

Provide a few APIs that archs can call to tell they are entering or exiting the kernel so that when we are in nohz adaptive mode we know precisely where we need to account the cputime.

The new APIs are:

- tick_nohz_enter_kernel() (called when we enter a syscall)
- tick_nohz_exit_kernel() (called when we exit a syscall)
- tick_nohz_enter_exception() (called when we enter any exception, trap, faults...but not irqs)
- tick_nohz_exit_exception() (called when we exit any exception)

Hooks into syscalls are typically driven by the TIF_NOHZ thread flag.

In addition, we use the value returned by user_mode(regs) from the timer interrupt to know where we are. Nonetheless, we can rely on user_mode(regs) != 0 to know we are in userspace, but we can't rely on user_mode(regs) == 0 to know we are in the system.

Consider the following scenario: we stop the tick after syscall return, so we set TIF_NOHZ but the syscall exit hook is behind us. If we haven't yet returned to userspace, then we have user_mode(regs) == 0. If on top of that we consider we are in system mode, and later we issue a syscall but restart the tick right before reaching the syscall entry hook, then we have no clue that the whole elapsed cputime was not in the system but in the userspace.

The only way to fix this is to only start entering nohz mode once we know we are in userspace a first time, like when we reach the kernel exit hook or when a timer tick with user_mode(regs) == 1 fires. Kernel threads don't have this worry.

This sucks but for now I have no better solution. Let's hope we can find better.

Quoted 6 lines
Quoted 1 line

TODO: wrap operation on jiffies?

Hmmm... Does the RCU dyntick-idle code need to know about exception entry and exit?

Thanx, Paul

At that time it doesn't because we don't yet call rcu_enter_nohz() when switching to userspace. Instead we shutdown the tick and restart it when needed when a remote CPU sends us an IPI to complete a grace period.

The patch that switches to extended qs is the 31/32 and it handles syscalls and exceptions as well.

I wanted to have support on rcu extended quiescent states late in the patchset so that it's considered as an incremental feature and not a core piece of the adaptive nohz (ie: it's no mandatory thing, just an optimization). This way we can use cpuset nohz without that rcu extended quiescent state feature and hence make that small part bisectable.

Patch 30 activates support for cpuset nohz (support from x86). Patch 31 activates the rcu extended quiescent state support in userspace as a bonus.

#113 lore raw

PATCH 25/32 nohz/cpuset: New API to flush cputimes on nohz cpusets

In reply to Frederic Weisbecker

Provide a new API that sends an IPI to every CPUs included in nohz cpusets in order to flush their cputimes. It's going to be useful for those that want to see accurate cputimes on a nohz cpuset.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/cpuset.h   |    2 ++
 include/linux/tick.h     |    2 +-
 kernel/cpuset.c          |   34 +++++++++++++++++++++++++++++++++-
 kernel/sched.c           |    2 +-
 kernel/time/tick-sched.c |    4 ++--
 5 files changed, 39 insertions(+), 5 deletions(-)
include/linux/cpuset.h +2 -0
index 7f9d78d..569da83 100644
@@ -265,9 +265,11 @@ static inline bool cpuset_adaptive_nohz(void)
 }
 
 extern void cpuset_update_nohz(void);
+extern void cpuset_nohz_flush_cputimes(void);
 extern void cpuset_exit_nohz_interrupt(void *unused);
 #else
 static inline void cpuset_update_nohz(void) { }
+static inline void cpuset_nohz_flush_cputimes(void) { }
 
 #endif /* CONFIG_CPUSETS_NO_HZ */
 
include/linux/tick.h +1 -1
index 3ad649f..9d0270e 100644
@@ -145,7 +145,7 @@ extern void tick_nohz_enter_exception(struct pt_regs *regs);
 extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
 extern bool tick_nohz_account_tick(void);
-extern void tick_nohz_flush_current_times(void);
+extern void tick_nohz_flush_current_times(bool restart_tick);
 #else /* !CPUSETS_NO_HZ */
 static inline void tick_nohz_enter_kernel(void) { }
 static inline void tick_nohz_exit_kernel(void) { }
kernel/cpuset.c +33 -1
index ee3b0d0..61c3f96 100644
@@ -59,6 +59,7 @@
 #include <linux/mutex.h>
 #include <linux/workqueue.h>
 #include <linux/cgroup.h>
+#include <linux/tick.h>
 
 /*
  * Workqueue for cpuset related tasks.
@@ -1199,6 +1200,23 @@ static void cpuset_change_flag(struct task_struct *tsk,
 
 DEFINE_PER_CPU(int, cpu_adaptive_nohz_ref);
 
+static cpumask_t nohz_cpuset_mask;
+
+static void flush_cputime_interrupt(void *unused)
+{
+	tick_nohz_flush_current_times(false);
+}
+
+void cpuset_nohz_flush_cputimes(void)
+{
+	preempt_disable();
+	smp_call_function_many(&nohz_cpuset_mask, flush_cputime_interrupt,
+			       NULL, true);
+	preempt_enable();
+	/* Make the utime/stime updates visible */
+	smp_mb();
+}
+
 static void cpu_exit_nohz(int cpu)
 {
 	preempt_disable();
@@ -1223,7 +1241,15 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 
 		val = per_cpu(cpu_adaptive_nohz_ref, cpu);
 
-		if (!val) {
+		if (val == 1) {
+			cpumask_set_cpu(cpu, &nohz_cpuset_mask);
+			/*
+			 * The mask update needs to be visible right away
+			 * so that this CPU is part of the cputime IPI
+			 * update right now.
+			 */
+			 smp_mb();
+		} else if (!val) {
 			/*
 			 * The update to cpu_adaptive_nohz_ref must be
 			 * visible right away. So that once we restart the tick
@@ -1232,6 +1258,12 @@ static void update_nohz_cpus(struct cpuset *old_cs, struct cpuset *cs)
 			 */
 			smp_mb();
 			cpu_exit_nohz(cpu);
+			/*
+			 * Now that the tick has been restarted and cputimes
+			 * flushed, we don't need anymore to be part of the
+			 * cputime flush IPI.
+			 */
+			cpumask_clear_cpu(cpu, &nohz_cpuset_mask);
 		}
 	}
 }
kernel/sched.c +1 -1
index c49c1b1..2bcd456 100644
@@ -2500,7 +2500,7 @@ bool cpuset_nohz_can_stop_tick(void)
 
 static void cpuset_nohz_restart_tick(void)
 {
-	tick_nohz_flush_current_times();
+	tick_nohz_flush_current_times(true);
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
 	clear_thread_flag(TIF_NOHZ);
kernel/time/tick-sched.c +2 -2
index d8f01b8..9a2ba5b 100644
@@ -908,13 +908,13 @@ bool tick_nohz_account_tick(void)
 	return true;
 }
 
-void tick_nohz_flush_current_times(void)
+void tick_nohz_flush_current_times(bool restart_tick)
 {
 	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
 	unsigned long delta_jiffies;
 	struct pt_regs *regs;
 
-	if (tick_nohz_account_tick())
+	if (tick_nohz_account_tick() && restart_tick)
 		ts->saved_jiffies_whence = JIFFIES_SAVED_NONE;
 }
 #else
Signature · 1 line
1.7.5.4
#114 lore raw

PATCH 26/32 nohz/cpuset: Flush cputime on threads in nohz cpusets when waiting leader

In reply to Frederic Weisbecker

When we wait for a zombie task, flush the cputimes on nohz cpusets in case we are waiting for a group leader that has threads running in nohz CPUs. This way thread_group_times() doesn't report stale values.

<doubts>
If I understood well the code, by the time we call that thread_group_times(), we may have childs that are still running, so this is necessary. But I need to check deeper.
</doubts>

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/exit.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/exit.c +2 -0
index f2b321b..43fb9ac 100644
@@ -51,6 +51,7 @@
 #include <trace/events/sched.h>
 #include <linux/hw_breakpoint.h>
 #include <linux/oom.h>
+#include <linux/cpuset.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -1262,6 +1263,7 @@ static int wait_task_zombie(struct wait_opts *wo, struct task_struct *p)
 		 * group, which consolidates times for all threads in the
 		 * group including the group leader.
 		 */
+		cpuset_nohz_flush_cputimes();
 		thread_group_times(p, &tgutime, &tgstime);
 		spin_lock_irq(&p->real_parent->sighand->siglock);
 		psig = p->real_parent->signal;
Signature · 1 line
1.7.5.4
#115 lore raw

PATCH 27/32 nohz/cpuset: Flush cputimes on procfs stat file read

In reply to Frederic Weisbecker

When we read a process's procfs stat file, we need to flush the cputimes of the tasks running in nohz cpusets in case some childs in the thread group are running there.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 fs/proc/array.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/proc/array.c +2 -0
index 9b45ee8..20e1c75 100644
@@ -397,6 +397,8 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,
 	cutime = cstime = utime = stime = cputime_zero;
 	cgtime = gtime = cputime_zero;
 
+	/* For thread group times */
+	cpuset_nohz_flush_cputimes();
 	if (lock_task_sighand(task, &flags)) {
 		struct signal_struct *sig = task->signal;
 
Signature · 1 line
1.7.5.4
#116 lore raw

PATCH 28/32 nohz/cpuset: Flush cputimes for getrusage() and times() syscalls

In reply to Frederic Weisbecker

Both syscalls need to iterate through the thread group to get the cputimes. As some threads of the group may be running on nohz cpuset, we need to flush the cputimes there.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 kernel/sys.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
kernel/sys.c +6 -0
index e4128b2..f6de4b1 100644
@@ -43,6 +43,7 @@
 #include <linux/syscalls.h>
 #include <linux/kprobes.h>
 #include <linux/user_namespace.h>
+#include <linux/cpuset.h>
 
 #include <linux/kmsg_dump.h>
 
@@ -908,6 +909,8 @@ void do_sys_times(struct tms *tms)
 {
 	cputime_t tgutime, tgstime, cutime, cstime;
 
+	cpuset_nohz_flush_cputimes();
+
 	spin_lock_irq(&current->sighand->siglock);
 	thread_group_times(current, &tgutime, &tgstime);
 	cutime = current->signal->cutime;
@@ -1536,6 +1539,9 @@ static void k_getrusage(struct task_struct *p, int who, struct rusage *r)
 		goto out;
 	}
 
+	/* For thread_group_times */
+	cpuset_nohz_flush_cputimes();
+
 	if (!lock_task_sighand(p, &flags))
 		return;
 
Signature · 1 line
1.7.5.4
#117 lore raw

PATCH 29/32 x86: Syscall hooks for nohz cpusets

In reply to Frederic Weisbecker

Add syscall hooks to notify syscall entry and exit on CPUs running in adative nohz mode.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 arch/x86/include/asm/thread_info.h |   10 +++++++---
 arch/x86/kernel/ptrace.c           |   10 ++++++++++
 2 files changed, 17 insertions(+), 3 deletions(-)
arch/x86/include/asm/thread_info.h +7 -3
index 1f2e61e..0e3329f 100644
@@ -87,6 +87,7 @@ struct thread_info {
 #define TIF_NOTSC		16	/* TSC is not accessible in userland */
 #define TIF_IA32		17	/* 32bit process */
 #define TIF_FORK		18	/* ret_from_fork */
+#define TIF_NOHZ		19	/* in nohz userspace mode */
 #define TIF_MEMDIE		20	/* is terminating due to OOM killer */
 #define TIF_DEBUG		21	/* uses debug registers */
 #define TIF_IO_BITMAP		22	/* uses I/O bitmap */
@@ -110,6 +111,7 @@ struct thread_info {
 #define _TIF_NOTSC		(1 << TIF_NOTSC)
 #define _TIF_IA32		(1 << TIF_IA32)
 #define _TIF_FORK		(1 << TIF_FORK)
+#define _TIF_NOHZ		(1 << TIF_NOHZ)
 #define _TIF_DEBUG		(1 << TIF_DEBUG)
 #define _TIF_IO_BITMAP		(1 << TIF_IO_BITMAP)
 #define _TIF_FREEZE		(1 << TIF_FREEZE)
@@ -121,12 +123,13 @@ struct thread_info {
 /* work to do in syscall_trace_enter() */
 #define _TIF_WORK_SYSCALL_ENTRY	\
 	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_EMU | _TIF_SYSCALL_AUDIT |	\
-	 _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)
+	 _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT |	\
+	 _TIF_NOHZ)
 
 /* work to do in syscall_trace_leave() */
 #define _TIF_WORK_SYSCALL_EXIT	\
 	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SINGLESTEP |	\
-	 _TIF_SYSCALL_TRACEPOINT)
+	 _TIF_SYSCALL_TRACEPOINT | _TIF_NOHZ)
 
 /* work to do on interrupt/exception return */
 #define _TIF_WORK_MASK							\
@@ -136,7 +139,8 @@ struct thread_info {
 
 /* work to do on any return to user space */
 #define _TIF_ALLWORK_MASK						\
-	((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT)
+	((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT |	\
+	_TIF_NOHZ)
 
 /* Only used for 64 bit */
 #define _TIF_DO_NOTIFY_MASK						\
arch/x86/kernel/ptrace.c +10 -0
index 6e619f2..486a65c 100644
@@ -21,6 +21,7 @@
 #include <linux/signal.h>
 #include <linux/perf_event.h>
 #include <linux/hw_breakpoint.h>
+#include <linux/tick.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
@@ -1385,6 +1386,9 @@ long syscall_trace_enter(struct pt_regs *regs)
 {
 	long ret = 0;
 
+	/* Notify nohz task syscall early so the rest can use rcu */
+	tick_nohz_enter_kernel();
+
 	/*
 	 * If we stepped into a sysenter/syscall insn, it trapped in
 	 * kernel mode; do_debug() cleared TF and set TIF_SINGLESTEP.
@@ -1446,4 +1450,10 @@ void syscall_trace_leave(struct pt_regs *regs)
 			!test_thread_flag(TIF_SYSCALL_EMU);
 	if (step || test_thread_flag(TIF_SYSCALL_TRACE))
 		tracehook_report_syscall_exit(regs, step);
+
+	/*
+	 * Notify nohz task exit syscall at last so the rest can
+	 * use rcu.
+	 */
+	tick_nohz_exit_kernel();
 }
Signature · 1 line
1.7.5.4
#118 lore raw

PATCH 30/32 x86: Exception hooks for nohz cpusets

In reply to Frederic Weisbecker

Add necessary hooks to x86 exception for nohz cpusets support. It includes traps, page fault, debug exceptions, etc...

TODO: handle do_notify_resume(). Not an exception but it's executed at the end of the syscall/irq path so it's beyond our hooks but shouldn't be considered as userspace code.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 arch/x86/Kconfig        |    1 +
 arch/x86/kernel/traps.c |   22 +++++++++++++++-------
 arch/x86/mm/fault.c     |   13 +++++++++++--
 3 files changed, 27 insertions(+), 9 deletions(-)
arch/x86/Kconfig +1 -0
index 67979f4..1b12b1a 100644
@@ -69,6 +69,7 @@ config X86
 	select IRQ_FORCED_THREADING
 	select USE_GENERIC_SMP_HELPERS if SMP
 	select HAVE_BPF_JIT if (X86_64 && NET)
+	select HAVE_CPUSETS_NO_HZ
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/kernel/traps.c +15 -7
index b9b6716..d98f2f4 100644
@@ -26,6 +26,7 @@
 #include <linux/sched.h>
 #include <linux/timer.h>
 #include <linux/init.h>
+#include <linux/tick.h>
 #include <linux/bug.h>
 #include <linux/nmi.h>
 #include <linux/mm.h>
@@ -456,24 +457,28 @@ void restart_nmi(void)
 /* May run on IST stack. */
 dotraplinkage void __kprobes do_int3(struct pt_regs *regs, long error_code)
 {
+	tick_nohz_enter_exception(regs);
+
 #ifdef CONFIG_KGDB_LOW_LEVEL_TRAP
 	if (kgdb_ll_trap(DIE_INT3, "int3", regs, error_code, 3, SIGTRAP)
 			== NOTIFY_STOP)
-		return;
+		goto exit;
 #endif /* CONFIG_KGDB_LOW_LEVEL_TRAP */
 #ifdef CONFIG_KPROBES
 	if (notify_die(DIE_INT3, "int3", regs, error_code, 3, SIGTRAP)
 			== NOTIFY_STOP)
-		return;
+		goto exit;
 #else
 	if (notify_die(DIE_TRAP, "int3", regs, error_code, 3, SIGTRAP)
 			== NOTIFY_STOP)
-		return;
+		goto exit;
 #endif
 
 	preempt_conditional_sti(regs);
 	do_trap(3, SIGTRAP, "int3", regs, error_code, NULL);
 	preempt_conditional_cli(regs);
+exit:
+	tick_nohz_exit_exception(regs);
 }
 
 #ifdef CONFIG_X86_64
@@ -534,6 +539,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
 	unsigned long dr6;
 	int si_code;
 
+	tick_nohz_enter_exception(regs);
+
 	get_debugreg(dr6, 6);
 
 	/* Filter out all the reserved bits which are preset to 1 */
@@ -549,7 +556,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
 
 	/* Catch kmemcheck conditions first of all! */
 	if ((dr6 & DR_STEP) && kmemcheck_trap(regs))
-		return;
+		goto exit;
 
 	/* DR6 may or may not be cleared by the CPU */
 	set_debugreg(0, 6);
@@ -564,7 +571,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
 
 	if (notify_die(DIE_DEBUG, "debug", regs, PTR_ERR(&dr6), error_code,
 							SIGTRAP) == NOTIFY_STOP)
-		return;
+		goto exit;
 
 	/* It's safe to allow irq's after DR6 has been saved */
 	preempt_conditional_sti(regs);
@@ -573,7 +580,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
 		handle_vm86_trap((struct kernel_vm86_regs *) regs,
 				error_code, 1);
 		preempt_conditional_cli(regs);
-		return;
+		goto exit;
 	}
 
 	/*
@@ -593,7 +600,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)
 		send_sigtrap(tsk, regs, error_code, si_code);
 	preempt_conditional_cli(regs);
 
-	return;
+exit:
+	tick_nohz_exit_exception(regs);
 }
 
 /*
arch/x86/mm/fault.c +11 -2
index 4d09df0..aa2e2e3 100644
@@ -13,6 +13,7 @@
 #include <linux/perf_event.h>		/* perf_sw_event		*/
 #include <linux/hugetlb.h>		/* hstate_index_to_shift	*/
 #include <linux/prefetch.h>		/* prefetchw			*/
+#include <linux/tick.h>
 
 #include <asm/traps.h>			/* dotraplinkage, ...		*/
 #include <asm/pgalloc.h>		/* pgd_*(), ...			*/
@@ -971,8 +972,8 @@ static int fault_in_kernel_space(unsigned long address)
  * and the problem, and then passes it off to one of the appropriate
  * routines.
  */
-dotraplinkage void __kprobes
-do_page_fault(struct pt_regs *regs, unsigned long error_code)
+static void __kprobes
+__do_page_fault(struct pt_regs *regs, unsigned long error_code)
 {
 	struct vm_area_struct *vma;
 	struct task_struct *tsk;
@@ -1180,3 +1181,11 @@ good_area:
 
 	up_read(&mm->mmap_sem);
 }
+
+dotraplinkage void __kprobes
+do_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+	tick_nohz_enter_exception(regs);
+	__do_page_fault(regs, error_code);
+	tick_nohz_exit_exception(regs);
+}
Signature · 1 line
1.7.5.4
#119 lore raw

PATCH 31/32 rcu: Switch to extended quiescent state in userspace from nohz cpuset

In reply to Frederic Weisbecker

When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.

However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.

To perform this, we just to use the RCU extended quiescent state APIs on the following points:

- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.

- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.

TODO: hook into do_notify_resume() because we may have called rcu_enter_nohz() from syscall exit hook, but we might call do_notify_resume() right after, which may use RCU.

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    2 ++
 kernel/sched.c           |    1 +
 kernel/time/tick-sched.c |   21 +++++++++++++++++++++
 3 files changed, 24 insertions(+), 0 deletions(-)
include/linux/tick.h +2 -0
index 9d0270e..4e7555f 100644
@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 
 #ifdef CONFIG_CPUSETS_NO_HZ
 DECLARE_PER_CPU(int, task_nohz_mode);
+DECLARE_PER_CPU(int, nohz_task_ext_qs);
 
 extern void tick_nohz_enter_kernel(void);
 extern void tick_nohz_exit_kernel(void);
 extern void tick_nohz_enter_exception(struct pt_regs *regs);
 extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
+extern void tick_nohz_cpu_exit_qs(void);
 extern bool tick_nohz_account_tick(void);
 extern void tick_nohz_flush_current_times(bool restart_tick);
 #else /* !CPUSETS_NO_HZ */
kernel/sched.c +1 -0
index 2bcd456..576d0bf 100644
@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
 	clear_thread_flag(TIF_NOHZ);
+	tick_nohz_cpu_exit_qs();
 }
 
 void cpuset_update_nohz(void)
kernel/time/tick-sched.c +21 -0
index 9a2ba5b..b611b77 100644
@@ -757,6 +757,7 @@ void tick_check_idle(int cpu)
 }
 
 #ifdef CONFIG_CPUSETS_NO_HZ
+DEFINE_PER_CPU(int, nohz_task_ext_qs);
 
 void tick_nohz_exit_kernel(void)
 {
@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void)
 	ts->saved_jiffies = jiffies;
 	ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 
+	__get_cpu_var(nohz_task_ext_qs) = 1;
+	rcu_enter_nohz();
+
 	local_irq_restore(flags);
 }
 
@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)
 		return;
 	}
 
+	if (__get_cpu_var(nohz_task_ext_qs) == 1) {
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+		rcu_exit_nohz();
+	}
+
 	ts = &__get_cpu_var(tick_cpu_sched);
 
 	WARN_ON_ONCE(ts->saved_jiffies_whence == JIFFIES_SAVED_SYS);
@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)
 	local_irq_restore(flags);
 }
 
+void tick_nohz_cpu_exit_qs(void)
+{
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+
+	if (__get_cpu_var(nohz_task_ext_qs)) {
+		rcu_exit_nohz();
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+	}
+}
+
 void tick_nohz_enter_exception(struct pt_regs *regs)
 {
 	if (user_mode(regs))
@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)
 		if (user) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 			ts->saved_jiffies = jiffies;
+			__get_cpu_var(nohz_task_ext_qs) = 1;
+			rcu_enter_nohz();
 		} else if (!current->mm) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
 			ts->saved_jiffies = jiffies;
Signature · 1 line
1.7.5.4
#120 lore raw

Re: [PATCH 31/32] rcu: Switch to extended quiescent state in userspace from nohz cpuset

Quoted 17 lines · Frederic Weisbecker

When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.

However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.

To perform this, we just to use the RCU extended quiescent state APIs on the following points:

- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.

Quoted 8 lines

- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.

TODO: hook into do_notify_resume() because we may have called rcu_enter_nohz() from syscall exit hook, but we might call do_notify_resume() right after, which may use RCU.

I don't see exactly how the exception path works, but this does reassure me a bit on the syscall path.

Thanx, Paul

Quoted 62 lines
Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    2 ++
 kernel/sched.c           |    1 +
 kernel/time/tick-sched.c |   21 +++++++++++++++++++++
 3 files changed, 24 insertions(+), 0 deletions(-)
include/linux/tick.h +2 -0
index 9d0270e..4e7555f 100644
@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 
 #ifdef CONFIG_CPUSETS_NO_HZ
 DECLARE_PER_CPU(int, task_nohz_mode);
+DECLARE_PER_CPU(int, nohz_task_ext_qs);
 
 extern void tick_nohz_enter_kernel(void);
 extern void tick_nohz_exit_kernel(void);
 extern void tick_nohz_enter_exception(struct pt_regs *regs);
 extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
+extern void tick_nohz_cpu_exit_qs(void);
 extern bool tick_nohz_account_tick(void);
 extern void tick_nohz_flush_current_times(bool restart_tick);
 #else /* !CPUSETS_NO_HZ */
kernel/sched.c +1 -0
index 2bcd456..576d0bf 100644
@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
 	clear_thread_flag(TIF_NOHZ);
+	tick_nohz_cpu_exit_qs();
 }
 
 void cpuset_update_nohz(void)
kernel/time/tick-sched.c +1 -0
index 9a2ba5b..b611b77 100644
@@ -757,6 +757,7 @@ void tick_check_idle(int cpu)
 }
 
 #ifdef CONFIG_CPUSETS_NO_HZ
+DEFINE_PER_CPU(int, nohz_task_ext_qs);
 
 void tick_nohz_exit_kernel(void)
 {

kernel/time/tick-sched.c void tick_nohz_exit_kernel(void) · line 788

Quoted 5 lines
kernel/time/tick-sched.c +2 -0
@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void)
 	ts->saved_jiffies = jiffies;
 	ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 
+	__get_cpu_var(nohz_task_ext_qs) = 1;
+	rcu_enter_nohz();

OK, I was wondering how this was going to work if RCU didn't know about kernel entry/exit. Whew!!! ;-)

Quoted 34 lines
kernel/time/tick-sched.c +16 -0
+
 	local_irq_restore(flags);
 }
 
@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)
 		return;
 	}
 
+	if (__get_cpu_var(nohz_task_ext_qs) == 1) {
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+		rcu_exit_nohz();
+	}
+
 	ts = &__get_cpu_var(tick_cpu_sched);
 
 	WARN_ON_ONCE(ts->saved_jiffies_whence == JIFFIES_SAVED_SYS);
@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)
 	local_irq_restore(flags);
 }
 
+void tick_nohz_cpu_exit_qs(void)
+{
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+
+	if (__get_cpu_var(nohz_task_ext_qs)) {
+		rcu_exit_nohz();
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+	}
+}
+
 void tick_nohz_enter_exception(struct pt_regs *regs)
 {
 	if (user_mode(regs))

kernel/time/tick-sched.c static void tick_nohz_cpuset_stop_tick(int user) · line 881

Quoted 5 lines
kernel/time/tick-sched.c +2 -0
@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)
 		if (user) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 			ts->saved_jiffies = jiffies;
+			__get_cpu_var(nohz_task_ext_qs) = 1;
+			rcu_enter_nohz();

When entering an exception, shouldn't we call rcu_exit_nohz() rather than rcu_exit_nohz()? Or is this a "didn't really mean an exception" code path?

Quoted 11 lines
kernel/time/tick-sched.c +0 -0
 		} else if (!current->mm) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
 			ts->saved_jiffies = jiffies;
Signature · 7 lines
1.7.5.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#121 lore raw

Re: [PATCH 31/32] rcu: Switch to extended quiescent state in userspace from nohz cpuset

Quoted 23 lines · Paul E. McKenney
Quoted 22 lines · Frederic Weisbecker

When we switch to adaptive nohz mode and we run in userspace, we can still receive IPIs from the RCU core if a grace period has been started by another CPU because we need to take part of its completion.

However running in userspace is similar to that of running in idle because we don't make use of RCU there, thus we can be considered as running in RCU extended quiescent state. The benefit when running into that mode is that we are not anymore disturbed by needless IPIs coming from the RCU core.

To perform this, we just to use the RCU extended quiescent state APIs on the following points:

- kernel exit or tick stop in userspace: here we switch to extended quiescent state because we run in userspace without the tick.

- kernel entry or tick restart: here we exit the extended quiescent state because either we enter the kernel and we may make use of RCU read side critical section anytime, or we need the timer tick for some reason and that takes care of RCU grace period in a traditional way.

Quoted 6 lines
Quoted 3 lines

TODO: hook into do_notify_resume() because we may have called rcu_enter_nohz() from syscall exit hook, but we might call do_notify_resume() right after, which may use RCU.

I don't see exactly how the exception path works, but this does reassure me a bit on the syscall path.

On the syscall path we directly call tick_nohz_enter,exit_kernel() and that takes care of all the rcu trickies.

The exception paths call tick_nohz_enter,exit_exception() which are essentially conditional wrappers around tick_nohz_enter,exit_kernel() after checking user_mode(regs). Hmm now I realize I can't rely on user_mode() to know if we should exit or not rcu extended quiescent state because even if user_mode(regs) != 1, we may not have yet called the syscall exit hook and thus not exited rcu quiescent state. So tick_nohz_enter_exception() may forget to call rcu_exit_nohz() sometimes.

I need to fix that.

Quoted 129 lines

Thanx, Paul

Quoted 62 lines
Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 include/linux/tick.h     |    2 ++
 kernel/sched.c           |    1 +
 kernel/time/tick-sched.c |   21 +++++++++++++++++++++
 3 files changed, 24 insertions(+), 0 deletions(-)
include/linux/tick.h +2 -0
index 9d0270e..4e7555f 100644
@@ -138,12 +138,14 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 
 #ifdef CONFIG_CPUSETS_NO_HZ
 DECLARE_PER_CPU(int, task_nohz_mode);
+DECLARE_PER_CPU(int, nohz_task_ext_qs);
 
 extern void tick_nohz_enter_kernel(void);
 extern void tick_nohz_exit_kernel(void);
 extern void tick_nohz_enter_exception(struct pt_regs *regs);
 extern void tick_nohz_exit_exception(struct pt_regs *regs);
 extern int tick_nohz_adaptive_mode(void);
+extern void tick_nohz_cpu_exit_qs(void);
 extern bool tick_nohz_account_tick(void);
 extern void tick_nohz_flush_current_times(bool restart_tick);
 #else /* !CPUSETS_NO_HZ */
kernel/sched.c +1 -0
index 2bcd456..576d0bf 100644
@@ -2504,6 +2504,7 @@ static void cpuset_nohz_restart_tick(void)
 	__get_cpu_var(task_nohz_mode) = 0;
 	tick_nohz_restart_sched_tick();
 	clear_thread_flag(TIF_NOHZ);
+	tick_nohz_cpu_exit_qs();
 }
 
 void cpuset_update_nohz(void)
kernel/time/tick-sched.c +1 -0
index 9a2ba5b..b611b77 100644
@@ -757,6 +757,7 @@ void tick_check_idle(int cpu)
 }
 
 #ifdef CONFIG_CPUSETS_NO_HZ
+DEFINE_PER_CPU(int, nohz_task_ext_qs);
 
 void tick_nohz_exit_kernel(void)
 {

kernel/time/tick-sched.c void tick_nohz_exit_kernel(void) · line 788

Quoted 5 lines
kernel/time/tick-sched.c +2 -0
@@ -783,6 +784,9 @@ void tick_nohz_exit_kernel(void)
 	ts->saved_jiffies = jiffies;
 	ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 
+	__get_cpu_var(nohz_task_ext_qs) = 1;
+	rcu_enter_nohz();

OK, I was wondering how this was going to work if RCU didn't know about kernel entry/exit. Whew!!! ;-)

Quoted 34 lines
kernel/time/tick-sched.c +16 -0
+
 	local_irq_restore(flags);
 }
 
@@ -799,6 +803,11 @@ void tick_nohz_enter_kernel(void)
 		return;
 	}
 
+	if (__get_cpu_var(nohz_task_ext_qs) == 1) {
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+		rcu_exit_nohz();
+	}
+
 	ts = &__get_cpu_var(tick_cpu_sched);
 
 	WARN_ON_ONCE(ts->saved_jiffies_whence == JIFFIES_SAVED_SYS);
@@ -814,6 +823,16 @@ void tick_nohz_enter_kernel(void)
 	local_irq_restore(flags);
 }
 
+void tick_nohz_cpu_exit_qs(void)
+{
+	struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+
+	if (__get_cpu_var(nohz_task_ext_qs)) {
+		rcu_exit_nohz();
+		__get_cpu_var(nohz_task_ext_qs) = 0;
+	}
+}
+
 void tick_nohz_enter_exception(struct pt_regs *regs)
 {
 	if (user_mode(regs))

kernel/time/tick-sched.c static void tick_nohz_cpuset_stop_tick(int user) · line 881

Quoted 5 lines
kernel/time/tick-sched.c +2 -0
@@ -858,6 +877,8 @@ static void tick_nohz_cpuset_stop_tick(int user)
 		if (user) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_USER;
 			ts->saved_jiffies = jiffies;
+			__get_cpu_var(nohz_task_ext_qs) = 1;
+			rcu_enter_nohz();

When entering an exception, shouldn't we call rcu_exit_nohz() rather than rcu_exit_nohz()? Or is this a "didn't really mean an exception" code path?

Quoted 11 lines
kernel/time/tick-sched.c +0 -0
 		} else if (!current->mm) {
 			ts->saved_jiffies_whence = JIFFIES_SAVED_SYS;
 			ts->saved_jiffies = jiffies;
Signature · 7 lines
1.7.5.4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#122 lore raw

PATCH 32/32 nohz/cpuset: Disable under some configs

In reply to Frederic Weisbecker

This shows the various things that are not yet handled by the nohz cpusets: perf events, irq work, irq time accounting.

But there are further things that have yet to be handled:
sched clock tick, runqueue clock, sched_class::task_tick(), rq clock, cpu load, complete handling of cputimes, ...

Cc Andrew Morton <[email protected]>
Cc Anton Blanchard <[email protected]>
Cc Avi Kivity <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Lai Jiangshan <[email protected]>
Cc Paul E . McKenney <[email protected]>
Cc Paul Menage <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Tim Pepper <[email protected]>
---
 init/Kconfig |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
init/Kconfig +1 -1
index 7a144ad..6b6adde 100644
@@ -626,7 +626,7 @@ config PROC_PID_CPUSET
 
 config CPUSETS_NO_HZ
        bool "Tickless cpusets"
-       depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS
+       depends on CPUSETS && HAVE_CPUSETS_NO_HZ && NO_HZ && HIGH_RES_TIMERS && !PERF_EVENTS && !PROFILING && !IRQ_WORK && !IRQ_TIME_ACCOUNTING
        help
          This options let you apply a nohz property to a cpuset such
 	 that the periodic timer tick tries to be avoided when possible on
Signature · 1 line
1.7.5.4
#123 lore raw
In reply to Frederic Weisbecker
Quoted 8 lines · Frederic Weisbecker

= What's the interface =

We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.

Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#124 lore raw
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Frederic Weisbecker

= What's the interface =

We use the cpuset interface by adding a nohz flag to it. As long as a CPU is part of a nohz cpuset, then this CPU will try to enter into adaptive nohz mode when it can, even if it is part of another cpuset that is not nohz.

Quoted 2 lines

Why not do it unconditionally? That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.

Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...

But perhaps if one day it is proven to behave better in most cases then we can make it enabled by default on cpusets?

#125 lore raw
Quoted 6 lines · Frederic Weisbecker
Quoted 3 lines

Why not do it unconditionally?  That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.

Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...

I agree with Avi. I'd be inclined to investigate further to see if there are any important workloads on which it's not a win - and then add the extra complexity to control it from cpusets if necessary. Unless there's really a good reason to make it configurable, it's simpler to make it unconditional.

Paul

#126 lore raw
Quoted 5 lines · Paul Menage
Quoted 4 lines · Frederic Weisbecker
Quoted 3 lines

Why not do it unconditionally?  That is, if all the conditions are fulfilled, disable the tick regardless of any cpuset settings.

Quoted 8 lines
Quoted 2 lines

Because I'm not sure it's a win on every workload. This involves some hooks here and there (syscall slow path), IPIs, etc...

I agree with Avi. I'd be inclined to investigate further to see if there are any important workloads on which it's not a win - and then add the extra complexity to control it from cpusets if necessary. Unless there's really a good reason to make it configurable, it's simpler to make it unconditional.

There is another things. We still need to have a CPU with the periodic tick to maintain the jiffies and walltime progression.

Also on some workloads like HPC (I mean, that's just a personal guess), it may make sense to also migrate the peripherals interrupts affinity to that same CPU that keeps the periodic tick, so that we have only one CPU handling those backrgound things and all the others can be fully dedicated to their main task.

So with these factors combined, we need to be able to precisely choose at least a CPU that doesn't run in adaptive nohz mode. Having that nohz cpuset solves that issue (although I haven't forced to keep one non-nohz cpu but I should).

Perhaps having a simple interface that defines a fixed CPU to handle jiffies/walltime would be enough but the cpuset offers more flexibility. I just can't test every possible workload to know if it has no downside somewhere.

May be a global toggle in sysfs for a global adaptive nohz would be enough? I have no idea.

Is anybody aware of any worload that involve very frequent syscalls? Like more than 100 Hz ?

#127 lore raw
In reply to Frederic Weisbecker

Hi,

Quoted 5 lines · Frederic Weisbecker

For those who want to play:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
       nohz/cpuset-v1

You caught me in playful mood, so I took it for a spin... :-)

I know this is far from being production ready, but I hope you'll find the feedback useful.

First a short description of my testing setup is in order, I believe:

I've set up a small x86 VM with 4 CPUs running your git tree and a minimal buildroot system. I've created 2 cpusets: sys and nohz, and then assigned every task I could to the sys cpuset and set adaptive_nohz on the nohz set.

To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.

I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using /proc/interrupts

Now, for the things I've noticed -

1. Before I turn adaptive_nohz to 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turn adaptive_nohz to 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.

2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for rcu_pending() in cpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.

3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when nr_running

Quoted 1 line

1. In my case, I got a kernel panic. Since it happened with some

debug code I added and with aforementioned experimental removal of rcu_pending check, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.

Cheers,
Gilad

Signature · 9 lines
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"Dance like no one is watching, love like you'll never be hurt, sing
like no one is listening... but for BEEP sake you better code like
you're going to maintain it for years!"
#128 lore raw
Quoted 22 lines · Gilad Ben-Yossef

Hi,

Quoted 5 lines · Frederic Weisbecker

For those who want to play:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
       nohz/cpuset-v1

You caught me in playful mood, so I took it for a spin... :-)

I know this is far from being production ready, but I hope you'll find the feedback useful.

First a short description of my testing setup is in order, I believe:

I've set up a small x86 VM with 4 CPUs running your git tree and a minimal buildroot system. I've created 2 cpusets: sys and nohz, and then assigned every task I could to the sys cpuset and set adaptive_nohz on the nohz set.

Quoted 4 lines

To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

Quoted 5 lines

I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using /proc/interrupts

Quoted 8 lines

Now, for the things I've noticed -

1. Before I turn adaptive_nohz to 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turn adaptive_nohz to 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.

Yeah I believe this is due to RCU that tries to wake up our nohz CPU. I need to have a deeper look there.

Quoted 10 lines

2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for rcu_pending() in cpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.

Are you sure the tick never goes off?
But yeah may be there is something that constantly requires RCU grace periods to complete in your system. I should drop the rcu_pending() check as long as we want to stop the tick from userspace because there we are off the RCU state machine.

Quoted 8 lines

3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when nr_running

Quoted 1 line

1. In my case, I got a kernel panic. Since it happened with some

debug code I added and with aforementioned experimental removal of rcu_pending check, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.

I got some panic too but haven't seen any for some time. I made a lot of changes since then though so I thought the condition to trigger it just went away.

IIRC, it was a locking inversion against the rq lock and some other lock. Very nice condition for a cool lockup ;)

#129 lore raw
Quoted 2 lines · Frederic Weisbecker

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

It is? Is it being replaced by something? (I find it useful)

-Mike

#130 lore raw
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

It is? Is it being replaced by something? (I find it useful)

cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.

[cpuisol seems to already be broken for SCHED_FIFO tasks, since they share the def_root_domain]

#131 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

It is? Is it being replaced by something? (I find it useful)

cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.

-Mike

#132 lore raw
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Mike Galbraith
Quoted 2 lines · Frederic Weisbecker

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

It is? Is it being replaced by something? (I find it useful)

cpusets? Afaict all it does is not include a number of cpus in the sched domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.

You need to play with the sched_load_balance file, no idea what this cset utility is though, never encountered it before.

#133 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line

cpusets? Afaict all it does is not include a number of cpus in the sched

I think you meant to write cpuisol here :-)

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus.  cset shield --cpu 4-63 didn't work well enough.

You need to play with the sched_load_balance file, no idea what this cset utility is though, never encountered it before.

It's just a python wrapper around the /cgroup file system.

Gilad

Signature · 9 lines
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"Dance like no one is watching, love like you'll never be hurt, sing
like no one is listening... but for BEEP sake you better code like
you're going to maintain it for years!"
#134 lore raw
Quoted 3 lines · Gilad Ben-Yossef
Quoted 1 line
Quoted 1 line
Quoted 1 line

cpusets? Afaict all it does is not include a number of cpus in the sched

I think you meant to write cpuisol here :-)

Uhh, yeah ;-)

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

domains, creating a bunch of independent scheduling cpus. You can use cpusets to get to the same state.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.

You need to play with the sched_load_balance file, no idea what this cset utility is though, never encountered it before.

It's just a python wrapper around the /cgroup file system.

ok, just never encountered it.

#135 lore raw
In reply to Mike Galbraith
Quoted 2 lines

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus.  cset shield --cpu 4-63 didn't work well enough.

Try adding:

# cset shield --kthread on

It will try to move non bound kernel threads to the system set from the root set.

The results seems to vary from introducing system instability, via doing nothing at all, all the way to actually achieving better isolation, depending on the specific kernel threads you have running and your kernel version. Sometime more then one of the results at the same time. It's really like Russian roulette in a way, only way more geekier...

Signature · 9 lines
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"Dance like no one is watching, love like you'll never be hurt, sing
like no one is listening... but for BEEP sake you better code like
you're going to maintain it for years!"
#136 lore raw
Quoted 6 lines · Gilad Ben-Yossef
Quoted 2 lines

Guess I'll have to try creating a cpuset per cpu, and see how it compares to isolcpus. cset shield --cpu 4-63 didn't work well enough.

Try adding:

# cset shield --kthread on

Yeah, did that.

Killing load balancing in the cpuset should eliminate the problem. Well, for the test app I was running anyway, which has everything pinned, and pure RT.

-Mike

#137 lore raw
In reply to Frederic Weisbecker
Quoted 10 lines · Frederic Weisbecker
Quoted 6 lines

To make double sure I have no task on my nohz cpuset CPU, I've booted the system with the isolcpus command line isolating the same cpu I've assigned to the nohz set. This shouldn't be needed of course, but just in case.

Ah I haven't tested with that isolcpus especially as it's headed toward removal.

I had the cpuisol option in the boot loader config but I did set up a proper cpuset as well, so I believe it made no difference.

I added the cpuisol option after noticing how many tasks I was unable to move from the root cpuset to the system cpuset due to them being bound per CPU and hoped (for no good reason, I admit) that cpuisol will somehow help with the task isolation I wanted to get to test the nohz task.

That's a different obstacle to workload of the kind where nohz cpuset would be useful, but should probably be discussed in another thread :-)

Quoted 15 lines
Quoted 6 lines

I then ran a silly program I've written that basically eats CPU cycles
(https://github.com/gby/cpueat) and assigned it to the nohz set and monitored the number of interrupts using /proc/interrupts

Quoted 8 lines

Now, for the things I've noticed -

1. Before I turn adaptive_nohz to 1, when no task is running on the nohz cpuset cpu, the tick is indeed idle (regular nohz case) and very few function call IPIs are seen. However, when I turn adaptive_nohz to 1 (but still with no task running on the CPU), the tick remains idle, but I get an IPI function call interrupt almost in the rate the tick would have been.

Quoted 2 lines

Yeah I believe this is due to RCU that tries to wake up our nohz CPU. I need to have a deeper look there.

I believe you are right with the reason for the IPI.

Before setting adaptive_nohz for the cpuset I did not get the IPI on an idle CPU. After setting it  I started getting the IPI regularly even when the CPU was idle.

Quoted 11 lines
Quoted 10 lines

2. When I run my little cpueat program on the nohz CPU, the tick does not actually goes off. Instead it ticks away as usual. I know it is the only legible task to run, since as soon as I kill it  the tick turns off (regular nohz mode again). I've tinkered around and found out that what stops the tick going away is the check for rcu_pending() in cpuset_nohz_can_stop_tick(). It seems to always be true. When I removed that check experimentally and repeat the test, the tick indeed stops with my cpueat task running. Of course, I don't suggest this is the sane thing to do - I just wondered if that what stopped the tick going away and it seems that it is.

Quoted 1 line

Are you sure the tick never goes off?

Yes, I put debug code in the cpuset_nohz_can_stop_tick(). Every time the function was called for that CPU rcu_pending() returned 1.

Quoted 4 lines

But yeah may be there is something that constantly requires RCU grace periods to complete in your system. I should drop the rcu_pending() check as long as we want to stop the tick from userspace because there we are off the RCU state machine.

I added debug code to rcu_pending() and noticed that the rcu_bh was the one pending each time.

I found that odd - my VM didn't even have a netowrk interface configured (except maybe for lo), let alone saw any network traffic. I thought rcu_bh was mostly used for networking code (Paul?)

Quoted 9 lines
Quoted 8 lines

3. My little cpueat program tries to fork a child process after 100k iteration of some CPU bound loop. It usually takes a few seconds to happen. The idea is to make sure that the tick resumes when nr_running

Quoted 1 line

1. In my case, I got a kernel panic. Since it happened with some

debug code I added and with aforementioned experimental removal of rcu_pending check, I'm assuming for now it's all my fault but will look into verifying it further and will send panic logs if it proves useful.

Quoted 6 lines

I got some panic too but haven't seen any for some time. I made a lot of changes since then though so I thought the condition to trigger it just went away.

IIRC, it was a locking inversion against the rq lock and some other lock. Very nice condition for a cool lockup ;)

It certainly sounds exciting :-)

Let me know if I an help test anything else.

Thanks,
Gilad

--
Gilad Ben-Yossef
Chief Coffee Drinker
[email protected]
Israel Cell: +972-52-8260388
US Cell: +1-973-8260388
http://benyossef.com
"Dance like no one is watching, love like you'll never be hurt, sing like no one is listening... but for BEEP sake you better code like you're going to maintain it for years!"

#138 lore raw
Quoted 3 lines · Gilad Ben-Yossef

I added the cpuisol option after noticing how many tasks I was unable to move from the root cpuset to the system cpuset due to them being bound per CPU

Right, so ideally those tasks should be idle and not interfere. Where this is not so, we should make it so.

When userspace didn't ask for anything to happen, nothing should happen. When it did ask for it, well then it shouldn't complain it does :-)

Furthermore things like:

linux-2.6# git grep on_each_cpu mm/

mm/page_alloc.c:        on_each_cpu(drain_local_pages, NULL, 1);
mm/slab.c:      on_each_cpu(do_drain, cachep, 1);
mm/slab.c:      on_each_cpu(do_ccupdate_local, (void *)new, 1);
mm/slub.c:      on_each_cpu(flush_cpu_slab, s, 1);
mm/swap.c:      return schedule_on_each_cpu(lru_add_drain_per_cpu);

Should be converted to smp_call_function_many() and for each we should keep a cpumask of cpus where there's work to do, avoiding disturbing cpus that have been quiet.