isolation: 1Hz residual tick offloading v3

PATCH New Frederic Weisbecker 1 participant 6 messages Dec 30, 2017

Subsystems (3)

Documentation Scheduler Nohz, dynticks support

lore

Share & tools

lore

Frederic Weisbecker

author maintainer

2017-12-30 03:55 UTC #0 lore raw

cover isolation: 1Hz residual tick offloading v3

No big change in this row, just a build failure fix on patch 4/5 due to sched_tick_start()/sched_tick_stop() not having off cases.

If no more comment arise, I'll do a pull request in a few days.

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git

	timers/0z-v3

HEAD: 4b07980eaca4012852b5f02c2a55b1114c475804

Thanks,
Frederic
---

Frederic Weisbecker (5):
sched: Rename init_rq_hrtick to hrtick_rq_init sched/isolation: Add scheduler tick offloading interface
nohz: Allow to check if remote CPU tick is stopped sched/isolation: Residual 1Hz scheduler tick offload sched/isolation: Document "nohz_offload" flag

 Documentation/admin-guide/kernel-parameters.txt |  7 +-
 include/linux/sched/isolation.h                 |  3 +-
 include/linux/tick.h                            |  2 +
 kernel/sched/core.c                             | 94 +++++++++++++++++++++++--
 kernel/sched/isolation.c                        | 10 +++
 kernel/sched/sched.h                            |  2 +
 kernel/time/tick-sched.c                        |  7 ++
 7 files changed, 117 insertions(+), 8 deletions(-)

Frederic Weisbecker

author maintainer

2017-12-30 03:55 UTC #1 lore raw

PATCH 1/5 sched: Rename init_rq_hrtick to hrtick_rq_init

Do that rename in order to normalize the hrtick namespace.

Signed-off-by Frederic Weisbecker <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Luiz Capitulino <[email protected]>

Cc Mike Galbraith <[email protected]>

Cc Paul E. McKenney <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Ingo Molnar <[email protected]>

---
 kernel/sched/core.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

kernel/sched/core.c +3 -3

index 644fa2e..d72d0e9 100644
@@ -333,7 +333,7 @@ void hrtick_start(struct rq *rq, u64 delay)
 }
 #endif /* CONFIG_SMP */
 
-static void init_rq_hrtick(struct rq *rq)
+static void hrtick_rq_init(struct rq *rq)
 {
 #ifdef CONFIG_SMP
 	rq->hrtick_csd_pending = 0;

@@ -351,7 +351,7 @@ static inline void hrtick_clear(struct rq *rq)
 {
 }
 
-static inline void init_rq_hrtick(struct rq *rq)
+static inline void hrtick_rq_init(struct rq *rq)
 {
 }
 #endif	/* CONFIG_SCHED_HRTICK */

@@ -5955,7 +5955,7 @@ void __init sched_init(void)
 		rq->last_sched_tick = 0;
 #endif
 #endif /* CONFIG_SMP */
-		init_rq_hrtick(rq);
+		hrtick_rq_init(rq);
 		atomic_set(&rq->nr_iowait, 0);
 	}

Signature · 1 line

2.7.4

Frederic Weisbecker

author maintainer

2017-12-30 03:55 UTC #2 lore raw

PATCH 2/5 sched/isolation: Add scheduler tick offloading interface

In reply to Frederic Weisbecker

Add the boot option that will allow us to offload the 1Hz scheduler tick to the housekeeping CPU.

Signed-off-by Frederic Weisbecker <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Luiz Capitulino <[email protected]>

Cc Mike Galbraith <[email protected]>

Cc Paul E. McKenney <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Ingo Molnar <[email protected]>

---
 include/linux/sched/isolation.h | 3 ++-
 kernel/sched/isolation.c        | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

include/linux/sched/isolation.h +2 -1

index d849431..c831855 100644
@@ -11,7 +11,8 @@ enum hk_flags {
 	HK_FLAG_MISC		= (1 << 2),
 	HK_FLAG_SCHED		= (1 << 3),
 	HK_FLAG_TICK		= (1 << 4),
-	HK_FLAG_DOMAIN		= (1 << 5),
+	HK_FLAG_TICK_SCHED	= (1 << 5),
+	HK_FLAG_DOMAIN		= (1 << 6),
 };
 
 #ifdef CONFIG_CPU_ISOLATION

kernel/sched/isolation.c +6 -0

index b71b436..264ddcd 100644
@@ -136,6 +136,12 @@ static int __init housekeeping_isolcpus_setup(char *str)
 			continue;
 		}
 
+		if (!strncmp(str, "nohz_offload,", 13)) {
+			str += 13;
+			flags |= HK_FLAG_TICK | HK_FLAG_TICK_SCHED;
+			continue;
+		}
+
 		if (!strncmp(str, "domain,", 7)) {
 			str += 7;
 			flags |= HK_FLAG_DOMAIN;

Signature · 1 line

2.7.4

Frederic Weisbecker

author maintainer

2017-12-30 03:56 UTC #3 lore raw

PATCH 3/5 nohz: Allow to check if remote CPU tick is stopped

In reply to Frederic Weisbecker

This check is racy but provides a good heuristic to determine whether a CPU may need a remote tick or not.

Signed-off-by Frederic Weisbecker <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Luiz Capitulino <[email protected]>

Cc Mike Galbraith <[email protected]>

Cc Paul E. McKenney <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Ingo Molnar <[email protected]>

---
 include/linux/tick.h     | 2 ++
 kernel/time/tick-sched.c | 7 +++++++
 2 files changed, 9 insertions(+)

include/linux/tick.h +2 -0

index f442d1a..417f8d2 100644
@@ -114,6 +114,7 @@ enum tick_dep_bits {
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;
 extern int tick_nohz_tick_stopped(void);
+extern int tick_nohz_tick_stopped_cpu(int cpu);
 extern void tick_nohz_idle_enter(void);
 extern void tick_nohz_idle_exit(void);
 extern void tick_nohz_irq_exit(void);

@@ -124,6 +125,7 @@ extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
 #else /* !CONFIG_NO_HZ_COMMON */
 #define tick_nohz_enabled (0)
 static inline int tick_nohz_tick_stopped(void) { return 0; }
+static inline int tick_nohz_tick_stopped_cpu(int cpu) { return 0; }
 static inline void tick_nohz_idle_enter(void) { }
 static inline void tick_nohz_idle_exit(void) { }

kernel/time/tick-sched.c +7 -0

index 99578f0..ad07d6e 100644
@@ -486,6 +486,13 @@ int tick_nohz_tick_stopped(void)
 	return __this_cpu_read(tick_cpu_sched.tick_stopped);
 }
 
+int tick_nohz_tick_stopped_cpu(int cpu)
+{
+	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
+
+	return ts->tick_stopped;
+}
+
 /**
  * tick_nohz_update_jiffies - update jiffies when idle was interrupted
  *

Signature · 1 line

2.7.4

Frederic Weisbecker

author maintainer

2017-12-30 03:55 UTC #4 lore raw

PATCH 4/5 sched/isolation: Residual 1Hz scheduler tick offload

In reply to Frederic Weisbecker

When a CPU runs in full dynticks mode, a 1Hz tick remains in order to keep the scheduler stats alive. However this residual tick is a burden for bare metal tasks that can't stand no interruption at all, or want to minimize them.

Adding the boot parameter "isolcpus=nohz_offload" will now outsource these scheduler ticks to the global workqueue so that a housekeeping CPU handles that tick remotely.

Note it's still up to the user to affine the global workqueues to the housekeeping CPUs through /sys/devices/virtual/workqueue/cpumask or domains isolation.

Signed-off-by Frederic Weisbecker <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Luiz Capitulino <[email protected]>

Cc Mike Galbraith <[email protected]>

Cc Paul E. McKenney <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Ingo Molnar <[email protected]>

---
 kernel/sched/core.c      | 88 ++++++++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/isolation.c |  4 +++
 kernel/sched/sched.h     |  2 ++
 3 files changed, 91 insertions(+), 3 deletions(-)

kernel/sched/core.c +85 -3

index d72d0e9..b964890 100644
@@ -3052,9 +3052,14 @@ void scheduler_tick(void)
  */
 u64 scheduler_tick_max_deferment(void)
 {
-	struct rq *rq = this_rq();
-	unsigned long next, now = READ_ONCE(jiffies);
+	struct rq *rq;
+	unsigned long next, now;
 
+	if (!housekeeping_cpu(smp_processor_id(), HK_FLAG_TICK_SCHED))
+		return ktime_to_ns(KTIME_MAX);
+
+	rq = this_rq();
+	now = READ_ONCE(jiffies);
 	next = rq->last_sched_tick + HZ;
 
 	if (time_before_eq(next, now))

@@ -3062,7 +3067,82 @@ u64 scheduler_tick_max_deferment(void)
 
 	return jiffies_to_nsecs(next - now);
 }
-#endif
+
+struct tick_work {
+	int			cpu;
+	struct delayed_work	work;
+};
+
+static struct tick_work __percpu *tick_work_cpu;
+
+static void sched_tick_remote(struct work_struct *work)
+{
+	struct delayed_work *dwork = to_delayed_work(work);
+	struct tick_work *twork = container_of(dwork, struct tick_work, work);
+	int cpu = twork->cpu;
+	struct rq *rq = cpu_rq(cpu);
+	struct rq_flags rf;
+
+	/*
+	 * Handle the tick only if it appears the remote CPU is running
+	 * in full dynticks mode. The check is racy by nature, but
+	 * missing a tick or having one too much is no big deal.
+	 */
+	if (!idle_cpu(cpu) && tick_nohz_tick_stopped_cpu(cpu)) {
+		rq_lock_irq(rq, &rf);
+		update_rq_clock(rq);
+		rq->curr->sched_class->task_tick(rq, rq->curr, 0);
+		rq_unlock_irq(rq, &rf);
+	}
+
+	queue_delayed_work(system_unbound_wq, dwork, HZ);
+}
+
+static void sched_tick_start(int cpu)
+{
+	struct tick_work *twork;
+
+	if (housekeeping_cpu(cpu, HK_FLAG_TICK_SCHED))
+		return;
+
+	WARN_ON_ONCE(!tick_work_cpu);
+
+	twork = per_cpu_ptr(tick_work_cpu, cpu);
+	twork->cpu = cpu;
+	INIT_DELAYED_WORK(&twork->work, sched_tick_remote);
+	queue_delayed_work(system_unbound_wq, &twork->work, HZ);
+}
+
+#ifdef CONFIG_HOTPLUG_CPU
+static void sched_tick_stop(int cpu)
+{
+	struct tick_work *twork;
+
+	if (housekeeping_cpu(cpu, HK_FLAG_TICK_SCHED))
+		return;
+
+	WARN_ON_ONCE(!tick_work_cpu);
+
+	twork = per_cpu_ptr(tick_work_cpu, cpu);
+	cancel_delayed_work_sync(&twork->work);
+}
+#endif /* CONFIG_HOTPLUG_CPU */
+
+int __init sched_tick_offload_init(void)
+{
+	tick_work_cpu = alloc_percpu(struct tick_work);
+	if (!tick_work_cpu) {
+		pr_err("Can't allocate remote tick struct\n");
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
+#else
+static void sched_tick_start(int cpu) { }
+static void sched_tick_stop(int cpu) { }
+#endif /* CONFIG_NO_HZ_FULL */
 
 #if defined(CONFIG_PREEMPT) && (defined(CONFIG_DEBUG_PREEMPT) || \
 				defined(CONFIG_PREEMPT_TRACER))

@@ -5713,6 +5793,7 @@ int sched_cpu_starting(unsigned int cpu)
 {
 	set_cpu_rq_start_time(cpu);
 	sched_rq_cpu_starting(cpu);
+	sched_tick_start(cpu);
 	return 0;
 }

@@ -5724,6 +5805,7 @@ int sched_cpu_dying(unsigned int cpu)
 
 	/* Handle pending wakeups and then migrate everything off */
 	sched_ttwu_pending();
+	sched_tick_stop(cpu);
 
 	rq_lock_irqsave(rq, &rf);
 	if (rq->rd) {

kernel/sched/isolation.c +4 -0

index 264ddcd..c5e7e90a 100644
@@ -12,6 +12,7 @@
 #include <linux/kernel.h>
 #include <linux/static_key.h>
 #include <linux/ctype.h>
+#include "sched.h"
 
 DEFINE_STATIC_KEY_FALSE(housekeeping_overriden);
 EXPORT_SYMBOL_GPL(housekeeping_overriden);

@@ -60,6 +61,9 @@ void __init housekeeping_init(void)
 
 	static_branch_enable(&housekeeping_overriden);
 
+	if (housekeeping_flags & HK_FLAG_TICK_SCHED)
+		sched_tick_offload_init();
+
 	/* We need at least one CPU to handle housekeeping work */
 	WARN_ON_ONCE(cpumask_empty(housekeeping_mask));
 }

kernel/sched/sched.h +2 -0

index b19552a2..5a3b82c 100644
@@ -1587,6 +1587,7 @@ extern void post_init_entity_util_avg(struct sched_entity *se);
 
 #ifdef CONFIG_NO_HZ_FULL
 extern bool sched_can_stop_tick(struct rq *rq);
+extern int __init sched_tick_offload_init(void);
 
 /*
  * Tick may be needed by tasks in the runqueue depending on their policy and

@@ -1611,6 +1612,7 @@ static inline void sched_update_tick_dependency(struct rq *rq)
 		tick_nohz_dep_set_cpu(cpu, TICK_DEP_BIT_SCHED);
 }
 #else
+static inline int sched_tick_offload_init(void) { return 0; }
 static inline void sched_update_tick_dependency(struct rq *rq) { }
 #endif

Signature · 1 line

2.7.4

Frederic Weisbecker

author maintainer

2017-12-30 03:55 UTC #5 lore raw

PATCH 5/5 sched/isolation: Document "nohz_offload" flag

In reply to Frederic Weisbecker

Document the interface to offload the 1Hz scheduler tick in full dynticks mode. Also improve the comment about the existing "nohz" flag in order to differentiate its behaviour.

Signed-off-by Frederic Weisbecker <[email protected]>

Cc Chris Metcalf <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Luiz Capitulino <[email protected]>

Cc Mike Galbraith <[email protected]>

Cc Paul E. McKenney <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Rik van Riel <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Wanpeng Li <[email protected]>

Cc Ingo Molnar <[email protected]>

---
 Documentation/admin-guide/kernel-parameters.txt | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

Documentation/admin-guide/kernel-parameters.txt +6 -1

index 1683107..fcc5fd9 100644
@@ -1745,7 +1745,12 @@
 			specified in the flag list (default: domain):
 
 			nohz
-			  Disable the tick when a single task runs.
+			  Disable the tick when a single task runs. A residual 1Hz
+			  tick remains to maintain scheduler stats alive.
+			nohz_offload
+			  Like nohz but the residual 1Hz tick is offloaded to
+			  housekeeping CPUs, leaving the CPU free of any tick if
+			  nothing else requests it.
 			domain
 			  Isolate from the general SMP balancing and scheduling
 			  algorithms. Note that performing domain isolation this way

Signature · 1 line

2.7.4