kernel/sched: run nohz idle load balancer on HK_FLAG_MISC CPUs

PATCH In review Nicholas Piggin 4 participants 6 messages Apr 12, 2019 – Apr 29, 2019 Scheduler

Share & tools

Nicholas Piggin

author

2019-04-12 04:26 UTC #0 lore raw

PATCH 1/1 kernel/sched: run nohz idle load balancer on HK_FLAG_MISC CPUs

The nohz idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.

HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.

The problem was observed with increased jitter on an application running on CPU0, caused by nohz idle load balancing being run on CPU1 (an SMT sibling).

Signed-off-by Nicholas Piggin <[email protected]>

---
 kernel/sched/fair.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

kernel/sched/fair.c +10 -6

index fdab7eb6f351..d29ca323214d 100644
@@ -9522,22 +9522,26 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notice that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ * - HK_FLAG_MISC CPUs are used for this task, because HK_FLAG_SCHED not set
+ *   anywhere yet.
  */
 
 static inline int find_new_ilb(void)
 {
-	int ilb = cpumask_first(nohz.idle_cpus_mask);
+	int ilb;
 
-	if (ilb < nr_cpu_ids && idle_cpu(ilb))
-		return ilb;
+	for_each_cpu_and(ilb, nohz.idle_cpus_mask,
+			      housekeeping_cpumask(HK_FLAG_MISC)) {
+		if (idle_cpu(ilb))
+			return ilb;
+	}
 
 	return nr_cpu_ids;
 }
 
 /*
- * Kick a CPU to do the nohz balancing, if it is time for it. We pick the
- * nohz_load_balancer CPU (if there is one) otherwise fallback to any idle
- * CPU (if there is one).
+ * Kick a CPU to do the nohz balancing, if it is time for it. We pick any
+ * idle CPU in the HK_FLAG_MISC housekeeping set (if there is one).
  */
 static void kick_ilb(unsigned int flags)
 {

Signature · 1 line

2.20.1

Peter Zijlstra

maintainer

2019-04-25 11:56 UTC #1 lore raw

Quoted 7 lines · Nicholas Piggin

The nohz idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.
HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.

Frederic? Anyway, I thnk I'll take this patch as is.

Thanks!

Quoted 49 lines

The problem was observed with increased jitter on an application running on CPU0, caused by nohz idle load balancing being run on CPU1 (an SMT sibling).

Signed-off-by Nicholas Piggin <[email protected]>

---
 kernel/sched/fair.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

kernel/sched/fair.c +10 -6

index fdab7eb6f351..d29ca323214d 100644
@@ -9522,22 +9522,26 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notice that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ * - HK_FLAG_MISC CPUs are used for this task, because HK_FLAG_SCHED not set
+ *   anywhere yet.
  */
 
 static inline int find_new_ilb(void)
 {
-	int ilb = cpumask_first(nohz.idle_cpus_mask);
+	int ilb;
 
-	if (ilb < nr_cpu_ids && idle_cpu(ilb))
-		return ilb;
+	for_each_cpu_and(ilb, nohz.idle_cpus_mask,
+			      housekeeping_cpumask(HK_FLAG_MISC)) {
+		if (idle_cpu(ilb))
+			return ilb;
+	}
 
 	return nr_cpu_ids;
 }
 
 /*
- * Kick a CPU to do the nohz balancing, if it is time for it. We pick the
- * nohz_load_balancer CPU (if there is one) otherwise fallback to any idle
- * CPU (if there is one).
+ * Kick a CPU to do the nohz balancing, if it is time for it. We pick any
+ * idle CPU in the HK_FLAG_MISC housekeeping set (if there is one).
  */
 static void kick_ilb(unsigned int flags)
 {

Signature · 1 line

2.20.1

Nicholas Piggin

author

2019-04-26 04:41 UTC #2 lore raw

Peter Zijlstra's on April 25, 2019 9:56 pm:

Quoted 10 lines · Nicholas Piggin

Quoted 7 lines
The nohz idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.
HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.
Frederic? Anyway, I thnk I'll take this patch as is.

That would be great, thanks. We've been testing it in a staging environment (this is where they noticed the noise in the first place), and results have been as expected:

I've been able to test Nick's idle-loop load balancer (ILB) patch, with and without the TEO cpuidle governor. With the ILB patch (and nohz_full) I get a very quiet noise profile with either cpuidle governor (menu or teo). For my tests, I don't see a meaningful difference between the two governors.

[...]

Bottom line: Nick's patch that constrains the ILB to run on non-nohz cores has a noticeable noise-reduction effect. For this type of workload, the choice of cpuidle governor, menu or teo, is immaterial.

This is against a slightly backported RHEL kernel they are using, but no significant differences from upstream in these areas.

Thanks,
Nick

Wanpeng Li

2019-04-28 07:01 UTC #3 lore raw

In reply to Nicholas Piggin

Quoted 32 lines · Nicholas Piggin

The nohz idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.
HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.
The problem was observed with increased jitter on an application running on CPU0, caused by nohz idle load balancing being run on CPU1 (an SMT sibling).
Signed-off-by Nicholas Piggin <[email protected]>
---
 kernel/sched/fair.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
kernel/sched/fair.c +2 -0
index fdab7eb6f351..d29ca323214d 100644
@@ -9522,22 +9522,26 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notice that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ * - HK_FLAG_MISC CPUs are used for this task, because HK_FLAG_SCHED not set
+ *   anywhere yet.
  */
 
 static inline int find_new_ilb(void)
 {

kernel/sched/fair.c static inline int on_null_domain(struct rq *rq) · line 9537

Quoted 10 lines

kernel/sched/fair.c +6 -3

-       int ilb = cpumask_first(nohz.idle_cpus_mask);
+       int ilb;
 
-       if (ilb < nr_cpu_ids && idle_cpu(ilb))
-               return ilb;
+       for_each_cpu_and(ilb, nohz.idle_cpus_mask,
+                             housekeeping_cpumask(HK_FLAG_MISC)) {
+               if (idle_cpu(ilb))
+                       return ilb;
+       }

What will happen if cpu1 is still idle currently && housekeeping?

Regards,
Wanpeng Li

Quoted 16 lines

kernel/sched/fair.c +2 -4

        return nr_cpu_ids;
 }
 
 /*
- * Kick a CPU to do the nohz balancing, if it is time for it. We pick the
- * nohz_load_balancer CPU (if there is one) otherwise fallback to any idle
- * CPU (if there is one).
+ * Kick a CPU to do the nohz balancing, if it is time for it. We pick any
+ * idle CPU in the HK_FLAG_MISC housekeeping set (if there is one).
  */
 static void kick_ilb(unsigned int flags)
 {
--

2.20.1

Nicholas Piggin

author

2019-04-28 11:58 UTC #4 lore raw

Wanpeng Li's on April 28, 2019 5:01 pm:

Quoted 36 lines · Nicholas Piggin

Quoted 35 lines
The nohz idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.
HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.
The problem was observed with increased jitter on an application running on CPU0, caused by nohz idle load balancing being run on CPU1 (an SMT sibling).
Signed-off-by Nicholas Piggin <[email protected]>
---
 kernel/sched/fair.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
kernel/sched/fair.c +3 -1
index fdab7eb6f351..d29ca323214d 100644
@@ -9522,22 +9522,26 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notice that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ * - HK_FLAG_MISC CPUs are used for this task, because HK_FLAG_SCHED not set
+ *   anywhere yet.
  */
 
 static inline int find_new_ilb(void)
 {
-       int ilb = cpumask_first(nohz.idle_cpus_mask);
+       int ilb;
 

Quoted 9 lines

kernel/sched/fair.c static inline int on_null_domain(struct rq *rq) · line 9537

Quoted 7 lines

kernel/sched/fair.c +5 -2

-       if (ilb < nr_cpu_ids && idle_cpu(ilb))
-               return ilb;
+       for_each_cpu_and(ilb, nohz.idle_cpus_mask,
+                             housekeeping_cpumask(HK_FLAG_MISC)) {
+               if (idle_cpu(ilb))
+                       return ilb;
+       }

What will happen if cpu1 is still idle currently && housekeeping?

Then it will run the ilb.

In the customer scenario, cpu1 will not be in the housekeeping mask.

Thanks,
Nick

tip-bot for Nicholas Piggin

merge bot

2019-04-29 06:38 UTC #5 lore raw

[tip:sched/core] sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs

In reply to Nicholas Piggin

sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs

Commit-ID:  9b019acb72e4b5741d88e8936d6f200ed44b66b2
Gitweb:     https://git.kernel.org/tip/9b019acb72e4b5741d88e8936d6f200ed44b66b2
Author:     Nicholas Piggin <[email protected]>
AuthorDate: Fri, 12 Apr 2019 14:26:13 +1000
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 29 Apr 2019 08:27:03 +0200

sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs

The NOHZ idle balancer runs on the lowest idle CPU. This can interfere with isolated CPUs, so confine it to HK_FLAG_MISC housekeeping CPUs.

HK_FLAG_SCHED is not used for this because it is not set anywhere at the moment. This could be folded into HK_FLAG_SCHED once that option is fixed.

The problem was observed with increased jitter on an application running on CPU0, caused by NOHZ idle load balancing being run on CPU1 (an SMT sibling).

Signed-off-by Nicholas Piggin <[email protected]>

Signed-off-by Peter Zijlstra (Intel) <[email protected]>

Cc Frederic Weisbecker <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Gleixner <[email protected]>

Link https://lkml.kernel.org/r/[email protected] open here

Signed-off-by Ingo Molnar <[email protected]>

---
 kernel/sched/fair.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

kernel/sched/fair.c +10 -6

index 13bafe350abf..7b0da7007da3 100644
@@ -9519,22 +9519,26 @@ static inline int on_null_domain(struct rq *rq)
  * - When one of the busy CPUs notice that there may be an idle rebalancing
  *   needed, they will kick the idle load balancer, which then does idle
  *   load balancing for all the idle CPUs.
+ * - HK_FLAG_MISC CPUs are used for this task, because HK_FLAG_SCHED not set
+ *   anywhere yet.
  */
 
 static inline int find_new_ilb(void)
 {
-	int ilb = cpumask_first(nohz.idle_cpus_mask);
+	int ilb;
 
-	if (ilb < nr_cpu_ids && idle_cpu(ilb))
-		return ilb;
+	for_each_cpu_and(ilb, nohz.idle_cpus_mask,
+			      housekeeping_cpumask(HK_FLAG_MISC)) {
+		if (idle_cpu(ilb))
+			return ilb;
+	}
 
 	return nr_cpu_ids;
 }
 
 /*
- * Kick a CPU to do the nohz balancing, if it is time for it. We pick the
- * nohz_load_balancer CPU (if there is one) otherwise fallback to any idle
- * CPU (if there is one).
+ * Kick a CPU to do the nohz balancing, if it is time for it. We pick any
+ * idle CPU in the HK_FLAG_MISC housekeeping set (if there is one).
  */
 static void kick_ilb(unsigned int flags)
 {