optimize the locking in the rebalance_domains()

Ingo, we've noticed that rebalance_domains() will try to take a lock every time it's called (every jiffy)

if SD_SERIALIZE is set (which is a
default configuration). This is done regardless the fact that maybe there hasn't passed enough time since the last rebalancing in which case there is no need to take a lock the first place.

The above creates a heavy false sharing problem on the "balancing" spin-lock on large SMP systems: try_lock() is implemented with an (atomic) xchng instruction which invalidates the cache line "balancing" belongs to and therefore creates an intensive cross-NUMA-nodes traffic.

The below patch will minimize the above phenomena to the time slots it's really needed, namely when the "interval" has really passed.

Pls., comment.
thanks,
vlad

---
 kernel/sched/fair.c |   20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

kernel/sched/fair.c +11 -9

index c099cc6..6777d38 100644
@@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
 		interval = msecs_to_jiffies(interval);
 		interval = clamp(interval, 1UL, max_load_balance_interval);
 
+                if (!time_after_eq(jiffies, sd->last_balance + interval))
+			goto out;
+
 		need_serialize = sd->flags & SD_SERIALIZE;
 
 		if (need_serialize) {

@@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
 				goto out;
 		}
 
-		if (time_after_eq(jiffies, sd->last_balance + interval)) {
-			if (load_balance(cpu, rq, sd, idle, &balance)) {
-				/*
-				 * We've pulled tasks over so either we're no
-				 * longer idle.
-				 */
-				idle = CPU_NOT_IDLE;
-			}
-			sd->last_balance = jiffies;
+		if (load_balance(cpu, rq, sd, idle, &balance)) {
+			/*
+			 * We've pulled tasks over so either we're no
+			 * longer idle.
+			 */
+			idle = CPU_NOT_IDLE;
 		}
+		sd->last_balance = jiffies;
+
 		if (need_serialize)
 			spin_unlock(&balancing);
 out:

Ingo, we've noticed that rebalance_domains() will try to take a lock every time it's called (every jiffy) if SD_SERIALIZE is set (which is a default configuration). This is done regardless the fact that maybe there hasn't passed enough time since the last rebalancing in which case there is no need to take a lock the first place.

The above creates a heavy false sharing problem on the "balancing" spin-lock on large SMP systems: try_lock() is implemented with an (atomic) xchng instruction which invalidates the cache line "balancing" belongs to and therefore creates an intensive cross-NUMA-nodes traffic.

The below patch will minimize the above phenomena to the time slots it's really needed, namely when the "interval" has really passed.

Pls., comment.

thanks,
vlad

---
 kernel/sched/fair.c |   20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

kernel/sched/fair.c +0 -0

index c099cc6..6777d38 100644

kernel/sched/fair.c +8 -9

 		need_serialize = sd->flags & SD_SERIALIZE;
 
 		if (need_serialize) {

@@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
 				goto out;
 		}
 
-		if (time_after_eq(jiffies, sd->last_balance + interval)) {
-			if (load_balance(cpu, rq, sd, idle, &balance)) {
-				/*
-				 * We've pulled tasks over so either we're no
-				 * longer idle.
-				 */
-				idle = CPU_NOT_IDLE;
-			}
-			sd->last_balance = jiffies;
+		if (load_balance(cpu, rq, sd, idle, &balance)) {
+			/*
+			 * We've pulled tasks over so either we're no
+			 * longer idle.
+			 */
+			idle = CPU_NOT_IDLE;
 		}
+		sd->last_balance = jiffies;
+
 		if (need_serialize)
 			spin_unlock(&balancing);
 out:

Hi, Vlad

Quoted 31 lines · Vlad Zolotarov

Ingo, we've noticed that rebalance_domains() will try to take a lock every time it's called (every jiffy) if SD_SERIALIZE is set (which is a default configuration). This is done regardless the fact that maybe there hasn't passed enough time since the last rebalancing in which case there is no need to take a lock the first place.

The above creates a heavy false sharing problem on the "balancing" spin-lock on large SMP systems: try_lock() is implemented with an (atomic) xchng instruction which invalidates the cache line "balancing" belongs to and therefore creates an intensive cross-NUMA-nodes traffic.

The below patch will minimize the above phenomena to the time slots it's really needed, namely when the "interval" has really passed.

Pls., comment.

thanks,
vlad

---
 kernel/sched/fair.c |   20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

kernel/sched/fair.c +0 -0

index c099cc6..6777d38 100644
@@ -4689,6 +4689,9 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
 		interval = msecs_to_jiffies(interval);
 		interval = clamp(interval, 1UL, max_load_balance_interval);
 

kernel/sched/fair.c static void rebalance_domains(int cpu, enum cpu_idle_type idle) · line 4694

Quoted 3 lines

kernel/sched/fair.c +3 -0

+                if (!time_after_eq(jiffies, sd->last_balance + interval))
+			goto out;
+

First line looks like white-space-damaged.

Anyway, wouldn't it be better using time_before() here?

Thanks,
Namhyung

Quoted 28 lines

kernel/sched/fair.c +8 -9

 		need_serialize = sd->flags & SD_SERIALIZE;
 
 		if (need_serialize) {

@@ -4696,16 +4699,15 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
 				goto out;
 		}
 
-		if (time_after_eq(jiffies, sd->last_balance + interval)) {
-			if (load_balance(cpu, rq, sd, idle, &balance)) {
-				/*
-				 * We've pulled tasks over so either we're no
-				 * longer idle.
-				 */
-				idle = CPU_NOT_IDLE;
-			}
-			sd->last_balance = jiffies;
+		if (load_balance(cpu, rq, sd, idle, &balance)) {
+			/*
+			 * We've pulled tasks over so either we're no
+			 * longer idle.
+			 */
+			idle = CPU_NOT_IDLE;
 		}
+		sd->last_balance = jiffies;
+
 		if (need_serialize)
 			spin_unlock(&balancing);
 out: