Skip to contentSkip to conversation

sched/fair: Improve nohz fields for large systems

PATCH Applied Shrikanth Hegde 3 participants 8 messages Jan 15 Scheduler

lore
Share & tools
lore
#0 lore raw

v5 cover sched/fair: Improve nohz fields for large systems

Running on large systems nohz.nr_cpus cacheline was seen as contended. There is atomic inc/dec and read happening on many
CPUs at a time and it is possible for this line to bounce often.

1st and 2nd patch are minor ones. Looks like correct things to do. Not very important ones.

3rd patch: Main patch which is to get rid of nr_cpus.Instead, use the cpumask which is always updated alongside with it. Functionally it should serve the same purpose. Rest of the fields aren't updated that often. So this line shouldn't bounce that often.

Contention issue with nohz.idle_cpus_mask still remains. Mostly it is in separate cacheline than nohz. There are ongoing efforts to mitigate it. It is not addressed by this series.

v4 -> v5:
- Collected tags (Thanks to K Prateek Nayak, Valentin Schneider)
- Added comment for patch 1, making note of a narrow window where kick_ilb will be called un-necessarily. (Vincent Guittot)

v3 -> v4:
- Added to changelog on one less cacheline being dirtied on idle entry/exit (Valentin Schneider)

v2 -> v3:
- Converted out to return when there are no CPU is in tickless mode since find_ilb_cpu returns anyway (K Prateek Nayak)

v1 -> v2:
- Dropped patch to check has_blocked based on time.
- Detailed changelog for removing nr_cpus (Thanks to Ingo Molnar)

v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/

Shrikanth Hegde (3):
sched/fair: Move checking for nohz cpus after time check sched/fair: Change likelyhood of nohz.nr_cpus sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

 kernel/sched/fair.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)
Signature · 1 line
2.51.0
#1 lore raw

PATCH v5 1/3 sched/fair: Move checking for nohz cpus after time check

Current code does.
- Read nohz.nr_cpus
- Check if the time has passed to do NOHZ idle balance

Instead do this.
- Check if the time has passed to do NOHZ idle balance
- Read nohz.nr_cpus

This will skip the read most of the time in normal system usage. i.e when there are nohz.nr_cpus (system is not 100% busy).

Note that when there are no idle CPUs(100% busy), even if the flag gets set to NOHZ_STATS_KICK | NOHZ_NEXT_KICK, find_new_ilb will fail and there will be no NOHZ idle balance. In such cases there will be a very narrow window where, kick_ilb will be called un-necessarily. However current functionality is still retained.

Note: This patch doesn't solve any cacheline overheads. No improvement in performance apart from saving a few cycles of reading nohz.nr_cpus

---
 kernel/sched/fair.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
kernel/sched/fair.c +16 -7
index c921cdc6c3ed..a4910658c5d6 100644
@@ -12445,20 +12445,29 @@ static void nohz_balancer_kick(struct rq *rq)
 	 */
 	nohz_balance_exit_idle(rq);
 
-	/*
-	 * None are in tickless mode and hence no need for NOHZ idle load
-	 * balancing:
-	 */
-	if (likely(!atomic_read(&nohz.nr_cpus)))
-		return;
-
 	if (READ_ONCE(nohz.has_blocked_load) &&
 	    time_after(now, READ_ONCE(nohz.next_blocked)))
 		flags = NOHZ_STATS_KICK;
 
+	/*
+	 * Most of the time system is not 100% busy. i.e nohz.nr_cpus > 0
+	 * Skip the read if time is not due.
+	 *
+	 * If none are in tickless mode, there maybe a narrow window
+	 * (28 jiffies, HZ=1000) where flags maybe set and kick_ilb called.
+	 * But idle load balancing is not done as find_new_ilb fails.
+	 * That's very rare. So read nohz.nr_cpus only if time is due.
+	 */
 	if (time_before(now, nohz.next_balance))
 		goto out;
 
+	/*
+	 * None are in tickless mode and hence no need for NOHZ idle load
+	 * balancing:
+	 */
+	if (likely(!atomic_read(&nohz.nr_cpus)))
+		return;
+
 	if (rq->nr_running >= 2) {
 		flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK;
 		goto out;
Signature · 1 line
2.51.0
#2 lore raw

[tip: sched/core] sched/fair: Move checking for nohz cpus after time check

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     6b67c8a72e56041f91f70ae5995bdb769761869a
Gitweb:        https://git.kernel.org/tip/6b67c8a72e56041f91f70ae5995bdb769761869a
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Thu, 15 Jan 2026 13:05:22 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 15 Jan 2026 22:41:26 +01:00

sched/fair: Move checking for nohz cpus after time check

Current code does.
- Read nohz.nr_cpus
- Check if the time has passed to do NOHZ idle balance

Instead do this.
- Check if the time has passed to do NOHZ idle balance
- Read nohz.nr_cpus

This will skip the read most of the time in normal system usage. i.e when there are nohz.nr_cpus (system is not 100% busy).

Note that when there are no idle CPUs(100% busy), even if the flag gets set to NOHZ_STATS_KICK | NOHZ_NEXT_KICK, find_new_ilb will fail and there will be no NOHZ idle balance. In such cases there will be a very narrow window where, kick_ilb will be called un-necessarily. However current functionality is still retained.

Note: This patch doesn't solve any cacheline overheads. No improvement in performance apart from saving a few cycles of reading nohz.nr_cpus

---
 kernel/sched/fair.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
kernel/sched/fair.c +16 -7
index af120e8..9afe0c6 100644
@@ -12441,20 +12441,29 @@ static void nohz_balancer_kick(struct rq *rq)
 	 */
 	nohz_balance_exit_idle(rq);
 
-	/*
-	 * None are in tickless mode and hence no need for NOHZ idle load
-	 * balancing:
-	 */
-	if (likely(!atomic_read(&nohz.nr_cpus)))
-		return;
-
 	if (READ_ONCE(nohz.has_blocked_load) &&
 	    time_after(now, READ_ONCE(nohz.next_blocked)))
 		flags = NOHZ_STATS_KICK;
 
+	/*
+	 * Most of the time system is not 100% busy. i.e nohz.nr_cpus > 0
+	 * Skip the read if time is not due.
+	 *
+	 * If none are in tickless mode, there maybe a narrow window
+	 * (28 jiffies, HZ=1000) where flags maybe set and kick_ilb called.
+	 * But idle load balancing is not done as find_new_ilb fails.
+	 * That's very rare. So read nohz.nr_cpus only if time is due.
+	 */
 	if (time_before(now, nohz.next_balance))
 		goto out;
 
+	/*
+	 * None are in tickless mode and hence no need for NOHZ idle load
+	 * balancing:
+	 */
+	if (likely(!atomic_read(&nohz.nr_cpus)))
+		return;
+
 	if (rq->nr_running >= 2) {
 		flags = NOHZ_STATS_KICK | NOHZ_BALANCE_KICK;
 		goto out;
#3 lore raw

PATCH v5 2/3 sched/fair: Change likelyhood of nohz.nr_cpus

In reply to Shrikanth Hegde

These days most of the system have multi cores. The likelyhood of at least one or more CPUs in nohz (idle state) is higher.

Give accurate hint to the branch predictor.

---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
index a4910658c5d6..3d843d1396ec 100644
@@ -12463,9 +12463,9 @@ static void nohz_balancer_kick(struct rq *rq)
 
 	/*
 	 * None are in tickless mode and hence no need for NOHZ idle load
-	 * balancing:
+	 * balancing
 	 */
-	if (likely(!atomic_read(&nohz.nr_cpus)))
+	if (unlikely(!atomic_read(&nohz.nr_cpus)))
 		return;
 
 	if (rq->nr_running >= 2) {
Signature · 1 line
2.51.0
#4 lore raw

[tip: sched/core] sched/fair: Change likelyhood of nohz.nr_cpus

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     94e70734b4d034b9df795bd1ad3452ea96e742ca
Gitweb:        https://git.kernel.org/tip/94e70734b4d034b9df795bd1ad3452ea96e742ca
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Thu, 15 Jan 2026 13:05:23 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 15 Jan 2026 22:41:27 +01:00

sched/fair: Change likelyhood of nohz.nr_cpus

These days most of the system have multi cores. The likelyhood of at least one or more CPUs in nohz (idle state) is higher.

Give accurate hint to the branch predictor.

---
 kernel/sched/fair.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/fair.c +2 -2
index 9afe0c6..4ae06ce 100644
@@ -12459,9 +12459,9 @@ static void nohz_balancer_kick(struct rq *rq)
 
 	/*
 	 * None are in tickless mode and hence no need for NOHZ idle load
-	 * balancing:
+	 * balancing
 	 */
-	if (likely(!atomic_read(&nohz.nr_cpus)))
+	if (unlikely(!atomic_read(&nohz.nr_cpus)))
 		return;
 
 	if (rq->nr_running >= 2) {
#5 lore raw

PATCH v5 3/3 sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

In reply to Shrikanth Hegde

nohz.nr_cpus was observed as contended cacheline when running enterprise workload on large systems.

Fundamental scalability challenge with nohz.idle_cpus_mask and nohz.nr_cpus is the following:

(1) nohz_balancer_kick() observes (reads) nohz.nr_cpus
(or nohz.idle_cpu_mask) and nohz.has_blocked to see whether there's any nohz balancing work to do, in every scheduler tick.

(2) nohz_balance_enter_idle() and nohz_balance_exit_idle() (through nohz_balancer_kick() via sched_tick()) modify (write) nohz.nr_cpus (and/or nohz.idle_cpu_mask) and nohz.has_blocked.

The characteristic frequencies are the following:

(1) nohz_balancer_kick() happens at scheduler (busy)tick frequency on CPU(which has not gone idle). This is a relatively constant frequency in the ~1 kHz range or lower.

(2) happens at idle enter/exit frequency on every CPU that goes to idle. This is workload dependent, but can easily be hundreds of kHz for IO-bound loads and high CPU counts. Ie. can be orders of magnitude higher than (1), in which case a cachemiss at every invocation of (1) is almost inevitable. idle exit will trigger (1) on the CPU which is coming out of idle.

There's two types of costs from these functions:

(A) scheduler tick cost via (1): this happens on busy CPUs too, and is thus a primary scalability cost. But the rate here is constant and typically much lower than (B), hence the absolute benefit to workload scalability will be lower as well.

(B) idle cost via (2): going-to-idle and coming-from-idle costs are secondary concerns, because they impact power efficiency more than they impact scalability. But in terms of absolute cost this scales up with nr_cpus as well, and a much faster rate, and thus may also approach and negatively impact system limits like memory bus/fabric bandwidth.

Note that nohz.idle_cpus_mask and nohz.nr_cpus may appear to reside in the same cacheline, however under CONFIG_CPUMASK_OFFSTACK=y the backing storage for nohz.idle_cpus_mask will be elsewhere. With CPUMASK_OFFSTACK=n, the nohz.idle_cpus_mask and rest of nohz fields are in different cachelines under typical NR_CPUS=512/2048. This implies two separate cachelines being dirtied upon idle entry / exit.

nohz.nr_cpus can be derived from the mask itself. Its usage doesn't warrant a functionally correct value. This means one less cacheline being dirtied in idle entry/exit path which helps to save some bus bandwidth w.r.t to those nohz functions(approx 50%). This in turn helps to improve enterprise workload throughput.

On system with 480 CPUs, running "hackbench 40 process 10000 loops" (Avg of 3 runs)
baseline:

     0.81%  hackbench          [k] nohz_balance_exit_idle
     0.21%  hackbench          [k] nohz_balancer_kick
     0.09%  swapper            [k] nohz_run_idle_balance

With patch:

     0.35%  hackbench          [k] nohz_balance_exit_idle
     0.09%  hackbench          [k] nohz_balancer_kick
     0.07%  swapper            [k] nohz_run_idle_balance
[Ingo Molnar: scalability analysis changlog]
---
 kernel/sched/fair.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/sched/fair.c +1 -4
index 3d843d1396ec..46ed16466be4 100644
@@ -7141,7 +7141,6 @@ static DEFINE_PER_CPU(cpumask_var_t, should_we_balance_tmpmask);
 
 static struct {
 	cpumask_var_t idle_cpus_mask;
-	atomic_t nr_cpus;
 	int has_blocked_load;		/* Idle CPUS has blocked load */
 	int needs_update;		/* Newly idle CPUs need their next_balance collated */
 	unsigned long next_balance;     /* in jiffy units */
@@ -12465,7 +12464,7 @@ static void nohz_balancer_kick(struct rq *rq)
 	 * None are in tickless mode and hence no need for NOHZ idle load
 	 * balancing
 	 */
-	if (unlikely(!atomic_read(&nohz.nr_cpus)))
+	if (unlikely(cpumask_empty(nohz.idle_cpus_mask)))
 		return;
 
 	if (rq->nr_running >= 2) {
@@ -12578,7 +12577,6 @@ void nohz_balance_exit_idle(struct rq *rq)
 
 	rq->nohz_tick_stopped = 0;
 	cpumask_clear_cpu(rq->cpu, nohz.idle_cpus_mask);
-	atomic_dec(&nohz.nr_cpus);
 
 	set_cpu_sd_state_busy(rq->cpu);
 }
@@ -12636,7 +12634,6 @@ void nohz_balance_enter_idle(int cpu)
 	rq->nohz_tick_stopped = 1;
 
 	cpumask_set_cpu(cpu, nohz.idle_cpus_mask);
-	atomic_inc(&nohz.nr_cpus);
 
 	/*
 	 * Ensures that if nohz_idle_balance() fails to observe our
Signature · 1 line
2.51.0
#6 lore raw

[tip: sched/core] sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     5d86d542f68fda7ef6d543ac631b741db734101a
Gitweb:        https://git.kernel.org/tip/5d86d542f68fda7ef6d543ac631b741db734101a
Author:        Shrikanth Hegde <[email protected]>
AuthorDate:    Thu, 15 Jan 2026 13:05:24 +05:30
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 15 Jan 2026 22:41:27 +01:00

sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

nohz.nr_cpus was observed as contended cacheline when running enterprise workload on large systems.

Fundamental scalability challenge with nohz.idle_cpus_mask and nohz.nr_cpus is the following:

(1) nohz_balancer_kick() observes (reads) nohz.nr_cpus
(or nohz.idle_cpu_mask) and nohz.has_blocked to see whether there's any nohz balancing work to do, in every scheduler tick.

(2) nohz_balance_enter_idle() and nohz_balance_exit_idle() (through nohz_balancer_kick() via sched_tick()) modify (write) nohz.nr_cpus (and/or nohz.idle_cpu_mask) and nohz.has_blocked.

The characteristic frequencies are the following:

(1) nohz_balancer_kick() happens at scheduler (busy)tick frequency on CPU(which has not gone idle). This is a relatively constant frequency in the ~1 kHz range or lower.

(2) happens at idle enter/exit frequency on every CPU that goes to idle. This is workload dependent, but can easily be hundreds of kHz for IO-bound loads and high CPU counts. Ie. can be orders of magnitude higher than (1), in which case a cachemiss at every invocation of (1) is almost inevitable. idle exit will trigger (1) on the CPU which is coming out of idle.

There's two types of costs from these functions:

(A) scheduler tick cost via (1): this happens on busy CPUs too, and is thus a primary scalability cost. But the rate here is constant and typically much lower than (B), hence the absolute benefit to workload scalability will be lower as well.

(B) idle cost via (2): going-to-idle and coming-from-idle costs are secondary concerns, because they impact power efficiency more than they impact scalability. But in terms of absolute cost this scales up with nr_cpus as well, and a much faster rate, and thus may also approach and negatively impact system limits like memory bus/fabric bandwidth.

Note that nohz.idle_cpus_mask and nohz.nr_cpus may appear to reside in the same cacheline, however under CONFIG_CPUMASK_OFFSTACK=y the backing storage for nohz.idle_cpus_mask will be elsewhere. With CPUMASK_OFFSTACK=n, the nohz.idle_cpus_mask and rest of nohz fields are in different cachelines under typical NR_CPUS=512/2048. This implies two separate cachelines being dirtied upon idle entry / exit.

nohz.nr_cpus can be derived from the mask itself. Its usage doesn't warrant a functionally correct value. This means one less cacheline being dirtied in idle entry/exit path which helps to save some bus bandwidth w.r.t to those nohz functions(approx 50%). This in turn helps to improve enterprise workload throughput.

On system with 480 CPUs, running "hackbench 40 process 10000 loops" (Avg of 3 runs)
baseline:

     0.81%  hackbench          [k] nohz_balance_exit_idle
     0.21%  hackbench          [k] nohz_balancer_kick
     0.09%  swapper            [k] nohz_run_idle_balance

With patch:

     0.35%  hackbench          [k] nohz_balance_exit_idle
     0.09%  hackbench          [k] nohz_balancer_kick
     0.07%  swapper            [k] nohz_run_idle_balance

[Ingo Molnar: scalability analysis changlog]

---
 kernel/sched/fair.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
kernel/sched/fair.c +1 -4
index 4ae06ce..04993c7 100644
@@ -7138,7 +7138,6 @@ static DEFINE_PER_CPU(cpumask_var_t, should_we_balance_tmpmask);
 
 static struct {
 	cpumask_var_t idle_cpus_mask;
-	atomic_t nr_cpus;
 	int has_blocked_load;		/* Idle CPUS has blocked load */
 	int needs_update;		/* Newly idle CPUs need their next_balance collated */
 	unsigned long next_balance;     /* in jiffy units */
@@ -12461,7 +12460,7 @@ static void nohz_balancer_kick(struct rq *rq)
 	 * None are in tickless mode and hence no need for NOHZ idle load
 	 * balancing
 	 */
-	if (unlikely(!atomic_read(&nohz.nr_cpus)))
+	if (unlikely(cpumask_empty(nohz.idle_cpus_mask)))
 		return;
 
 	if (rq->nr_running >= 2) {
@@ -12574,7 +12573,6 @@ void nohz_balance_exit_idle(struct rq *rq)
 
 	rq->nohz_tick_stopped = 0;
 	cpumask_clear_cpu(rq->cpu, nohz.idle_cpus_mask);
-	atomic_dec(&nohz.nr_cpus);
 
 	set_cpu_sd_state_busy(rq->cpu);
 }
@@ -12632,7 +12630,6 @@ void nohz_balance_enter_idle(int cpu)
 	rq->nohz_tick_stopped = 1;
 
 	cpumask_set_cpu(cpu, nohz.idle_cpus_mask);
-	atomic_inc(&nohz.nr_cpus);
 
 	/*
 	 * Ensures that if nohz_idle_balance() fails to observe our
maintainer Reviewed-by on the v5 series
#7 lore raw
In reply to Shrikanth Hegde
Quoted 13 lines · Shrikanth Hegde

Running on large systems nohz.nr_cpus cacheline was seen as contended. There is atomic inc/dec and read happening on many
CPUs at a time and it is possible for this line to bounce often.

1st and 2nd patch are minor ones. Looks like correct things to do. Not very important ones.

3rd patch: Main patch which is to get rid of nr_cpus.Instead, use the cpumask which is always updated alongside with it. Functionally it should serve the same purpose. Rest of the fields aren't updated that often. So this line shouldn't bounce that often.

Quoted 8 lines

Contention issue with nohz.idle_cpus_mask still remains. Mostly it is in separate cacheline than nohz. There are ongoing efforts to mitigate it. It is not addressed by this series.

v4 -> v5:
- Collected tags (Thanks to K Prateek Nayak, Valentin Schneider)
- Added comment for patch 1, making note of a narrow window where kick_ilb will be called un-necessarily. (Vincent Guittot)

Looks good to me

Quoted 29 lines

v3 -> v4:
- Added to changelog on one less cacheline being dirtied on idle entry/exit (Valentin Schneider)

v2 -> v3:
- Converted out to return when there are no CPU is in tickless mode since find_ilb_cpu returns anyway (K Prateek Nayak)

v1 -> v2:
- Dropped patch to check has_blocked based on time.
- Detailed changelog for removing nr_cpus (Thanks to Ingo Molnar)

v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/

Shrikanth Hegde (3):
sched/fair: Move checking for nohz cpus after time check sched/fair: Change likelyhood of nohz.nr_cpus sched/fair: Remove nohz.nr_cpus and use weight of cpumask instead

 kernel/sched/fair.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

--
2.51.0