Skip to contentSkip to conversation

sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

PATCH In review Adam Li 5 participants 21 messages Aug 19, 2025 – Sep 11, 2025 Scheduler

lore
Share & tools
lore

Adam Li

author
#0 lore raw

PATCH 1/1 sched/nohz: Fix NOHZ imbalance by adding options for ILB CPU

A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
2) and in nohz.idle_cpus_mask
3) and idle
4) and not current CPU

If most CPUs are in nohz_full CPU list there is few housekeeping CPU left. In the worst case if all CPUs are in nohz_full only the boot CPU is used for housekeeping. And the housekeeping CPU is usually busier so it will be unlikely added to nohz.idle_cpus_mask.

Therefore if there is few housekeeping CPUs, find_new_ilb() may likely failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may stay idle while other CPUs are busy.

This patch adds fallback options when looking for ILB CPU if there is no CPU meeting above requirements. Then it searches in bellow order:
1) Try looking for the first idle housekeeping CPU
2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
3) Select the first housekeeping CPU even if it is busy.

With this patch the NOHZ idle balancing happens more frequently.

---
 kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++---
 1 file changed, 29 insertions(+), 3 deletions(-)
kernel/sched/fair.c +29 -3
index b173a059315c..12bcc3f81f9b 100644
@@ -12194,19 +12194,45 @@ static inline int on_null_domain(struct rq *rq)
 static inline int find_new_ilb(void)
 {
 	const struct cpumask *hk_mask;
-	int ilb_cpu;
+	struct cpumask ilb_mask;
+	int ilb_cpu, this_cpu = smp_processor_id();
 
 	hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
 
-	for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) {
+	/*
+	 * Look for an idle cpu who is both NOHZ_idle and housekeeping.
+	 * If no such cpu, look for an idle housekeeping cpu.
+	 */
+	if (!cpumask_and(&ilb_mask, nohz.idle_cpus_mask, hk_mask))
+		cpumask_copy(&ilb_mask, hk_mask);
 
-		if (ilb_cpu == smp_processor_id())
+	for_each_cpu(ilb_cpu, &ilb_mask) {
+		if (ilb_cpu == this_cpu)
 			continue;
 
 		if (idle_cpu(ilb_cpu))
 			return ilb_cpu;
 	}
 
+	/*
+	 * If CPU has no SMT, look for an idle NOHZ_idle cpu.
+	 * Run NOHZ ILB may cause jitter on SMT sibling CPU.
+	 */
+	if (!sched_smt_active()) {
+		for_each_cpu(ilb_cpu, nohz.idle_cpus_mask) {
+			if (ilb_cpu == this_cpu)
+				continue;
+
+			if (idle_cpu(ilb_cpu))
+				return ilb_cpu;
+		}
+	}
+
+	/* Select the first housekeeping cpu anyway. */
+	ilb_cpu = cpumask_first(hk_mask);
+	if (ilb_cpu < nr_cpu_ids)
+		return ilb_cpu;
+
 	return -1;
 }
 
Signature · 1 line
2.34.1
#1 lore raw
Quoted 15 lines · Adam Li

A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
2) and in nohz.idle_cpus_mask
3) and idle
4) and not current CPU

If most CPUs are in nohz_full CPU list there is few housekeeping CPU left. In the worst case if all CPUs are in nohz_full only the boot CPU is used for housekeeping. And the housekeeping CPU is usually busier so it will be unlikely added to nohz.idle_cpus_mask.

Therefore if there is few housekeeping CPUs, find_new_ilb() may likely failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may stay idle while other CPUs are busy.

Quoted 8 lines

This patch adds fallback options when looking for ILB CPU if there is no CPU meeting above requirements. Then it searches in bellow order:
1) Try looking for the first idle housekeeping CPU
2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
3) Select the first housekeeping CPU even if it is busy.

With this patch the NOHZ idle balancing happens more frequently.

I'm not understanding why, in the scenarios outlined above, more NOHZ idle balancing is a good thing.

Considering only housekeeping CPUs, they're all covered by wakeup, periodic and idle balancing (on top of NOHZ idle balancing when relevant). So if find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs are either always busy or never stopping the tick when going idle, IOW they always have some work to do within a jiffy boundary.

Am I missing something?

author
#2 lore raw

Hi Valentin,

Thanks for your comments.

Quoted 25 lines · Valentin Schneider
Quoted 15 lines · Adam Li

A qualified CPU to run NOHZ idle load balancing (ILB) has to be:
1) housekeeping CPU in housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
2) and in nohz.idle_cpus_mask
3) and idle
4) and not current CPU

If most CPUs are in nohz_full CPU list there is few housekeeping CPU left. In the worst case if all CPUs are in nohz_full only the boot CPU is used for housekeeping. And the housekeeping CPU is usually busier so it will be unlikely added to nohz.idle_cpus_mask.

Therefore if there is few housekeeping CPUs, find_new_ilb() may likely failed to find any CPU to do NOHZ idle load balancing. Some NOHZ CPUs may stay idle while other CPUs are busy.

Quoted 8 lines

This patch adds fallback options when looking for ILB CPU if there is no CPU meeting above requirements. Then it searches in bellow order:
1) Try looking for the first idle housekeeping CPU
2) Try looking for the first idle CPU in nohz.idle_cpus_mask if no SMT.
3) Select the first housekeeping CPU even if it is busy.

With this patch the NOHZ idle balancing happens more frequently.

Quoted 10 lines

I'm not understanding why, in the scenarios outlined above, more NOHZ idle balancing is a good thing.

Considering only housekeeping CPUs, they're all covered by wakeup, periodic and idle balancing (on top of NOHZ idle balancing when relevant). So if find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs are either always busy or never stopping the tick when going idle, IOW they always have some work to do within a jiffy boundary.

Quoted 1 line

Am I missing something?

I agree with your description about the housekeeping CPUs. In the worst case, the system only has one housekeeping CPU and this housekeeping CPU is so busy that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected to do NOHZ idle load balancing.

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask). Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL is set.

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask', this bug is fixed by another patch:
https://lore.kernel.org/all/[email protected]/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to do NOHZ idle load balancing because conditions in find_new_ilb() is too strict. This patch tries to solve this issue.

Hope this information helps.

Thanks,
-adam

#3 lore raw
Quoted 22 lines · Adam Li
Quoted 11 lines · Valentin Schneider

I'm not understanding why, in the scenarios outlined above, more NOHZ idle balancing is a good thing.

Considering only housekeeping CPUs, they're all covered by wakeup, periodic and idle balancing (on top of NOHZ idle balancing when relevant). So if find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs are either always busy or never stopping the tick when going idle, IOW they always have some work to do within a jiffy boundary.

Quoted 1 line

Am I missing something?

I agree with your description about the housekeeping CPUs. In the worst case, the system only has one housekeeping CPU and this housekeeping CPU is so busy that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected to do NOHZ idle load balancing.

Quoted 7 lines

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask). Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL is set.

I assume you mean all but one CPU is in 'nohz_full=' since you need at least one housekeeping CPU. But in that case this becomes a slightly different problem, since no CPU in 'nohz_full' will be in

housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)

Quoted 11 lines

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask', this bug is fixed by another patch:
https://lore.kernel.org/all/[email protected]/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to do NOHZ idle load balancing because conditions in find_new_ilb() is too strict. This patch tries to solve this issue.

Hope this information helps.

I hadn't seen that patch; that cclist is quite small, you'll want to add the scheduler people to our next submission.

So IIUC:
- Pretty much all your CPUs are NOHZ_FULL
- When they go idle they remain so for a while despite work being available

My first question would be: is NOHZ_FULL really right for your workload? It's mainly designed to be used with always-running userspace tasks, generally affined to a CPU by the system administrator.
Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

Could you share your system topology and your actual nohz_full cmdline?

Quoted 4 lines

Thanks,
-adam

author
#4 lore raw
Quoted 31 lines · Valentin Schneider
Quoted 22 lines · Adam Li
Quoted 11 lines · Valentin Schneider

I'm not understanding why, in the scenarios outlined above, more NOHZ idle balancing is a good thing.

Considering only housekeeping CPUs, they're all covered by wakeup, periodic and idle balancing (on top of NOHZ idle balancing when relevant). So if find_new_ilb() never finds a NOHZ-idle CPU, then that means your HK CPUs are either always busy or never stopping the tick when going idle, IOW they always have some work to do within a jiffy boundary.

Quoted 1 line

Am I missing something?

I agree with your description about the housekeeping CPUs. In the worst case, the system only has one housekeeping CPU and this housekeeping CPU is so busy that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected to do NOHZ idle load balancing.

Quoted 7 lines

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask). Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL is set.

Quoted 6 lines

I assume you mean all but one CPU is in 'nohz_full=' since you need at least one housekeeping CPU. But in that case this becomes a slightly different problem, since no CPU in 'nohz_full' will be in

housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)

I ran llama workload on a system with 192 CPUs. I set "nohz_full=0-191" so all CPUs are in 'nohz_full' list. In this case, kernel uses the boot CPU for housekeeping:

Kernel message: "Housekeeping: must include one present CPU, using boot CPU:0"

find_new_ilb() looks for qualified CPU from housekeeping CPUs. The searching is likely to fail if there is only one housekeeping CPU.

Quoted 9 lines
Quoted 9 lines

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask', this bug is fixed by another patch:
https://lore.kernel.org/all/[email protected]/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to do NOHZ idle load balancing because conditions in find_new_ilb() is too strict. This patch tries to solve this issue.

Quoted 6 lines
Quoted 2 lines

Hope this information helps.

I hadn't seen that patch; that cclist is quite small, you'll want to add the scheduler people to our next submission.

Sure. The first patch involves both 'tick' and 'scheduler' subsystem. I can resend the first patch to broader reviewers if you don't mind.

Quoted 4 lines

So IIUC:
- Pretty much all your CPUs are NOHZ_FULL
- When they go idle they remain so for a while despite work being available

Exactly.

Quoted 8 lines

My first question would be: is NOHZ_FULL really right for your workload? It's mainly designed to be used with always-running userspace tasks, generally affined to a CPU by the system administrator.
Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

I ran the llama workload to do 'Large Language Model' reference. The workload creates 'always-running userspace' threads doing math computing. There is *few* sleep, wakeup and context switch. IIUC NOHZ_IDLE cannot help always-running task?

'nohz_full' option is supposed to benefit performance by reducing kernel noise I think. Could you please give more detail on 'NOHZ_FULL context switch overhead'?

Quoted 6 lines

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

I think the newidle balance and wakeup balance do not help in this case because the workload has few sleep and wakeup.

Quoted 2 lines

Could you share your system topology and your actual nohz_full cmdline?

The system has 192 CPUs. I set "nohz_full=0-191".

Thanks,
-adam

#5 lore raw
Quoted 26 lines · Adam Li
Quoted 18 lines · Valentin Schneider
Quoted 9 lines · Adam Li

I agree with your description about the housekeeping CPUs. In the worst case, the system only has one housekeeping CPU and this housekeeping CPU is so busy that:
1) This housekeeping CPU is unlikely idle;
2) and this housekeeping CPU is unlikely in 'nohz.idle_cpus_mask' because tick is not stopped.
Therefore find_new_ilb() may very likely return -1. *No* CPU can be selected to do NOHZ idle load balancing.

Quoted 7 lines

This patch tries to fix the imbalance of NOHZ idle CPUs (CPUs in nohz.idle_cpus_mask). Here is more background:

When running llama on arm64 server, some CPUs *keep* idle while others are 100% busy. All CPUs are in 'nohz_full=' cpu list, and CONFIG_NO_HZ_FULL is set.

Quoted 6 lines

I assume you mean all but one CPU is in 'nohz_full=' since you need at least one housekeeping CPU. But in that case this becomes a slightly different problem, since no CPU in 'nohz_full' will be in

housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)

Quoted 8 lines

I ran llama workload on a system with 192 CPUs. I set "nohz_full=0-191" so all CPUs are in 'nohz_full' list. In this case, kernel uses the boot CPU for housekeeping:

Kernel message: "Housekeeping: must include one present CPU, using boot CPU:0"

find_new_ilb() looks for qualified CPU from housekeeping CPUs. The searching is likely to fail if there is only one housekeeping CPU.

Right

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

The problem is caused by two issues:
1) Some idle CPUs cannot be added to 'nohz.idle_cpus_mask', this bug is fixed by another patch:
https://lore.kernel.org/all/[email protected]/

2) Even if the idle CPUs are in 'nohz.idle_cpus_mask', *no* CPU can be selected to do NOHZ idle load balancing because conditions in find_new_ilb() is too strict. This patch tries to solve this issue.

Hope this information helps.

Quoted 7 lines
Quoted 3 lines

I hadn't seen that patch; that cclist is quite small, you'll want to add the scheduler people to our next submission.

Sure. The first patch involves both 'tick' and 'scheduler' subsystem. I can resend the first patch to broader reviewers if you don't mind.

I'd say resend the whole series with the right folks cc'd.

Quoted 16 lines
Quoted 4 lines

So IIUC:
- Pretty much all your CPUs are NOHZ_FULL
- When they go idle they remain so for a while despite work being available

Exactly.

Quoted 8 lines

My first question would be: is NOHZ_FULL really right for your workload? It's mainly designed to be used with always-running userspace tasks, generally affined to a CPU by the system administrator.
Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

Quoted 5 lines

I ran the llama workload to do 'Large Language Model' reference. The workload creates 'always-running userspace' threads doing math computing. There is *few* sleep, wakeup and context switch. IIUC NOHZ_IDLE cannot help always-running task?

Right, NOHZ_IDLE is really about power savings while a CPU is idle (and IIRC it helps some virtualization cases).

Quoted 4 lines

'nohz_full' option is supposed to benefit performance by reducing kernel noise I think. Could you please give more detail on 'NOHZ_FULL context switch overhead'?

The doc briefly touches on that:

https://docs.kernel.org/timers/no_hz.html#omit-scheduling-clock-ticks-for-cpus-with-only-one-runnable-task

The longer story is have a look at kernel/context_tracking.c; every transition into and out of the kernel to and from user or idle requires additional atomic operations and synchronization.

It would be worth for you to quantify how much these processes sleep/context switch, it could be that keep the tick enabled incurs a lower throughput penalty than the NO_HZ_FULL overheads.

Quoted 10 lines
Quoted 6 lines

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

I think the newidle balance and wakeup balance do not help in this case because the workload has few sleep and wakeup.

Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you could manually affine the threads of the workload. Depending on how much control you have over how many threads it spawn, you could either pin on thread per CPU, or just spawn the workload into a cpuset covering the NO_HZ_FULL CPUs.

Having the scheduler do the balancing is bit of a precarious situation. Your single housekeeping CPU is pretty much going to be always running things, does it make sense to have it run the NOHZ idle balance when there are available idle NOHZ_FULL CPUs? And in the same sense, does it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on other NOHZ_FULL CPUs? Admins that manually affine their threads will probably say no.

9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs") also mentions SMT being an issue.

Quoted 7 lines
Quoted 2 lines

Could you share your system topology and your actual nohz_full cmdline?

The system has 192 CPUs. I set "nohz_full=0-191".

Thanks,
-adam

author
#6 lore raw
Quoted 3 lines · Valentin Schneider

I'd say resend the whole series with the right folks cc'd.

OK. I resent the patch series.
Please refer to: https://lore.kernel.org/all/[email protected]/

Quoted 9 lines
Quoted 4 lines

'nohz_full' option is supposed to benefit performance by reducing kernel noise I think. Could you please give more detail on 'NOHZ_FULL context switch overhead'?

The doc briefly touches on that:

https://docs.kernel.org/timers/no_hz.html#omit-scheduling-clock-ticks-for-cpus-with-only-one-runnable-task

Quoted 8 lines

The longer story is have a look at kernel/context_tracking.c; every transition into and out of the kernel to and from user or idle requires additional atomic operations and synchronization.

It would be worth for you to quantify how much these processes sleep/context switch, it could be that keep the tick enabled incurs a lower throughput penalty than the NO_HZ_FULL overheads.

Thanks for the information.

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

I think the newidle balance and wakeup balance do not help in this case because the workload has few sleep and wakeup.

Quoted 6 lines

Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you could manually affine the threads of the workload. Depending on how much control you have over how many threads it spawn, you could either pin on thread per CPU, or just spawn the workload into a cpuset covering the NO_HZ_FULL CPUs.

Yes, binding the threads to CPU can work around the performance issue caused by load imbalance. Should we document that 'nohz_full' may cause the scheduler load balancing not working well and CPU affinity is preferred?

Quoted 8 lines

Having the scheduler do the balancing is bit of a precarious situation. Your single housekeeping CPU is pretty much going to be always running things, does it make sense to have it run the NOHZ idle balance when there are available idle NOHZ_FULL CPUs? And in the same sense, does it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on other NOHZ_FULL CPUs? Admins that manually affine their threads will probably say no.

I think when the NOHZ_FULL CPU is added to nohz.idle_cpus_mask and its tick is stopped, the CPU is 'very' idle. We can safely assign some work to it.

Quoted 3 lines

9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs") also mentions SMT being an issue.

From the commit message of 9b019acb72e4:
"The problem was observed with increased jitter on an application running on CPU0, caused by NOHZ idle load balancing being run on CPU1 (an SMT sibling)."

Can we say if *no* SMT, it is safe to run NOHZ idle load balancing on CPU in nohz.idle_cpus_mask? My patch checks '!sched_smt_active()' when searching from nohz.idle_cpus_mask.

Thanks,
-adam

#7 lore raw
Quoted 8 lines · Adam Li
Quoted 6 lines · Valentin Schneider

Right. So other than the NO_HZ_FULL vs NO_HZ_IDLE considerations above, you could manually affine the threads of the workload. Depending on how much control you have over how many threads it spawn, you could either pin on thread per CPU, or just spawn the workload into a cpuset covering the NO_HZ_FULL CPUs.

Quoted 4 lines

Yes, binding the threads to CPU can work around the performance issue caused by load imbalance. Should we document that 'nohz_full' may cause the scheduler load balancing not working well and CPU affinity is preferred?

Yeah I guess we could highlight that.

I think it's kind of a gray area; technically we could change load balancing to make NO_HZ_FULL CPUs better at pulling tasks, but that only works up to the point where, if you have N NO_HZ_FULL CPUs, you have pulled N tasks. So there is an underlying assumption that the workload threading matches your NO_HZ_FULL topology; and if that's the case, you might as well affine the tasks by hand and avoid any surprises.

Put in another way: yes we can probably make load balancing better for NO_HZ_FULL CPUs, but that only really works if we have one task to pull per NO_HZ_FULL CPU, in which case manual affinity binding works just as well, and I prefer that approach since it means we don't have to add a NO_HZ_FULL load balancing logic which may end up interfering with NO_HZ_FULL itself. At least, that is my opinion.

Quoted 16 lines
Quoted 8 lines

Having the scheduler do the balancing is bit of a precarious situation. Your single housekeeping CPU is pretty much going to be always running things, does it make sense to have it run the NOHZ idle balance when there are available idle NOHZ_FULL CPUs? And in the same sense, does it make sense to disturb an idle NOHZ_FULL CPU to get it to spread load on other NOHZ_FULL CPUs? Admins that manually affine their threads will probably say no.

I think when the NOHZ_FULL CPU is added to nohz.idle_cpus_mask and its tick is stopped, the CPU is 'very' idle. We can safely assign some work to it.

Quoted 3 lines

9b019acb72e4 ("sched/nohz: Run NOHZ idle load balancer on HK_FLAG_MISC CPUs") also mentions SMT being an issue.

Quoted 9 lines

From the commit message of 9b019acb72e4:
"The problem was observed with increased jitter on an application running on CPU0, caused by NOHZ idle load balancing being run on CPU1 (an SMT sibling)."

Can we say if *no* SMT, it is safe to run NOHZ idle load balancing on CPU in nohz.idle_cpus_mask? My patch checks '!sched_smt_active()' when searching from nohz.idle_cpus_mask.

I suppose we could still make this work for SMT with e.g. is_core_idle(), but see my point above.

Quoted 2 lines

Thanks,
-adam

#8 lore raw
Quoted 6 lines · Valentin Schneider
Quoted 4 lines

Yes, binding the threads to CPU can work around the performance issue caused by load imbalance. Should we document that 'nohz_full' may cause the scheduler load balancing not working well and CPU affinity is preferred?

Yeah I guess we could highlight that.

We need to make sure that the idle cpus are used when available and needed. Otherwise the scheduler is buggy.

Such a load balancing action means that there is a cpu that is running multiple processes. Therefore the timer interrrupt and the scheduler processing is active on at least one cpu. We can therefore do something about the situation.

The scheduler needs to move one of the processes onto the idle cpu.

#9 lore raw
Quoted 12 lines · Christoph Lameter (Ampere)
Quoted 6 lines · Valentin Schneider
Quoted 4 lines

Yes, binding the threads to CPU can work around the performance issue caused by load imbalance. Should we document that 'nohz_full' may cause the scheduler load balancing not working well and CPU affinity is preferred?

Yeah I guess we could highlight that.

We need to make sure that the idle cpus are used when available and needed. Otherwise the scheduler is buggy.

Quoted 6 lines

Such a load balancing action means that there is a cpu that is running multiple processes. Therefore the timer interrrupt and the scheduler processing is active on at least one cpu. We can therefore do something about the situation.

The scheduler needs to move one of the processes onto the idle cpu.

AFAICT we have (at least) two options:
1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)

This is somewhat against idle load balancing rules (only spend CPU time on that if there is no "genuine" work to run), but I guess from a CPU isolation PoV this can be tallied as just another housekeeping activity

2) Trigger NOHZ balancing on an idle NOHZ_FULL CPU

That doesn't steal useful CPU time, but that also potentially causes interference, albeit only if racing with the NOHZ_FULL workload spawning (which shouldn't be the steady state).

The more I think about it the more I'm leaning towards 1), but I'd like other folks' opinion.

maintainer
#10 lore raw
Quoted 19 lines · Valentin Schneider
Quoted 17 lines · Christoph Lameter (Ampere)
Quoted 6 lines · Valentin Schneider
Quoted 4 lines

Yes, binding the threads to CPU can work around the performance issue caused by load imbalance. Should we document that 'nohz_full' may cause the scheduler load balancing not working well and CPU affinity is preferred?

Yeah I guess we could highlight that.

We need to make sure that the idle cpus are used when available and needed. Otherwise the scheduler is buggy.

Such a load balancing action means that there is a cpu that is running multiple processes. Therefore the timer interrrupt and the scheduler processing is active on at least one cpu. We can therefore do something about the situation.

Quoted 8 lines
Quoted 1 line

The scheduler needs to move one of the processes onto the idle cpu.

AFAICT we have (at least) two options:
1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)

This is somewhat against idle load balancing rules (only spend CPU time on that if there is no "genuine" work to run), but I guess from a CPU isolation PoV this can be tallied as just another housekeeping activity

In this case, this should only be done for full nohz case and not for other cases because the ILB overhead is not negligible on a busy cpu and I don't see anything that enable 1) only for full no hz

Quoted 2 lines

2) Trigger NOHZ balancing on an idle NOHZ_FULL CPU

this patch also does 2) for no smt case

I wonder why this happens only for no smt case ? If the sibling is
used by another thread with full nohz, it already interferes with this one

But we might want to do is_core_idle() instead

Quoted 8 lines

That doesn't steal useful CPU time, but that also potentially causes interference, albeit only if racing with the NOHZ_FULL workload spawning (which shouldn't be the steady state).

The more I think about it the more I'm leaning towards 1), but I'd like other folks' opinion.

#11 lore raw
Quoted 2 lines · Vincent Guittot
Quoted 2 lines

AFAICT we have (at least) two options:
1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)

Isnt there a third option?

3) Trigger load balancing if on a NOHZ_FULL cpu and multiple processes are running on it.

maintainer
#12 lore raw
Quoted 11 lines · Christoph Lameter (Ampere)
Quoted 2 lines · Vincent Guittot
Quoted 2 lines

AFAICT we have (at least) two options:
1) Trigger NOHZ balancing on a busy housekeeping CPU (what this patch does)

Isnt there a third option?

3) Trigger load balancing if on a NOHZ_FULL cpu and multiple processes are running on it.

If there are several threads on the NOHZ_FULL CPu then the CPU has its tick firing which will trigger the busy load balance for this CPU

Quoted 2 lines
#13 lore raw
In reply to Valentin Schneider
Quoted 1 line · Valentin Schneider

My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that they need to do compute and then rendezvous for data exchange. Variations in the runtime due to timer ticks cause idle periods where the rendezvous cannot be completed because some cpus are delayed.

The more frequent rendezvous can be performed the better the performance numbers will be.

Quoted 1 line

It's mainly designed to be used with always-running userspace

tasks, > generally affined to a CPU by the system administrator.

hohz full has been reworked somewhat since the early days and works in a more general way today.

Quoted 4 lines

Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

The benchmarks show a regression of 10-20% if the tick is operational. The context switch overhead is negligible since the cpus are doing compute and not system calls.

Quoted 5 lines

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

That should work in general and not depend on any hardware topology. In this case we have a linear sched domain including all processors.

#14 lore raw
Quoted 11 lines · Christoph Lameter (Ampere)
Quoted 1 line · Valentin Schneider

My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that they need to do compute and then rendezvous for data exchange. Variations in the runtime due to timer ticks cause idle periods where the rendezvous cannot be completed because some cpus are delayed.

The more frequent rendezvous can be performed the better the performance numbers will be.

[...]

Quoted 4 lines

The benchmarks show a regression of 10-20% if the tick is operational. The context switch overhead is negligible since the cpus are doing compute and not system calls.

Ah good, that's useful information, thanks!

Quoted 8 lines
Quoted 5 lines

As for the actual balancing, yeah if you have idle NOHZ_FULL CPUs they won't do the periodic balance; the residual 1Hz remote tick doesn't do that either. But they should still do the newidle balance to pull work before going tickless idle, and wakeup balance should help as well, albeit that also depends on your topology.

That should work in general and not depend on any hardware topology. In this case we have a linear sched domain including all processors.

Wakeup balance not so much, select_idle_sibling() won't move tasks outside of the waker's LLC - but AIUI in your case you have just a single node and I'm assuming one big LLC.

#15 lore raw
In reply to Christoph Lameter (Ampere)
Quoted 6 lines · Christoph Lameter (Ampere)
Quoted 1 line · Valentin Schneider

My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that they need to do compute and then rendezvous for data exchange.

Ok, I was about to say that this is the first known (for me) usecase of nohz_full that is about performance and doesn't strictly require low-latency guarantee. But...

Quoted 6 lines

Variations
in the runtime due to timer ticks cause idle periods where the rendezvous cannot be completed because some cpus are delayed.

The more frequent rendezvous can be performed the better the performance numbers will be.

...that is low-latency requirement...for performance :-)

That's an argument _not_ in favour of dynamic balancing such as ILB, even for this usecase in nohz_full (all the other usecases of nohz_full I know really want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

Quoted 6 lines
Quoted 2 lines

It's mainly designed to be used with always-running userspace

tasks, > generally affined to a CPU by the system administrator.

hohz full has been reworked somewhat since the early days and works in a more general way today.

Not sure about that. Although it was not initially intended to, it has been very single purpose since the early days: ie: run a single task in userspace without being disturbed.

Quoted 6 lines
Quoted 4 lines

Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

The benchmarks show a regression of 10-20% if the tick is operational.

Impressive!

Quoted 2 lines

The context switch overhead is negligible since the cpus are doing compute and not system calls.

And not many syscalls, right?

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#16 lore raw
Quoted 11 lines · Frederic Weisbecker
Quoted 6 lines · Christoph Lameter (Ampere)
Quoted 1 line · Valentin Schneider

My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that they need to do compute and then rendezvous for data exchange.

Ok, I was about to say that this is the first known (for me) usecase of nohz_full that is about performance and doesn't strictly require low-latency guarantee. But...

For me it was always about both. Low latency is required for a high number of compute cycles in HPC apps. It is a requiremen for high performance parallelized compute.

Quoted 4 lines
Quoted 2 lines

The more frequent rendezvous can be performed the better the performance numbers will be.

...that is low-latency requirement...for performance :-)

Yea thats why we want this in HPC/HFT and AI applications.

Quoted 5 lines

That's an argument _not_ in favour of dynamic balancing such as ILB, even for this usecase in nohz_full (all the other usecases of nohz_full I know really want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

Static affinities are great but they keep the tick active and thus the rendevous can be off off one or the other compute thread.

Quoted 6 lines
Quoted 2 lines

hohz full has been reworked somewhat since the early days and works in a more general way today.

Not sure about that. Although it was not initially intended to, it has been very single purpose since the early days: ie: run a single task in userspace without being disturbed.

The restrictions have been reduced from what I see in the code and syscalls are possible without incurring a 2 second penalty of ticks.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

Quoted 8 lines
Quoted 1 line

The benchmarks show a regression of 10-20% if the tick is operational.

Impressive!

Quoted 2 lines

The context switch overhead is negligible since the cpus are doing compute and not system calls.

And not many syscalls, right?

Periodically the data needs to be saved but that can be done from special threads or after a large number of compute cycles is complete.

#17 lore raw
Quoted 10 lines · Christoph Lameter (Ampere)
Quoted 8 lines · Frederic Weisbecker
Quoted 6 lines · Christoph Lameter (Ampere)
Quoted 1 line · Valentin Schneider

My first question would be: is NOHZ_FULL really right for your workload?

Yes performance is improved. AI workloads are like HPC workloads in that they need to do compute and then rendezvous for data exchange.

Quoted 7 lines
Quoted 3 lines

Ok, I was about to say that this is the first known (for me) usecase of nohz_full that is about performance and doesn't strictly require low-latency guarantee. But...

For me it was always about both. Low latency is required for a high number of compute cycles in HPC apps. It is a requiremen for high performance parallelized compute.

Right, it's just that until now I was used to workloads that would even be broken if the occasional jitter reached some threshold, which doesn't appear to be your case.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The more frequent rendezvous can be performed the better the performance numbers will be.

...that is low-latency requirement...for performance :-)

Yea thats why we want this in HPC/HFT and AI applications.

Ok.

Quoted 8 lines
Quoted 5 lines

That's an argument _not_ in favour of dynamic balancing such as ILB, even for this usecase in nohz_full (all the other usecases of nohz_full I know really want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

Static affinities are great but they keep the tick active and thus the rendevous can be off off one or the other compute thread.

How do static affinities keep the tick active?

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

hohz full has been reworked somewhat since the early days and works in a more general way today.

Not sure about that. Although it was not initially intended to, it has been very single purpose since the early days: ie: run a single task in userspace without being disturbed.

The restrictions have been reduced from what I see in the code and syscalls are possible without incurring a 2 second penalty of ticks.

Yes the isolation has been improved overall but the basic constraints remain.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Here AIUI you're relying on the scheduler load balancing to distribute work to the NOHZ_FULL CPUs, so you're going to be penalized a lot by the NOHZ_FULL context switch overheads. What's the point? Wouldn't you have less overhead with just NOHZ_IDLE?

The benchmarks show a regression of 10-20% if the tick is operational.

Impressive!

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The context switch overhead is negligible since the cpus are doing compute and not system calls.

And not many syscalls, right?

Periodically the data needs to be saved but that can be done from special threads or after a large number of compute cycles is complete.

Got it!

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#18 lore raw
Quoted 10 lines · Frederic Weisbecker
Quoted 8 lines
Quoted 5 lines

That's an argument _not_ in favour of dynamic balancing such as ILB, even for this usecase in nohz_full (all the other usecases of nohz_full I know really want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

Static affinities are great but they keep the tick active and thus the rendevous can be off off one or the other compute thread.

How do static affinities keep the tick active?

If you dont use hohz_full and only do static affinities then the tick is active.

My wish for the future would be that nohz_full would be the default and that the scheduler does correct load balancing regardless of the cpu being in tick mode or not. It should automatically switch the tick off if a cpu goes into 100% compute and ideally there would be no performance regression if the tick is off.

#19 lore raw
Quoted 8 lines · Christoph Lameter (Ampere)
Quoted 6 lines · Frederic Weisbecker
Quoted 6 lines
Quoted 5 lines

That's an argument _not_ in favour of dynamic balancing such as ILB, even for this usecase in nohz_full (all the other usecases of nohz_full I know really want static affinity and no balancing at all).

So I have to ask, what would be wrong with static affinities to these tasks?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Static affinities are great but they keep the tick active and thus the rendevous can be off off one or the other compute thread.

How do static affinities keep the tick active?

If you dont use hohz_full and only do static affinities then the tick is active.

If you don't use nohz_full then the tick is active, yes :-)

Quoted 3 lines

My wish for the future would be that nohz_full would be the default and that the scheduler does correct load balancing regardless of the cpu being in tick mode or not.

Provided we ever manage to run nohz_full without tradeoffs (offloaded 1Hz tick, RCU user/kernel overhead, cputime accounting overhead, use hrtick when preemption must be maintained) then yes it would make sense but we are very far from that. I'm not even sure this will ever be possible.

Quoted 3 lines

It should automatically switch the tick off if a cpu goes into 100% compute and ideally there would be no performance regression if the tick is off.

Automatically means the kernel has to guess and make conclusion after some execution time. That doesn't sound right, especially when nohz_full is used with strong jitter-free guarantee, which has been the case for every usecases so far.

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#20 lore raw
Quoted 8 lines · Frederic Weisbecker
Quoted 3 lines

My wish for the future would be that nohz_full would be the default and that the scheduler does correct load balancing regardless of the cpu being in tick mode or not.

Provided we ever manage to run nohz_full without tradeoffs (offloaded 1Hz tick, RCU user/kernel overhead, cputime accounting overhead, use hrtick when preemption must be maintained) then yes it would make sense but we are very far from that. I'm not even sure this will ever be possible.

We need a genius to have a bright idea on how to handle these tradeoffs to avoid manual configurations.