Skip to contentSkip to conversation

nohz: Revert "nohz: Set isolcpus when nohz_full is set"

PATCH In review Frederic Weisbecker 6 participants 15 messages Oct 12, 2015 – Oct 20, 2015 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nohz: Revert "nohz: Set isolcpus when nohz_full is set"

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

Cc Chris Metcalf <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Jones <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -3
index 6159531..3c35b5f 100644
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-
 	sched_init_numa();
 
 	/*
Signature · 1 line
2.5.3
#1 lore raw
Quoted 47 lines · Frederic Weisbecker

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

Cc Chris Metcalf <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Jones <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -0
index 6159531..3c35b5f 100644

kernel/sched/core.c void __init sched_init_smp(void) · line 7243

Quoted 6 lines
kernel/sched/core.c +0 -3
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-

Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".

Thanx, Paul

Quoted 6 lines
kernel/sched/core.c +0 -0
 	sched_init_numa();
 
 	/*
Signature · 1 line
2.5.3
#2 lore raw
Quoted 51 lines · Paul E. McKenney
Quoted 50 lines · Frederic Weisbecker

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

Cc Chris Metcalf <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Jones <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -0
index 6159531..3c35b5f 100644
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
Quoted 7 lines

kernel/sched/core.c void __init sched_init_smp(void) · line 7243

Quoted 3 lines
kernel/sched/core.c +0 -3
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-

Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".

Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32

#3 lore raw
Quoted 60 lines · Frederic Weisbecker
Quoted 51 lines · Paul E. McKenney
Quoted 50 lines · Frederic Weisbecker

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

Cc Chris Metcalf <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Jones <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -0
index 6159531..3c35b5f 100644
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
Quoted 7 lines

kernel/sched/core.c void __init sched_init_smp(void) · line 7243

Quoted 3 lines
kernel/sched/core.c +0 -3
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-

Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".

Quoted 1 line

Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32

True enough. Not sure that having to repeat the CPU list twice qualifies as "easy to use", though. Why not a nohz_full_iso or some such that isolates whatever CPUs you specified?

But we really need people using this to weigh in.

Thanx, Paul

#4 lore raw
Quoted 59 lines · Paul E. McKenney
Quoted 58 lines · Frederic Weisbecker
Quoted 57 lines · Paul E. McKenney
Quoted 47 lines · Frederic Weisbecker

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

Cc Chris Metcalf <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Christoph Lameter <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Dave Jones <[email protected]>
Cc Oleg Nesterov <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Alexey Dobriyan <[email protected]>
Cc Andrew Morton <[email protected]>
---
  kernel/sched/core.c | 3 ---
  1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -0
index 6159531..3c35b5f 100644

kernel/sched/core.c void __init sched_init_smp(void) · line 7243

Quoted 6 lines
kernel/sched/core.c +0 -3
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
  	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
  	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-

Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".

Quoted 4 lines
Quoted 1 line

Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32

True enough. Not sure that having to repeat the CPU list twice qualifies as "easy to use", though. Why not a nohz_full_iso or some such that isolates whatever CPUs you specified?

Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the nohz_full and isolcpus options under the hood?

Signature · 2 lines
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
#5 lore raw
Quoted 6 lines · Chris Metcalf

On 10/12/2015 12:53 PM, Paul E. McKenney wrote:

Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the nohz_full and isolcpus options under the hood?

Yeah if I could do it again, I would have rather created something like cpu_isolation= (which name would conflict with isolcpus though) instead of nohz_full=, because nohz_full= is really just a subset of what people want.

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

task_isolation is really just about tasks so it should be another name.

#6 lore raw
Quoted 2 lines · Frederic Weisbecker

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

Could we also add the rcu settings?

Quoted 1 line

task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

#7 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

Could we also add the rcu settings?

Yeah, those are implied by nohz_full anyway.

Now nohz_full shouldn't imply anything else than nohz but I'm not sure I can change that. Kernel parameters are ABIs of some sort.

Ideally we should have something like cpu_isolation= which implies everything.

Quoted 6 lines
Quoted 2 lines

task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

No, we can still do syscalls :-)

But noise_free_cpus= ?

#8 lore raw
In reply to Christoph Lameter
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

Could we also add the rcu settings?

I thought that I already had. What am I still missing?

Quoted 5 lines
Quoted 1 line

task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

os_jitter_free_cpus?

clean_cpus?

quiet_cpus?

I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.

Thanx, Paul

#9 lore raw
Quoted 16 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

Could we also add the rcu settings?

I thought that I already had. What am I still missing?

Quoted 5 lines
Quoted 1 line

task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

Quoted 8 lines

os_jitter_free_cpus?

clean_cpus?

quiet_cpus?

I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.

I moved forward quite slowly with nohz full over the years but at least I'm proud that this subsystem introduced some of the most interesting naming challenges!

#10 lore raw
Quoted 21 lines · Frederic Weisbecker
Quoted 20 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker

But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.

Could we also add the rcu settings?

I thought that I already had. What am I still missing?

Quoted 5 lines
Quoted 1 line

task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

os_jitter_free_cpus?

clean_cpus?

Quoted 7 lines
Quoted 4 lines

quiet_cpus?

I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.

I moved forward quite slowly with nohz full over the years but at least I'm proud that this subsystem introduced some of the most interesting naming challenges!

;-) ;-) ;-)

Thanx, Paul

#11 lore raw
Quoted 1 line · Paul E. McKenney

;-) ;-) ;-)

bare_metal_cpus=?

metal=

iron_maiden=

uhh skip the last one.

#12 lore raw
In reply to Frederic Weisbecker
Quoted 9 lines · Frederic Weisbecker
Quoted 6 lines · Chris Metcalf
Quoted 6 lines · Paul E. McKenney

Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the nohz_full and isolcpus options under the hood?

Yeah if I could do it again, I would have rather created something like cpu_isolation= (which name would conflict with isolcpus though)

Well, it only sort of conflicts. I think given that they would be clearly documented they are different enough not to create too much confusion. And cpu_isolation really does seem like a good name.

Signature · 2 lines
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
Acked-by on v1 1/1
#13 lore raw
In reply to Frederic Weisbecker
Quoted 15 lines · Frederic Weisbecker

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.

Quoted 7 lines

Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

#14 lore raw
In reply to Frederic Weisbecker
nohz: Revert "nohz: Set isolcpus when nohz_full is set"
Commit-ID:  0baabb385eb4bce699ddab0db015112be6cf1e6a
Gitweb:     http://git.kernel.org/tip/0baabb385eb4bce699ddab0db015112be6cf1e6a
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Mon, 12 Oct 2015 17:21:23 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 20 Oct 2015 10:13:36 +0200

nohz: Revert "nohz: Set isolcpus when nohz_full is set"

This reverts:

8cb9764fc88b ("nohz: Set isolcpus when nohz_full is set")

We assumed that full-nohz users always want scheduler isolation on full dynticks CPUs, therefore we included full-nohz CPUs on cpu_isolated_map.

This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP - which is true in a sense.

Some full-nohz users appear to be interested in running normal workloads either before or after an isolation workload. Full-nohz isn't optimized toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.

---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -3
index 10a8faa..5bd7d60 100644
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-
 	sched_init_numa();
 
 	/*