nohz: Revert "nohz: Set isolcpus when nohz_full is set"
PATCH 1/1 nohz: Revert "nohz: Set isolcpus when nohz_full is set"
This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map. This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.
This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.
Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.
--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -3
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL ); -/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -sched_init_numa ();/*
Signature · 1 line
2.5.3
Quoted 47 lines · Frederic Weisbecker
This reverts commit
8cb9764fc88b41db11f251e8b2a0d006578b7eb4.We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on
cpu_isolated_map. This means that tasks run by default on CPUs outside thenohz_fullrange unless their affinity is explicity overwritten.This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when
CONFIG_NO_HZ_FULL_ALLis enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_mapinclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.Reported-by Ingo Molnar <[email protected]>Reported-by Mike Galbraith <[email protected]>Cc Chris Metcalf <[email protected]>Cc Rik van Riel <[email protected]>Cc Christoph Lameter <[email protected]>Cc Mike Galbraith <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Dave Jones <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Alexey Dobriyan <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c void __init sched_init_smp(void) · line 7243
Quoted 6 lines
kernel/sched/core.c +0
-3@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL ); -/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -
Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".
Thanx, Paul
Quoted 6 lines
kernel/sched/core.c +0
-0sched_init_numa ();/* Signature · 1 line
2.5.3
Quoted 51 lines · Paul E. McKenney
Quoted 50 lines · Frederic Weisbecker
This reverts commit
8cb9764fc88b41db11f251e8b2a0d006578b7eb4.We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on
cpu_isolated_map. This means that tasks run by default on CPUs outside thenohz_fullrange unless their affinity is explicity overwritten.This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when
CONFIG_NO_HZ_FULL_ALLis enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_mapinclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.Reported-by Ingo Molnar <[email protected]>Reported-by Mike Galbraith <[email protected]>Cc Chris Metcalf <[email protected]>Cc Rik van Riel <[email protected]>Cc Christoph Lameter <[email protected]>Cc Mike Galbraith <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Dave Jones <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Alexey Dobriyan <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)kernel/sched/core.c +0
-0@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL );
Quoted 7 lines
kernel/sched/core.c void __init sched_init_smp(void) · line 7243
Quoted 3 lines
kernel/sched/core.c +0
-3-/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".
Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32
Quoted 60 lines · Frederic Weisbecker
Quoted 51 lines · Paul E. McKenney
Quoted 50 lines · Frederic Weisbecker
This reverts commit
8cb9764fc88b41db11f251e8b2a0d006578b7eb4.We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on
cpu_isolated_map. This means that tasks run by default on CPUs outside thenohz_fullrange unless their affinity is explicity overwritten.This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when
CONFIG_NO_HZ_FULL_ALLis enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_mapinclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.Reported-by Ingo Molnar <[email protected]>Reported-by Mike Galbraith <[email protected]>Cc Chris Metcalf <[email protected]>Cc Rik van Riel <[email protected]>Cc Christoph Lameter <[email protected]>Cc Mike Galbraith <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Dave Jones <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Alexey Dobriyan <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)kernel/sched/core.c +0
-0@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL );Quoted 7 lines
kernel/sched/core.c void __init sched_init_smp(void) · line 7243
Quoted 3 lines
kernel/sched/core.c +0
-3-/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".
Quoted 1 line
Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32
True enough. Not sure that having to repeat the CPU list twice qualifies as "easy to use", though. Why not a nohz_full_iso or some such that isolates whatever CPUs you specified?
But we really need people using this to weigh in.
Thanx, Paul
Quoted 59 lines · Paul E. McKenney
Quoted 58 lines · Frederic Weisbecker
Quoted 57 lines · Paul E. McKenney
Quoted 47 lines · Frederic Weisbecker
This reverts commit
8cb9764fc88b41db11f251e8b2a0d006578b7eb4.We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on
cpu_isolated_map. This means that tasks run by default on CPUs outside thenohz_fullrange unless their affinity is explicity overwritten.This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when
CONFIG_NO_HZ_FULL_ALLis enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_mapinclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.Reported-by Ingo Molnar <[email protected]>Reported-by Mike Galbraith <[email protected]>Cc Chris Metcalf <[email protected]>Cc Rik van Riel <[email protected]>Cc Christoph Lameter <[email protected]>Cc Mike Galbraith <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Dave Jones <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Alexey Dobriyan <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)kernel/sched/core.c +0
-0kernel/sched/core.c void __init sched_init_smp(void) · line 7243
Quoted 6 lines
kernel/sched/core.c +0
-3@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL ); -/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -Why not make this controlled by a boot parameter? That preserves the ease of use for those needing it, but avoids problems from people doing "make randconfig".
Quoted 4 lines
Quoted 1 line
Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32
True enough. Not sure that having to repeat the CPU list twice qualifies as "easy to use", though. Why not a
nohz_full_isoor some such that isolates whatever CPUs you specified?
Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the nohz_full and isolcpus options under the hood?
Signature · 2 lines
Chris Metcalf, EZChip Semiconductor http://www.ezchip.com
Quoted 6 lines · Chris Metcalf
On 10/12/2015 12:53 PM, Paul E. McKenney wrote:
Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the
nohz_fulland isolcpus options under the hood?
Yeah if I could do it again, I would have rather created something like cpu_isolation= (which name would conflict with isolcpus though) instead of nohz_full=, because nohz_full= is really just a subset of what people want.
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
task_isolation is really just about tasks so it should be another name.
Quoted 2 lines · Frederic Weisbecker
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
Could we also add the rcu settings?
Quoted 1 line
task_isolationis really just about tasks so it should be another name.
no_os_cpus?
os_free_cpus?
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
Could we also add the rcu settings?
Yeah, those are implied by nohz_full anyway.
Now nohz_full shouldn't imply anything else than nohz but I'm not sure I can change that. Kernel parameters are ABIs of some sort.
Ideally we should have something like cpu_isolation= which implies everything.
Quoted 6 lines
Quoted 2 lines
task_isolationis really just about tasks so it should be another name.
no_os_cpus?
os_free_cpus?
No, we can still do syscalls :-)
But noise_free_cpus= ?
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
Could we also add the rcu settings?
I thought that I already had. What am I still missing?
Quoted 5 lines
Quoted 1 line
task_isolationis really just about tasks so it should be another name.
no_os_cpus?
os_free_cpus?
os_jitter_free_cpus?
clean_cpus?
quiet_cpus?
I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.
Thanx, Paul
Quoted 16 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
Could we also add the rcu settings?
I thought that I already had. What am I still missing?
Quoted 5 lines
Quoted 1 line
task_isolationis really just about tasks so it should be another name.
no_os_cpus?
os_free_cpus?
Quoted 8 lines
os_jitter_free_cpus?
clean_cpus?
quiet_cpus?I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.
I moved forward quite slowly with nohz full over the years but at least I'm proud that this subsystem introduced some of the most interesting naming challenges!
Quoted 21 lines · Frederic Weisbecker
Quoted 20 lines · Paul E. McKenney
Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Frederic Weisbecker
But yeah if you guys want to create a new parameter that gathers nohz and isolcpus I think we can.
Could we also add the rcu settings?
I thought that I already had. What am I still missing?
Quoted 5 lines
Quoted 1 line
task_isolationis really just about tasks so it should be another name.
no_os_cpus?
os_free_cpus?
os_jitter_free_cpus?
clean_cpus?
Quoted 7 lines
Quoted 4 lines
quiet_cpus?I have no problem with any of the names proposed thus far, but obviously could not resist the urge to propose a few more.
I moved forward quite slowly with nohz full over the years but at least I'm proud that this subsystem introduced some of the most interesting naming challenges!
;-) ;-) ;-)
Thanx, Paul
Quoted 1 line · Paul E. McKenney
;-) ;-) ;-)
bare_metal_cpus=?
metal=
iron_maiden=
uhh skip the last one.
Quoted 9 lines · Frederic Weisbecker
Quoted 6 lines · Chris Metcalf
Quoted 6 lines · Paul E. McKenney
Is it worth starting to think about grouping things under the "task isolation" model somehow? "task_isolation_cpus=1-31" or some such for this, and then that just sets up the
nohz_fulland isolcpus options under the hood?Yeah if I could do it again, I would have rather created something like cpu_isolation= (which name would conflict with isolcpus though)
Well, it only sort of conflicts. I think given that they would be clearly documented they are different enough not to create too much confusion. And cpu_isolation really does seem like a good name.
Signature · 2 lines
Chris Metcalf, EZChip Semiconductor http://www.ezchip.com
Quoted 15 lines · Frederic Weisbecker
This reverts commit
8cb9764fc88b41db11f251e8b2a0d006578b7eb4.We assumed that nohz full users always want scheduler isolation on full dynticks CPUs, therefore we included nohz full CPUs on
cpu_isolated_map. This means that tasks run by default on CPUs outside thenohz_fullrange unless their affinity is explicity overwritten.This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when
CONFIG_NO_HZ_FULL_ALLis enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP.
Quoted 7 lines
Some nohz full users appear to be interested in running normal workloads either before or after an isolation workload. Nohz full isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_mapinclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.
tip-bot for Frederic Weisbecker
merge botnohz: Revert "nohz: Set isolcpus when nohz_full is set"
Commit-ID: 0baabb385eb4bce699ddab0db015112be6cf1e6a Gitweb: http://git.kernel.org/tip/0baabb385eb4bce699ddab0db015112be6cf1e6a Author: Frederic Weisbecker <[email protected]> AuthorDate: Mon, 12 Oct 2015 17:21:23 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 20 Oct 2015 10:13:36 +0200
nohz: Revert "nohz: Set isolcpus when nohz_full is set"
This reverts:
8cb9764fc88b ("nohz: Set isolcpus when nohz_full is set")
We assumed that full-nohz users always want scheduler isolation on full dynticks CPUs, therefore we included full-nohz CPUs on cpu_isolated_map.
This means that tasks run by default on CPUs outside the nohz_full range unless their affinity is explicity overwritten.
This suits pure isolation workloads but when the machine is needed to run common workloads, the available sets of CPUs to run common tasks becomes reduced.
We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it leaves only CPU 0 for non-isolation tasks, which makes people think that their supercomputer regressed to 90's UP - which is true in a sense.
Some full-nohz users appear to be interested in running normal workloads either before or after an isolation workload. Full-nohz isn't optimized toward normal workloads but it's still better than UP performance.
We are reaching a limitation in kernel presets here. Lets revert this cpu_isolated_map inclusion and let userspace do its own scheduler isolation using cpusets or explicit affinity settings.
--- kernel/sched/core.c | 3 --- 1 file changed, 3 deletions(-)
kernel/sched/core.c +0 -3
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)alloc_cpumask_var (&non_isolated_cpus,GFP_KERNEL );alloc_cpumask_var (&fallback_doms,GFP_KERNEL ); -/* nohz_full won't take effect without isolating the cpus. */ -tick_nohz_full_add_cpus_to (cpu_isolated_map); -sched_init_numa ();/*