percpu data: only iterate over possible CPUs
In-Reply-To: <[email protected]>
Quoted 8 lines · Andrew Morton
Quoted 4 lines
Its even documented in line 332 of
include/linux/cpumask.h* #ifdef
CONFIG_HOTPLUG_CPU
*cpu_possible_map- allNR_CPUSbits setThat seems a quite bad idea. If we know which CPUs are possible we should populate
cpu_possible_mapcorrectly, whether or notCONFIG_HOTPLUG_CPUis set.
I don't think that's, um, "possible." Even if you could discover how many empty sockets there were in a system, someone might be able to hotplug a board with more of them on it. And there's no way to tell how many CPUs to reserve for each socket anyway, e.g. AMD has already announced quad-core processors.
But what really surprised me is that for_each_cpu() actually walks cpu_possible_map and not cpu_present_map as I had assumed. This violates the Principle Of Least Surprise. Maybe renaming for_each_cpu to for_each_possible_cpu might be a good idea?
Signature · 2 lines
Chuck "Equations are the Devil's sentences." --Stephen Colbert
Quoted 3 lines · Chuck
I don't think that's, um, "possible." Even if you could discover how many empty sockets there were in a system, someone might be able to hotplug a board with more of them on it.
That's going to depend on your system hardware configuration.
cpu_possible_map should be whatever is the largest set of cpus you might possibly want to deal with plugging in.
Some hardware configurations will support more addition of hardware cpus than others.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 14 lines · Chuck Ebbert
In-Reply-To: <[email protected]>
Quoted 8 lines · Andrew Morton
Quoted 4 lines
Its even documented in line 332 of
include/linux/cpumask.h* #ifdef
CONFIG_HOTPLUG_CPU
*cpu_possible_map- allNR_CPUSbits setThat seems a quite bad idea. If we know which CPUs are possible we should populate
cpu_possible_mapcorrectly, whether or notCONFIG_HOTPLUG_CPUis set.
Quoted 5 lines
I don't think that's, um, "possible." Even if you could discover how many empty sockets there were in a system, someone might be able to hotplug a board with more of them on it. And there's no way to tell how many CPUs to reserve for each socket anyway, e.g. AMD has already announced quad-core processors.
Well maybe. But it's awfully presumptuous for us to say that no platform will be capable of telling us what its maximum number of CPUs is, or even whether certain CPUs within that range aren't possible.
<checks>
Yup, on my 2-way x86 test box, with NR_CPUS=16 we have cpu_possible_map=0xffff.
That's just insane - the default setting for a distro kernel should be (or will become) NR_CPUS=lots, HOTPLUG_CPU=y. All those for_each_cpu() loops are iterating across 16 CPUs.
aargh.
Quoted 4 lines
But what really surprised me is that
for_each_cpu()actually walkscpu_possible_mapand notcpu_present_mapas I had assumed. This violates the Principle Of Least Surprise. Maybe renamingfor_each_cputofor_each_possible_cpumight be a good idea?
That would make sense.
Quoted 3 lines · Andrew Morton
aargh.
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use CPU_MASK_NONE (or just forget to initialise it and hope that CPU_MASK_NONE equals all-zeroes).
Quoted 4 lines
Quoted 1 line
aargh.
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).
s390 will join, as soon as the cpu_possible_map fix is merged...
Heiko
Quoted 7 lines · Heiko Carstens
Quoted 5 lines
Quoted 2 lines
aargh.
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).s390 will join, as soon as the
cpu_possible_mapfix is merged...
What cpu_possible_map fix?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
aargh.
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).s390 will join, as soon as the
cpu_possible_mapfix is merged...What
cpu_possible_mapfix?
This one:
http://lkml.org/lkml/2006/2/8/162
Heiko
Quoted 6 lines · Heiko Carstens
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
aargh.
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
s390 will join, as soon as the
cpu_possible_mapfix is merged...What
cpu_possible_mapfix?This one:
Oh, OK. Ow, I don't think you want to do that. It means that all those for_each_cpu() loops will now be iterating over all NR_CPUS cpus, whether or not they're even possible.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).s390 will join, as soon as the
cpu_possible_mapfix is merged...What
cpu_possible_mapfix?
Quoted 8 lines
Quoted 4 lines
This one:
Oh, OK. Ow, I don't think you want to do that. It means that all those
for_each_cpu()loops will now be iterating over allNR_CPUScpus, whether or not they're even possible.
That's ok. We're mainly running under z/VM where you can attach new virtual cpus on the fly to the virtual machine (up to 64 cpus).
The only difference to before is that it was possible to limit the waste of resources by passing a number with 'maxcpus'. This value was used to generate the cpu_possible_map.
But since the map needs to be ready when we return from setup_arch, we don't have access to max_cpus, unless we parse commandline on our own...
Heiko
Quoted 14 lines · Heiko Carstens
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Actually, x86 appears to be the only arch which suffers this braindamage. The rest use
CPU_MASK_NONE(or just forget to initialise it and hope thatCPU_MASK_NONEequals all-zeroes).s390 will join, as soon as the
cpu_possible_mapfix is merged...Quoted 8 lines
Quoted 5 lines
Quoted 1 line
What
cpu_possible_mapfix?This one:
Oh, OK. Ow, I don't think you want to do that. It means that all those
for_each_cpu()loops will now be iterating over allNR_CPUScpus, whether or not they're even possible.
Quoted 8 lines
That's ok. We're mainly running under z/VM where you can attach new virtual cpus on the fly to the virtual machine (up to 64 cpus).
The only difference to before is that it was possible to limit the waste of resources by passing a number with 'maxcpus'. This value was used to generate thecpu_possible_map.
But since the map needs to be ready when we return fromsetup_arch, we don't have access tomax_cpus, unless we parse commandline on our own...
Then it's OK to clear bits from cpu_possible_map once you have max_cpus value
for (cpu = max_cpus ; cpu < NR_CPUS ; cpu++)
cpu_clear(cpu, cpu_possible_map);
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Oh, OK. Ow, I don't think you want to do that. It means that all those
for_each_cpu()loops will now be iterating over allNR_CPUScpus, whether or not they're even possible.That's ok. We're mainly running under z/VM where you can attach new virtual cpus on the fly to the virtual machine (up to 64 cpus).
The only difference to before is that it was possible to limit the waste of resources by passing a number with 'maxcpus'. This value was used to generate thecpu_possible_map.
But since the map needs to be ready when we return fromsetup_arch, we don't have access tomax_cpus, unless we parse commandline on our own...
Quoted 4 lines
Then it's OK to clear bits from
cpu_possible_maponce you havemax_cpusvaluefor (cpu =
max_cpus; cpu <NR_CPUS; cpu++)cpu_clear(cpu, cpu_possible_map);
Hmm... I don't think the semantics of cpu_possible_map allow to change it. Also any code that uses for_each_cpu() may allocate memory _before_ cpu_possible_map is changed back to reflect a smaller number of cpus. Doesn't look like the correct way to fix this.
Thinking a bit longer this was probably a reason why initialization of this map was done in smp_prepare_cpus() before it silently moved to setup_arch().
Heiko
Quoted 10 lines · Heiko Carstens
Quoted 2 lines
for (cpu =
max_cpus; cpu <NR_CPUS; cpu++)cpu_clear(cpu, cpu_possible_map);Hmm... I don't think the semantics of
cpu_possible_mapallow to change it. Also any code that usesfor_each_cpu()may allocate memory _before_cpu_possible_mapis changed back to reflect a smaller number of cpus. Doesn't look like the correct way to fix this.
Thinking a bit longer this was probably a reason why initialization of this map was done insmp_prepare_cpus()before it silently moved tosetup_arch().
Hum... of course you may loose some bits of memory (percpu data for example) but clearing a cpu in cpu_possible_map is allowed.
See arch/alpha/kernel/process.c and arch/x86_64/kernel/smpboot.c for uses of cpu_clear()