disabled APICs being counted as processors ?
I have a system with 4 cores (configured with CONFIG_NR_CPUS=4) that shows during boot..
[ 0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4
it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Should the CPU counting code be ignoring those disabled APICs ?
Dave
Quoted 15 lines · Dave Jones
I have a system with 4 cores (configured with
CONFIG_NR_CPUS=4) that shows during boot..[ 0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 1 line
Should the CPU counting code be ignoring those disabled APICs ?
Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?
But perhaps the boot message could be improved to say something like:
Quoted 1 line
[ 0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4
?
Thanks,
Ingo
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Dave Jones
I have a system with 4 cores (configured with
CONFIG_NR_CPUS=4) that shows during boot..[ 0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 9 lines
Quoted 1 line
Should the CPU counting code be ignoring those disabled APICs ?
Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?
But perhaps the boot message could be improved to say something like:
Quoted 1 line
[ 0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4
It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHz
Dave
Quoted 17 lines · Dave Jones
Quoted 17 lines
Quoted 11 lines
it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)Quoted 1 line
Should the CPU counting code be ignoring those disabled APICs ?
Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?
Quoted 7 lines
Quoted 3 lines
But perhaps the boot message could be improved to say something like:
Quoted 1 line
[ 0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHz
I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.
That said, I think the "smpboot: 8 Processors exceeds NR_CPUS limit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you have CONFIG_HOTPLUG_CPU enabled and it's counting the disabled cpus that were found when acpi_register_lapic() was done. The warning is only really meaningful for cpus in cpu_possible_map, which aren't set for your disabled four, in the hotplug case where NR_CPUS is too small.
Quoted 33 lines · David Rientjes
Quoted 17 lines · Dave Jones
Quoted 17 lines
Quoted 11 lines
it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)Quoted 1 line
Should the CPU counting code be ignoring those disabled APICs ?
Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?
Quoted 7 lines
Quoted 3 lines
But perhaps the boot message could be improved to say something like:
Quoted 1 line
[ 0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHzI don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.
Quoted 7 lines
That said, I think the "smpboot: 8 Processors exceeds
NR_CPUSlimit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you haveCONFIG_HOTPLUG_CPUenabled and it's counting the disabled cpus that were found whenacpi_register_lapic()was done. The warning is only really meaningful for cpus incpu_possible_map, which aren't set for your disabled four, in the hotplug case whereNR_CPUSis too small.
No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.
So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.
Thoughts?
Thanks,
Ingo
arch/x86/kernel/smpboot.c +1 -4
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus) - possible += disabled_cpus; -#else +#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif
Quoted 19 lines · Ingo Molnar
No, this message is printed in
prefill_possible_map()which _generates_cpu_possible_map, so '8' is the number of bits incpu_possible_map.So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.
arch/x86/kernel/smpboot.c +0
-0@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;
arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229
Quoted 10 lines
arch/x86/kernel/smpboot.c +1
-4if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus) - possible += disabled_cpus; -#else +#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif
Agreed.
Most happens when one BIOS support different configuration.
Like system support 10 cores cpu and 15 cores cpus or system could be configured with 4 sockets or 8 sockets
Thanks
Yinghai
Quoted 24 lines · Yinghai Lu
Quoted 23 lines · Ingo Molnar
No, this message is printed in
prefill_possible_map()which _generates_cpu_possible_map, so '8' is the number of bits incpu_possible_map.So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.
arch/x86/kernel/smpboot.c +0
-2@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus)
Quoted 8 lines
arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229
Quoted 6 lines
arch/x86/kernel/smpboot.c +1
-2- possible += disabled_cpus; -#else +#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif Agreed.
A question would be kexec and virtualization: do any of those variants boot a kernel with 'disabled but working' CPUs, which could be hot-onlined later on?
Thanks,
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 6 lines
I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.
Quoted 7 lines
That said, I think the "smpboot: 8 Processors exceeds
NR_CPUSlimit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you haveCONFIG_HOTPLUG_CPUenabled and it's counting the disabled cpus that were found whenacpi_register_lapic()was done. The warning is only really meaningful for cpus incpu_possible_map, which aren't set for your disabled four, in the hotplug case whereNR_CPUSis too small.
Quoted 4 lines
No, this message is printed in
prefill_possible_map()which _generates_cpu_possible_map, so '8' is the number of bits incpu_possible_map.
Yeah, because I bet Dave has CONFIG_HOTPLUG_CPU enabled and it's adding this to the number of possible cpus when in reality, per the spec, these cpus aren't possible at all because their enable bit isn't set in their lapic flags.
Quoted 2 lines
So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.
In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.
Quoted 18 lines
Since I haven't actually seen _true_ hotplug CPU
hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.Thoughts?
Thanks,
Ingo
arch/x86/kernel/smpboot.c +0
-0@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;
arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229
Quoted 10 lines
arch/x86/kernel/smpboot.c +1
-4if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus) - possible += disabled_cpus; -#else +#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif
Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used possible_cpus.
I think you should also just do "total_cpus = possible" though and forget about disabled_cpus or /sys/devices/system/cpu/offline is still going to show him 4-7.
This function could benefit from a cleanup at the same time, it's not looking good:
- "i" is a horribly named variable that stores the value so at least one cpu is possible when "nosmp" is used,
- what's with the
#ifdef CONFIG_HOTPLUG_CPU
if (!setup_max_cpus)
#endif ?
if I do "maxcpus=4 nr_cpus=6 possible_cpus=8" what's the expected behavior? We're not only testing for "nosmp" use here, "possible" should still be 4, and
- the warning references "max_cpus" but the kernel command line option is "maxcpus"
Quoted 21 lines · David Rientjes
Quoted 14 lines · Ingo Molnar
Quoted 6 lines
I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.
Quoted 7 lines
That said, I think the "smpboot: 8 Processors exceeds
NR_CPUSlimit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you haveCONFIG_HOTPLUG_CPUenabled and it's counting the disabled cpus that were found whenacpi_register_lapic()was done. The warning is only really meaningful for cpus incpu_possible_map, which aren't set for your disabled four, in the hotplug case whereNR_CPUSis too small.Quoted 4 lines
No, this message is printed in
prefill_possible_map()which _generates_cpu_possible_map, so '8' is the number of bits incpu_possible_map.
Quoted 4 lines
Yeah, because I bet Dave has
CONFIG_HOTPLUG_CPUenabled and it's adding this to the number of possible cpus when in reality, per the spec, these cpus aren't possible at all because their enable bit isn't set in their lapic flags.
Yeah, I suspect Dave has a distro-ish .config on his desktop, and distros generally enable all things hot-plug.
Quoted 6 lines
Quoted 2 lines
So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.
In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.
Correct. Is there a robust distinction possible between the two?
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/smpboot.c +0
-4@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus) - possible += disabled_cpus; -#else
Quoted 8 lines
arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229
Quoted 4 lines
arch/x86/kernel/smpboot.c +1
-0+#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used
possible_cpus.
Not just that, it also reduces the number of possible CPUs, which should reduce percpu memory allocation overhead, amongst other things, right?
Quoted 3 lines
I think you should also just do "
total_cpus= possible" though and forget aboutdisabled_cpusor/sys/devices/system/cpu/offlineis still going to show him 4-7.
Agreed.
Quoted 12 lines
This function could benefit from a cleanup at the same time, it's not looking good:
- "i" is a horribly named variable that stores the value so at least one cpu is possible when "nosmp" is used,
- what's with the
#ifdef
CONFIG_HOTPLUG_CPU
if (!setup_max_cpus)
#endif ?
Quoted 6 lines
if I do "maxcpus=4 nr_cpus=6 possible_cpus=8" what's the expected behavior? We're not only testing for "nosmp" use here, "possible" should still be 4, and
- the warning references "
max_cpus" but the kernel command line option is "maxcpus"
Ack.
I wouldn't object to someone sending a changelogged, tested patch that does all that. Maybe two patches: first the cleanups, then the CPU count trimming. Just in case it regresses ...
Thanks,
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 6 lines
Quoted 2 lines
So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.
In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.
Correct. Is there a robust distinction possible between the two?
Not with acpi, I'm afraid, which allows for both possibilities to either have no lapic entry or have ACPI_MADT_ENABLED clear.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/smpboot.c +0
-4@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void) i = setup_max_cpus ?:1 ;if (setup_possible_cpus ==-1 ) { possible = num_processors; -#ifdef CONFIG_HOTPLUG_CPU -if (setup_max_cpus) - possible += disabled_cpus; -#else Quoted 8 lines
arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229
Quoted 4 lines
arch/x86/kernel/smpboot.c +1
-0+#ifndef CONFIG_HOTPLUG_CPU if (possible > i) possible = i;#endif Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used
possible_cpus.
Quoted 4 lines
Not just that, it also reduces the number of possible CPUs, which should reduce percpu memory allocation overhead, amongst other things, right?
Indeed, it gives people a good motivation for clearing out those unnecessary lapic entries :)
Quoted 4 lines
I wouldn't object to someone sending a changelogged, tested patch that does all that. Maybe two patches: first the cleanups, then the CPU count trimming. Just in case it regresses ...
Sounds good. I need to look into your point about kexec as far as the possible count trimming first for the second patch.
Quoted 16 lines · Ingo Molnar
Quoted 15 lines · Dave Jones
I have a system with 4 cores (configured with
CONFIG_NR_CPUS=4) that shows during boot..[ 0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4it looks like this is because..
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 5 lines
Quoted 1 line
Should the CPU counting code be ignoring those disabled APICs ?
Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?
It is sort of a standard practice for the BIOS to report ACPI tables listing all possible cores for the largest possible processor the motherboard (sometimes several motherboards that share the same BIOS) could handle. Really easy to find that quirk in anything by SuperMicro, for example.
The kernel things these extra CPU cores will show up eventually, and lists them (and reserves resources to handle them) as hotpluggable CPUs.
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh