Skip to contentSkip to conversation

disabled APICs being counted as processors ?

DISCUSSION Dave Jones 5 participants 11 messages Jan 23, 2014 – Jan 26, 2014 x86

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

I have a system with 4 cores (configured with CONFIG_NR_CPUS=4) that shows during boot..

[    0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)

Should the CPU counting code be ignoring those disabled APICs ?

Dave

#1 lore raw
Quoted 15 lines · Dave Jones

I have a system with 4 cores (configured with CONFIG_NR_CPUS=4) that shows during boot..

[    0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 1 line

Should the CPU counting code be ignoring those disabled APICs ?

Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?

But perhaps the boot message could be improved to say something like:

Quoted 1 line
[    0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4

?

Thanks,

Ingo

author
#2 lore raw
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Dave Jones

I have a system with 4 cores (configured with CONFIG_NR_CPUS=4) that shows during boot..

[    0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 9 lines
Quoted 1 line

Should the CPU counting code be ignoring those disabled APICs ?

Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?

But perhaps the boot message could be improved to say something like:

Quoted 1 line
[    0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4

It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHz

Dave

#3 lore raw
Quoted 17 lines · Dave Jones
Quoted 17 lines
Quoted 11 lines

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 1 line

Should the CPU counting code be ignoring those disabled APICs ?

Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?

Quoted 7 lines
Quoted 3 lines

But perhaps the boot message could be improved to say something like:

Quoted 1 line
[    0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4

It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHz

I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.

That said, I think the "smpboot: 8 Processors exceeds NR_CPUS limit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you have CONFIG_HOTPLUG_CPU enabled and it's counting the disabled cpus that were found when acpi_register_lapic() was done. The warning is only really meaningful for cpus in cpu_possible_map, which aren't set for your disabled four, in the hotplug case where NR_CPUS is too small.

#4 lore raw
Quoted 33 lines · David Rientjes
Quoted 17 lines · Dave Jones
Quoted 17 lines
Quoted 11 lines

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 1 line

Should the CPU counting code be ignoring those disabled APICs ?

Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?

Quoted 7 lines
Quoted 3 lines

But perhaps the boot message could be improved to say something like:

Quoted 1 line
[    0.000000] smpboot: 8 possible processors exceeds NR_CPUS limit of 4

It's not possible though. It's an i5-4670T, in a single socket board.
It doesn't even have hyperthreading. http://ark.intel.com/products/75050/Intel-Core-i5-4670T-Processor-6M-Cache-up-to-3_30-GHz

I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.

Quoted 7 lines

That said, I think the "smpboot: 8 Processors exceeds NR_CPUS limit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you have CONFIG_HOTPLUG_CPU enabled and it's counting the disabled cpus that were found when acpi_register_lapic() was done. The warning is only really meaningful for cpus in cpu_possible_map, which aren't set for your disabled four, in the hotplug case where NR_CPUS is too small.

No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.

Thoughts?

Thanks,

Ingo

arch/x86/kernel/smpboot.c +1 -4
index a32da80..75a351a 100644
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
 	i = setup_max_cpus ?: 1;
 	if (setup_possible_cpus == -1) {
 		possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += disabled_cpus;
-#else
+#ifndef CONFIG_HOTPLUG_CPU
 		if (possible > i)
 			possible = i;
 #endif
#5 lore raw
Quoted 19 lines · Ingo Molnar

No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.

arch/x86/kernel/smpboot.c +0 -0
index a32da80..75a351a 100644
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
        i = setup_max_cpus ?: 1;

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229

Quoted 10 lines
arch/x86/kernel/smpboot.c +1 -4
        if (setup_possible_cpus == -1) {
                possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-               if (setup_max_cpus)
-                       possible += disabled_cpus;
-#else
+#ifndef CONFIG_HOTPLUG_CPU
                if (possible > i)
                        possible = i;
 #endif

Agreed.

Most happens when one BIOS support different configuration.

Like system support 10 cores cpu and 15 cores cpus or system could be configured with 4 sockets or 8 sockets

Thanks

Yinghai

#6 lore raw
Quoted 24 lines · Yinghai Lu
Quoted 23 lines · Ingo Molnar

No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager. Since I haven't actually seen _true_ hotplug CPU hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.

arch/x86/kernel/smpboot.c +0 -2
index a32da80..75a351a 100644
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
        i = setup_max_cpus ?: 1;
        if (setup_possible_cpus == -1) {
                possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-               if (setup_max_cpus)
Quoted 8 lines

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229

Quoted 6 lines
arch/x86/kernel/smpboot.c +1 -2
-                       possible += disabled_cpus;
-#else
+#ifndef CONFIG_HOTPLUG_CPU
                if (possible > i)
                        possible = i;
 #endif

Agreed.

A question would be kexec and virtualization: do any of those variants boot a kernel with 'disabled but working' CPUs, which could be hot-onlined later on?

Thanks,

Ingo

#7 lore raw
In reply to Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 6 lines

I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.

Quoted 7 lines

That said, I think the "smpboot: 8 Processors exceeds NR_CPUS limit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you have CONFIG_HOTPLUG_CPU enabled and it's counting the disabled cpus that were found when acpi_register_lapic() was done. The warning is only really meaningful for cpus in cpu_possible_map, which aren't set for your disabled four, in the hotplug case where NR_CPUS is too small.

Quoted 4 lines

No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.

Yeah, because I bet Dave has CONFIG_HOTPLUG_CPU enabled and it's adding this to the number of possible cpus when in reality, per the spec, these cpus aren't possible at all because their enable bit isn't set in their lapic flags.

Quoted 2 lines

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.

In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.

Quoted 18 lines

Since I haven't actually seen _true_ hotplug CPU
hardware yet, I'd argue we do the change below - allocating space for never-present CPUs is stupid. If there's true hot-plug CPUs around that could come online after we've booted, then we want to know about them explicitly.

Thoughts?

Thanks,

Ingo

arch/x86/kernel/smpboot.c +0 -0
index a32da80..75a351a 100644
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
 	i = setup_max_cpus ?: 1;

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229

Quoted 10 lines
arch/x86/kernel/smpboot.c +1 -4
 	if (setup_possible_cpus == -1) {
 		possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += disabled_cpus;
-#else
+#ifndef CONFIG_HOTPLUG_CPU
 		if (possible > i)
 			possible = i;
 #endif

Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used possible_cpus.

I think you should also just do "total_cpus = possible" though and forget about disabled_cpus or /sys/devices/system/cpu/offline is still going to show him 4-7.

This function could benefit from a cleanup at the same time, it's not looking good:

- "i" is a horribly named variable that stores the value so at least one cpu is possible when "nosmp" is used,

- what's with the

#ifdef CONFIG_HOTPLUG_CPU
if (!setup_max_cpus)
#endif ?

if I do "maxcpus=4 nr_cpus=6 possible_cpus=8" what's the expected behavior? We're not only testing for "nosmp" use here, "possible" should still be 4, and

- the warning references "max_cpus" but the kernel command line option is "maxcpus"

#8 lore raw
Quoted 21 lines · David Rientjes
Quoted 14 lines · Ingo Molnar
Quoted 6 lines

I don't think the "ACPI: LAPIC (... disabled)" lines are problematic, they are simply reporting the acpi processor id and apic id for processors that do not have their enabled flag set. The acpi spec allows for these to exist without the enabled flag set when the processor isn't present at all because the kernel will make no attempt to use it.

Quoted 7 lines

That said, I think the "smpboot: 8 Processors exceeds NR_CPUS limit of 4" line is unnecessary since, as you said, these processors don't physically exist. I betcha that's because you have CONFIG_HOTPLUG_CPU enabled and it's counting the disabled cpus that were found when acpi_register_lapic() was done. The warning is only really meaningful for cpus in cpu_possible_map, which aren't set for your disabled four, in the hotplug case where NR_CPUS is too small.

Quoted 4 lines

No, this message is printed in prefill_possible_map() which _generates_ cpu_possible_map, so '8' is the number of bits in cpu_possible_map.

Quoted 4 lines

Yeah, because I bet Dave has CONFIG_HOTPLUG_CPU enabled and it's adding this to the number of possible cpus when in reality, per the spec, these cpus aren't possible at all because their enable bit isn't set in their lapic flags.

Yeah, I suspect Dave has a distro-ish .config on his desktop, and distros generally enable all things hot-plug.

Quoted 6 lines
Quoted 2 lines

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.

In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.

Correct. Is there a robust distinction possible between the two?

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/smpboot.c +0 -4
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
 	i = setup_max_cpus ?: 1;
 	if (setup_possible_cpus == -1) {
 		possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += disabled_cpus;
-#else
Quoted 8 lines

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229

Quoted 4 lines
arch/x86/kernel/smpboot.c +1 -0
+#ifndef CONFIG_HOTPLUG_CPU
 		if (possible > i)
 			possible = i;
 #endif

Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used possible_cpus.

Not just that, it also reduces the number of possible CPUs, which should reduce percpu memory allocation overhead, amongst other things, right?

Quoted 3 lines

I think you should also just do "total_cpus = possible" though and forget about disabled_cpus or /sys/devices/system/cpu/offline is still going to show him 4-7.

Agreed.

Quoted 12 lines

This function could benefit from a cleanup at the same time, it's not looking good:

- "i" is a horribly named variable that stores the value so at least one cpu is possible when "nosmp" is used,

- what's with the

#ifdef CONFIG_HOTPLUG_CPU
if (!setup_max_cpus)
#endif ?

Quoted 6 lines

if I do "maxcpus=4 nr_cpus=6 possible_cpus=8" what's the expected behavior? We're not only testing for "nosmp" use here, "possible" should still be 4, and

- the warning references "max_cpus" but the kernel command line option is "maxcpus"

Ack.

I wouldn't object to someone sending a changelogged, tested patch that does all that. Maybe two patches: first the cleanups, then the CPU count trimming. Just in case it regresses ...

Thanks,

Ingo

#9 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 6 lines
Quoted 2 lines

So the problem is that the counting of disabled but hotpluggable CPUs is over-eager.

In the kernel, yeah, and we don't distinguish between physically absent processors that have lapic entries and physically present but disabled processors.

Correct. Is there a robust distinction possible between the two?

Not with acpi, I'm afraid, which allows for both possibilities to either have no lapic entry or have ACPI_MADT_ENABLED clear.

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/smpboot.c +0 -4
@@ -1223,10 +1223,7 @@ __init void prefill_possible_map(void)
 	i = setup_max_cpus ?: 1;
 	if (setup_possible_cpus == -1) {
 		possible = num_processors;
-#ifdef CONFIG_HOTPLUG_CPU
-		if (setup_max_cpus)
-			possible += disabled_cpus;
-#else
Quoted 8 lines

arch/x86/kernel/smpboot.c __init void prefill_possible_map(void) · line 1229

Quoted 4 lines
arch/x86/kernel/smpboot.c +1 -0
+#ifndef CONFIG_HOTPLUG_CPU
 		if (possible > i)
 			possible = i;
 #endif

Yeah, this should suppress the warning for Dave. This way, the only way the log reports the number of "hotplug CPUs" is because we used possible_cpus.

Quoted 4 lines

Not just that, it also reduces the number of possible CPUs, which should reduce percpu memory allocation overhead, amongst other things, right?

Indeed, it gives people a good motivation for clearing out those unnecessary lapic entries :)

Quoted 4 lines

I wouldn't object to someone sending a changelogged, tested patch that does all that. Maybe two patches: first the cleanups, then the CPU count trimming. Just in case it regresses ...

Sounds good. I need to look into your point about kexec as far as the possible count trimming first for the second patch.

#10 lore raw
In reply to Ingo Molnar
Quoted 16 lines · Ingo Molnar
Quoted 15 lines · Dave Jones

I have a system with 4 cores (configured with CONFIG_NR_CPUS=4) that shows during boot..

[    0.000000] smpboot: 8 Processors exceeds NR_CPUS limit of 4

it looks like this is because..

[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0xff] disabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0xff] disabled)
Quoted 5 lines
Quoted 1 line

Should the CPU counting code be ignoring those disabled APICs ?

Hm, so to the kernel it looks like as if those were 'possible CPUs', in theory hotpluggable. Not sure what they are - disabled cores in an 8-core system? Or BIOS reporting crap?

It is sort of a standard practice for the BIOS to report ACPI tables listing all possible cores for the largest possible processor the motherboard (sometimes several motherboards that share the same BIOS) could handle. Really easy to find that quirk in anything by SuperMicro, for example.

The kernel things these extra CPU cores will show up eventually, and lists them (and reserves resources to handle them) as hotpluggable CPUs.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh