Skip to contentSkip to conversation

WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...

DISCUSSION Borislav Petkov 5 participants 9 messages Oct 14, 2011 – Oct 17, 2011

lore
Share & tools
lore

Borislav Petkov

author
#0 lore raw

Hi,

we're hitting the warning below

<arch/x86/kernel/apic/apic.c>:

	...
        i = early_per_cpu(x86_cpu_to_logical_apicid, cpu);
        WARN_ON(i != BAD_APICID && i != logical_smp_processor_id());		<--- HERE
        /* always use the value from LDR */
        early_per_cpu(x86_cpu_to_logical_apicid, cpu) =
                logical_smp_processor_id();

during 32-bit testing, config is attached.

acb8bc09c6185e4d3d582d0076aaa6a89f19d8c5 added this warning and triggers on the second part of the condition. I've dumped x86_cpu_to_logical_apicid and logical_smp_processor_id and here's what I get on a 32-cores box:

...

[    4.268493] CPU 31 irqstacks, hard=e5b1e000 soft=e5b20000
[    4.269358]  #31 Ok.
[    4.270358] smpboot cpu 31: start_ip = 80000
[    0.003999] Initializing CPU#31
[    0.003999] ------------[ cut here ]------------
[    0.003999] WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+0x12c/0x3bd()
[    0.003999] Hardware name: None
[    0.003999] i: -2147483648, lg_smp_id: 79
[    0.003999] Modules linked in:
[    0.003999] Pid: 0, comm: kworker/0:1 Tainted: G        W   3.1.0-rc9-37cf9516-linus+ #1
[    0.003999] Call Trace:
[    0.003999]  [<c1035066>] warn_slowpath_common+0x65/0x7a
[    0.003999]  [<c10350ee>] warn_slowpath_fmt+0x2b/0x2f
[    0.003999]  [<c16603bb>] setup_local_APIC+0x12c/0x3bd
[    0.003999]  [<c165fcae>] start_secondary+0x9f/0x18f
[    0.003999] ---[ end trace 4eaa2a86a8e2da41 ]---

That's right, x86_cpu_to_logical_apicid wraps around so if I were to boot a 32-bit kernel on a 64 cores box, I'd get each two cores to have the same logical apic id.

Oh, and logical_smp_processor_id(), i.e. logical APIC ID starts at 0x20, acc. to SRAT:

[    0.000000] SRAT: PXM 0 -> APIC 0x20 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x21 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x22 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x23 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x24 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x25 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x26 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x27 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2a -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2b -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2c -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2d -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2e -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x2f -> Node 1
[    0.000000] SRAT: PXM 2 -> APIC 0x40 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x41 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x42 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x43 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x44 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x45 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x46 -> Node 2
[    0.000000] SRAT: PXM 2 -> APIC 0x47 -> Node 2
[    0.000000] SRAT: PXM 3 -> APIC 0x48 -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x49 -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4a -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4b -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4c -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4d -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4e -> Node 3
[    0.000000] SRAT: PXM 3 -> APIC 0x4f -> Node 3

The warning triggers IMHO because we switch to bigsmp APIC in default_setup_apic_routing() but the early x86_cpu_to_logical_apicid enumeration comes from default_x86_32_early_logical_apicid() which gets called as part of the default apic deal as part of generic_processor_info().

So, in the end, and AFAICR, the warning triggers because we're comparing logical APIC IDs from the APIC Logical Destination Register (0xD0) which have been assigned by BIOS with "1 << cpu" shifted values which wrap on 32-bit.

I'd very much like to know why?

Thanks.

Signature · 5 lines
Regards/Gruss,
Boris.

Operating Systems Research Center
Advanced Micro Devices, Inc.
author
#1 lore raw
Quoted 6 lines · Borislav Petkov

So, in the end, and AFAICR, the warning triggers because we're comparing logical APIC IDs from the APIC Logical Destination Register (0xD0) which have been assigned by BIOS with "1 << cpu" shifted values which wrap on 32-bit.

I'd very much like to know why?

Update: upon a second thought, we think we know why: The WARN_ON cannot stomach the switch to bigsmp apic because, on the one hand, bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid contains the early percpu value which got assigned by the default apic, i.e. the shifting thing as it is spec'ed for the LDR register.

So, the way we see it, the WARN_ON should know whether we switched apics or should be removed completely. Btw, why was it added in the first place, the commit message doesn't quite explain why.

Thanks.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#2 lore raw

Hello,

(cc'ing Jan Beulich)

Quoted 15 lines · Borislav Petkov
Quoted 6 lines · Borislav Petkov

So, in the end, and AFAICR, the warning triggers because we're comparing logical APIC IDs from the APIC Logical Destination Register (0xD0) which have been assigned by BIOS with "1 << cpu" shifted values which wrap on 32-bit.

I'd very much like to know why?

Update: upon a second thought, we think we know why: The WARN_ON cannot stomach the switch to bigsmp apic because, on the one hand, bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid contains the early percpu value which got assigned by the default apic, i.e. the shifting thing as it is spec'ed for the LDR register.

Quoted 3 lines

So, the way we see it, the WARN_ON should know whether we switched apics or should be removed completely. Btw, why was it added in the first place, the commit message doesn't quite explain why.

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

I can't seem to find the patch in mainline. Jan, Ingo? Borislav, would the patch there resolve the warning you're seeing too?

Anyways, to summarize, the early apic thing isn't necessary anymore and can be ripped out and the warning is pretty much bogus at this point. I'll rip it out soonish.

Thank you.

Signature · 1 line
tejun
author
#3 lore raw
Quoted 6 lines · Tejun Heo

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

I can't seem to find the patch in mainline. Jan, Ingo? Borislav, would the patch there resolve the warning you're seeing too?

There's even a -v2 at http://marc.info/?l=linux-kernel&m=131722471001895

We'll run that one to test.

Quoted 3 lines

Anyways, to summarize, the early apic thing isn't necessary anymore and can be ripped out and the warning is pretty much bogus at this point. I'll rip it out soonish.

Ok, thanks.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#4 lore raw
Quoted 5 lines · On Sat, 15 Oct 2011 18:12:15 -0400 Borislav Petkov <[email protected]> wrote:
Quoted 4 lines · Tejun Heo

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

Quoted 7 lines
Quoted 2 lines

I can't seem to find the patch in mainline. Jan, Ingo? Borislav, would the patch there resolve the warning you're seeing too?

There's even a -v2 at
http://marc.info/?l=linux-kernel&m=131722471001895

We'll run that one to test.

Looks good. The warning is gone.

Quoted 6 lines
Quoted 3 lines

Anyways, to summarize, the early apic thing isn't necessary anymore and can be ripped out and the warning is pretty much bogus at this point. I'll rip it out soonish.

Ok, thanks.

--
regards,
Conny Seidel

##################################################################
# Email : [email protected] GnuPG-Key : 0xA6AB055D #
# Fingerprint: 17C4 5DB2 7C4C C1C7 1452 8148 F139 7C09 A6AB 055D #
##################################################################
# Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach #
# General Managers: Alberto Bozzo #
# Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen #
# HRB Nr. 43632 #
##################################################################

author Acked-by Tested-by Conny Seidel
#5 lore raw
Quoted 11 lines · Conny Seidel
Quoted 8 lines · On Sat, 15 Oct 2011 18:12:15 -0400 Borislav Petkov <[email protected]> wrote:
Quoted 6 lines · Tejun Heo

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

I can't seem to find the patch in mainline. Jan, Ingo? Borislav, would the patch there resolve the warning you're seeing too?

Quoted 6 lines
Quoted 4 lines

There's even a -v2 at
http://marc.info/?l=linux-kernel&m=131722471001895

We'll run that one to test.

Looks good. The warning is gone.

Good. Ingo, you can have our

Acked-by Borislav Petkov <[email protected]>
Tested-by Conny Seidel <[email protected]>

for it, it being the -v2 at http://marc.info/?l=linux-kernel&m=131722471001895

Thanks to all for their work.

Signature · 5 lines
Regards/Gruss,
Boris.

Operating Systems Research Center
Advanced Micro Devices, Inc.
#6 lore raw
Quoted 17 lines · Borislav Petkov
Quoted 16 lines · Conny Seidel
Quoted 12 lines · On Sat, 15 Oct 2011 18:12:15 -0400 Borislav Petkov <[email protected]> wrote:
Quoted 6 lines · Tejun Heo

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

I can't seem to find the patch in mainline. Jan, Ingo? Borislav, would the patch there resolve the warning you're seeing too?

There's even a -v2 at
http://marc.info/?l=linux-kernel&m=131722471001895

We'll run that one to test.

Quoted 8 lines
Quoted 1 line

Looks good. The warning is gone.

Good. Ingo, you can have our

Acked-by Borislav Petkov <[email protected]>
Tested-by Conny Seidel <[email protected]>

for it, it being the -v2 at http://marc.info/?l=linux-kernel&m=131722471001895

Yeah, it's still lined up in -tip:x86/apic for a v3.2 merge.

Thanks,

Ingo

author
#7 lore raw
Quoted 1 line · Ingo Molnar

Yeah, it's still lined up in -tip:x86/apic for a v3.2 merge.

Cool,
thanks.

Signature · 8 lines
Regards/Gruss,
Boris.

Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
#8 lore raw
In reply to Tejun Heo
Quoted 25 lines
Quoted 1 line
Quoted 1 line · Tejun Heo

Hello,

(cc'ing Jan Beulich)

Quoted 15 lines · Borislav Petkov
Quoted 6 lines · Borislav Petkov

So, in the end, and AFAICR, the warning triggers because we're comparing logical APIC IDs from the APIC Logical Destination Register (0xD0) which have been assigned by BIOS with "1 << cpu" shifted values which wrap on 32-bit.

I'd very much like to know why?

Update: upon a second thought, we think we know why: The WARN_ON cannot stomach the switch to bigsmp apic because, on the one hand, bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid contains the early percpu value which got assigned by the default apic, i.e. the shifting thing as it is spec'ed for the LDR register.

Quoted 3 lines

So, the way we see it, the WARN_ON should know whether we switched apics or should be removed completely. Btw, why was it added in the first place, the commit message doesn't quite explain why.

Quoted 5 lines

Jan hit similar problem a while back.

http://thread.gmane.org/gmane.linux.kernel/1180954

I can't seem to find the patch in mainline.

Ingo let me know at the end of September that v2 got applied to -tip; at that point, I took it that it would make only 3.2 (and perhaps then earlier ones via -stable).

Jan