x86/smp: Validate APIC ID before parking CPU in INIT
PATCH 1/1 x86/smp: Validate APIC ID before parking CPU in INIT
Below commit is causing kexec to hang in certain scenarios with >255 CPUs.
Reproduce steps:
- We are using 2 socket system with 384 CPUs
- Booting first kernel with kernel command line intremap=off This disabled x2apic in kernel and booted with apic mode
- During kexec it tries to send INIT to all CPUs except boot CPU If APIC ID is 0x100 (like in our case) then it will send CPU0 to INIT mode and system hangs (in APIC mode DEST field is 8bit)
Fix this issue by adding apic->apic_id_valid() check before sending INIT sequence.
--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/smpboot.c +1 -1
@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }
Signature · 1 line
2.31.1
Hi
Did you get a chance to look into this patch?
-Vasant
Quoted 34 lines · Vasant Hegde
Below commit is causing kexec to hang in certain scenarios with >255 CPUs.
Reproduce steps:
- We are using 2 socket system with 384 CPUs
- Booting first kernel with kernel command line intremap=off This disabled x2apic in kernel and booted with apic mode
- During kexec it tries to send INIT to all CPUs except boot CPU If APIC ID is0x100(like in our case) then it will send CPU0 to INIT mode and system hangs (in APIC mode DEST field is 8bit)Fix this issue by adding
apic->apic_id_valid()check before sending INIT sequence.Fixes 45e34c8af58f ("x86/smp: Put CPUs into INIT on shutdown if possible")Reported-by Dheeraj Kumar Srivastava <[email protected]>Signed-off-by Vasant Hegde <[email protected]>--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/smpboot.c +1
-1@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }
Quoted 9 lines · Vasant Hegde
Below commit is causing kexec to hang in certain scenarios with >255 CPUs.
Reproduce steps:
- We are using 2 socket system with 384 CPUs
- Booting first kernel with kernel command line intremap=off This disabled x2apic in kernel and booted with apic mode
- During kexec it tries to send INIT to all CPUs except boot CPU If APIC ID is0x100(like in our case) then it will send CPU0 to INIT mode and system hangs (in APIC mode DEST field is 8bit)
It took me a while to decode the above.
Quoted 2 lines
Fix this issue by adding
apic->apic_id_valid()check before sending INIT sequence.
Sigh, yes.
Quoted 21 lines
Fixes 45e34c8af58f ("x86/smp: Put CPUs into INIT on shutdown if possible")Reported-by Dheeraj Kumar Srivastava <[email protected]>Signed-off-by Vasant Hegde <[email protected]>--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kernel/smpboot.c +1
-1@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }
Quoted 14 lines · Thomas Gleixner
arch/x86/kernel/smpboot.c bool smp_park_other_cpus_in_init(void) · line 1366
Quoted 13 lines · Vasant Hegde
arch/x86/kernel/smpboot.c +1
-1@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }
I think this papers over the underlying problem that this sends INIT to an APIC which was never booted. The below is curing the root cause.
Thanks,
tglx
---
arch/x86/kernel/smpboot.c +1 -1
@@ -1356,7 +1356,7 @@ bool smp_park_other_cpus_in_init(void)if (this_cpu)return false; -for_each_present_cpu (cpu) { +for_each_cpu_and (cpu, &cpus_booted_once_mask, cpu_present_mask) {if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu);
Vasant Hegde
Tested-byHi,
Quoted 18 lines · Thomas Gleixner
Quoted 14 lines · Thomas Gleixner
arch/x86/kernel/smpboot.c bool smp_park_other_cpus_in_init(void) · line 1366
Quoted 13 lines · Vasant Hegde
arch/x86/kernel/smpboot.c +1
-1@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }I think this papers over the underlying problem that this sends INIT to an APIC which was never booted. The below is curing the root cause.
I have tested below patch and it fixes the issue. Thanks
-Vasant
Quoted 16 lines
Thanks,
tglx
---arch/x86/kernel/smpboot.c +1
-1@@ -1356,7 +1356,7 @@ bool smp_park_other_cpus_in_init(void)if (this_cpu)return false; -for_each_present_cpu (cpu) { +for_each_cpu_and (cpu, &cpus_booted_once_mask, cpu_present_mask) {if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu);
Hi Thomas,
Quoted 23 lines · Vasant Hegde
Hi,
Quoted 18 lines · Thomas Gleixner
Quoted 14 lines · Thomas Gleixner
arch/x86/kernel/smpboot.c bool smp_park_other_cpus_in_init(void) · line 1366
Quoted 13 lines · Vasant Hegde
arch/x86/kernel/smpboot.c +1
-1@@ -1360,7 +1360,7 @@ bool smp_park_other_cpus_in_init(void)if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu); -if (apicid ==BAD_APICID ) +if (apicid ==BAD_APICID || !apic->apic_id_valid (apicid))continue ;send_init_sequence (apicid); }I think this papers over the underlying problem that this sends INIT to an APIC which was never booted. The below is curing the root cause.
Quoted 3 lines
I have tested below patch and it fixes the issue. Thanks
Tested-by Vasant Hegde <[email protected]>
Ping.
Will you be posting/picking below patch -OR- Do you want me to resend below patch?
-Vasant
Quoted 19 lines
-Vasant
Quoted 16 lines
Thanks,
tglx
---arch/x86/kernel/smpboot.c +1
-1@@ -1356,7 +1356,7 @@ bool smp_park_other_cpus_in_init(void)if (this_cpu)return false; -for_each_present_cpu (cpu) { +for_each_cpu_and (cpu, &cpus_booted_once_mask, cpu_present_mask) {if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu);
[tip: x86/urgent] x86/smp: Don't send INIT to non-present and non-booted CPUs
In reply to Thomas GleixnerThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 3f874c9b2aae8e30463efc1872bea4baa9ed25dc Gitweb: https://git.kernel.org/tip/3f874c9b2aae8e30463efc1872bea4baa9ed25dc Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 09 Aug 2023 20:52:20 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 04 Sep 2023 15:41:42 +02:00
x86/smp: Don't send INIT to non-present and non-booted CPUs
Vasant reported that kexec() can hang or reset the machine when it tries to park CPUs via INIT. This happens when the kernel is using extended APIC, but the present mask has APIC IDs >= 0x100 enumerated.
As extended APIC can only handle 8 bit of APIC ID sending INIT to APIC ID 0x100 sends INIT to APIC ID 0x0. That's the boot CPU which is special on x86 and INIT causes the system to hang or resets the machine.
Prevent this by sending INIT only to those CPUs which have been booted once.
--- arch/x86/kernel/smpboot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/smpboot.c +1 -1
@@ -1250,7 +1250,7 @@ bool smp_park_other_cpus_in_init(void)if (this_cpu)return false; -for_each_present_cpu (cpu) { +for_each_cpu_and (cpu, &cpus_booted_once_mask, cpu_present_mask) {if (cpu == this_cpu)continue ; apicid = apic->cpu_present_to_apicid (cpu);