arm64: smp: distinguish secondary CPUs that hang after reaching head.S
PATCH 1/1 arm64: smp: distinguish secondary CPUs that hang after reaching head.S
When a secondary CPU fails to come online, __cpu_up() falls back to __early_cpu_boot_status, but boot status 0x0 is ambiguous: it cannot distinguish a CPU that never executed head.S (firmware/hypervisor never dispatched it, so it never ran a single instruction) from one that entered head.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.
Write a sentinel (CPU_BOOT_ASM_ENTERED, 0x10) as the first thing in secondary_entry so the two cases can be told apart:
0x0 - CPU never ran head.S
0x10 - CPU entered head.S but did not come online
__cpu_up() resets the word before releasing each secondary, so a stale value is not misattributed. Nothing clears the marker on the success path. Reason being, the CPU instead sets CPU_BOOT_SUCCESS in secondary_data.status, which __cpu_up() checks first, so __early_cpu_boot_status is only read on failure. A stall before CPU_BOOT_SUCCESS - including in secondary_start_kernel() - is therefore reported as 0x10. The sentinel sits in the low byte and sets no CPU_STUCK_REASON_* bits, so existing status decoding is unchanged.
Limitations: best-effort only. The status variable is global with no CPU identity, and spin-table secondaries boot via secondary_holding_pen and never write it, so 0x0 is a strong hint rather than a guarantee.
--- arch/arm64/include/asm/smp.h | 10 ++++++++++ arch/arm64/kernel/head.S | 24 +++++++++++++++++++++++- arch/arm64/kernel/smp.c | 21 +++++++++++++++++++++ 3 files changed, 54 insertions(+), 1 deletion(-)
arch/arm64/include/asm/smp.h +10 -0
@@ -20,6 +20,16 @@/* Fatal system error detected by secondary CPU, crash the system */ #define CPU_PANIC_KERNEL (3) +/* +* Best-effort marker written early in secondary_entry (PSCI/hotplug +* path) so __cpu_up() can tell whether a stuck CPU ever ran head.S. +* Not authoritative: spin-table secondaries use secondary_holding_pen +* and never write it, and the word has no CPU identity. Value must sit +* in the low byte, stay out of {0,1,2,3}, and set no CPU_STUCK_REASON_* +* bits. +*/ +#define CPU_BOOT_ASM_ENTERED (0x10) +#define CPU_STUCK_REASON_52_BIT_VA (UL(1) << CPU_STUCK_REASON_SHIFT)#define CPU_STUCK_REASON_NO_GRAN (UL(2) << CPU_STUCK_REASON_SHIFT)
arch/arm64/kernel/head.S +23 -1
@@ -353,6 +353,22 @@ SYM_FUNC_END(secondary_holding_pen) * be used where CPUs are brought online dynamically by the kernel. */ SYM_FUNC_START(secondary_entry) + /* + * Record that this CPU reached head.S, before init_kernel_el() + * sets SCTLR_ELx.EE. A single-byte store to the reader's low + * byte (__cpu_up() zeroed the word) is endian-agnostic, so a + * big-endian kernel still reads 0x10. Flush it like + * update_early_cpu_boot_status. + */ + adr_l x2, __early_cpu_boot_status + mov w1, #CPU_BOOT_ASM_ENTERED +#ifdef CONFIG_CPU_BIG_ENDIAN + strb w1, [x2, #7] // low byte at top address +#else + strb w1, [x2] +#endif + dmb sy + dc ivac, x2 mov x0, xzr bl init_kernel_el // w0=cpu_boot_mode b secondary_startup
@@ -386,7 +402,6 @@ SYM_FUNC_START_LOCAL(__secondary_switched) mov x0, x20 bl finalise_el2 - str_l xzr, __early_cpu_boot_status, x3 adr_l x5, vectors msr vbar_el1, x5 isb
@@ -395,6 +410,13 @@ SYM_FUNC_START_LOCAL(__secondary_switched) ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow + /* + * Don't clear the marker here; nothing on the secondary clears + * it. Once the CPU sets CPU_BOOT_SUCCESS in + * secondary_data.status, __cpu_up() stops reading + * __early_cpu_boot_status (and resets it before the next + * release), so a stall before that still reports 0x10. + */ init_cpu_task x2, x1, x3 #ifdef CONFIG_ARM64_PTR_AUTH
arch/arm64/kernel/smp.c +21 -0
@@ -120,6 +120,17 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) secondary_data.task = idle;update_cpu_boot_status (CPU_MMU_OFF ); +/* +* Reset the marker before releasing this secondary: head.S +* never clears it, so without this a CPU that never reaches +* head.S would inherit a stale 0x10 from a previous CPU. Flush +* it so the MMU-off secondary sees the reset. +*/ +WRITE_ONCE (__early_cpu_boot_status,0 ); +dcache_clean_inval_poc ((unsigned long )&__early_cpu_boot_status, + (unsigned long )&__early_cpu_boot_status + +sizeof (__early_cpu_boot_status)); +/* Now bring the CPU into our world */ ret =boot_secondary (cpu, idle);if (ret) {
@@ -145,10 +156,20 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)switch (status &CPU_BOOT_STATUS_MASK ) {default : +/* +* 0x0 usually means the CPU never ran head.S, but +* treat it as a hint: spin-table secondaries never +* write the marker and the word has no CPU identity. +*/ pr_err ("CPU%u: failed in unknown state : 0x%lx\n" , cpu, status); cpus_stuck_in_kernel++;break ; +case CPU_BOOT_ASM_ENTERED : +pr_err ("CPU%u: entered head.S but not online : 0x%lx\n" , + cpu, status); + cpus_stuck_in_kernel++; +break ;case CPU_KILL_ME :if (!op_cpu_kill (cpu)) {pr_crit ("CPU%u: died during early boot\n" , cpu);
base-commit: 290aaf24a551d5a0dce037e3fab30820f9113a10
Signature · 1 line
2.43.0
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 142 lines
Write a sentinel (
CPU_BOOT_ASM_ENTERED,0x10) as the first thing insecondary_entryso the two cases can be told apart:
0x0- CPU never ranhead.S
0x10- CPU enteredhead.Sbut did not come online
__cpu_up()resets the word before releasing each secondary, so a stale value is not misattributed. Nothing clears the marker on the success path. Reason being, the CPU instead setsCPU_BOOT_SUCCESSin secondary_data.status, which__cpu_up()checks first, so__early_cpu_boot_statusis only read on failure. A stall beforeCPU_BOOT_SUCCESS- including insecondary_start_kernel()- is therefore reported as0x10. The sentinel sits in the low byte and sets no CPU_STUCK_REASON_* bits, so existing status decoding is unchanged.Limitations: best-effort only. The status variable is global with no CPU identity, and spin-table secondaries boot via
secondary_holding_penand never write it, so0x0is a strong hint rather than a guarantee.Signed-off-by Naman Jain <[email protected]>--- arch/arm64/include/asm/smp.h | 10 ++++++++++ arch/arm64/kernel/head.S | 24 +++++++++++++++++++++++- arch/arm64/kernel/smp.c | 21 +++++++++++++++++++++ 3 files changed, 54 insertions(+), 1 deletion(-)arch/arm64/include/asm/smp.h +10
-0@@ -20,6 +20,16 @@/* Fatal system error detected by secondary CPU, crash the system */ #define CPU_PANIC_KERNEL (3) +/* +* Best-effort marker written early in secondary_entry (PSCI/hotplug +* path) so __cpu_up() can tell whether a stuck CPU ever ran head.S. +* Not authoritative: spin-table secondaries use secondary_holding_pen +* and never write it, and the word has no CPU identity. Value must sit +* in the low byte, stay out of {0,1,2,3}, and set no CPU_STUCK_REASON_* +* bits. +*/ +#define CPU_BOOT_ASM_ENTERED (0x10) +#define CPU_STUCK_REASON_52_BIT_VA (UL(1) << CPU_STUCK_REASON_SHIFT)#define CPU_STUCK_REASON_NO_GRAN (UL(2) << CPU_STUCK_REASON_SHIFT)arch/arm64/kernel/head.S +23
-1@@ -353,6 +353,22 @@ SYM_FUNC_END(secondary_holding_pen) * be used where CPUs are brought online dynamically by the kernel. */ SYM_FUNC_START(secondary_entry) + /* + * Record that this CPU reached head.S, before init_kernel_el() + * sets SCTLR_ELx.EE. A single-byte store to the reader's low + * byte (__cpu_up() zeroed the word) is endian-agnostic, so a + * big-endian kernel still reads 0x10. Flush it like + * update_early_cpu_boot_status. + */ + adr_l x2, __early_cpu_boot_status + mov w1, #CPU_BOOT_ASM_ENTERED +#ifdef CONFIG_CPU_BIG_ENDIAN + strb w1, [x2, #7] // low byte at top address +#else + strb w1, [x2] +#endif + dmb sy + dc ivac, x2 mov x0, xzr bl init_kernel_el // w0=cpu_boot_mode b secondary_startup@@ -386,7 +402,6 @@ SYM_FUNC_START_LOCAL(__secondary_switched) mov x0, x20 bl finalise_el2 - str_l xzr, __early_cpu_boot_status, x3 adr_l x5, vectors msr vbar_el1, x5 isb@@ -395,6 +410,13 @@ SYM_FUNC_START_LOCAL(__secondary_switched) ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow + /* + * Don't clear the marker here; nothing on the secondary clears + * it. Once the CPU sets CPU_BOOT_SUCCESS in + * secondary_data.status, __cpu_up() stops reading + * __early_cpu_boot_status (and resets it before the next + * release), so a stall before that still reports 0x10. + */ init_cpu_task x2, x1, x3 #ifdef CONFIG_ARM64_PTR_AUTHarch/arm64/kernel/smp.c +21
-0@@ -120,6 +120,17 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) secondary_data.task = idle;update_cpu_boot_status (CPU_MMU_OFF ); +/* +* Reset the marker before releasing this secondary: head.S +* never clears it, so without this a CPU that never reaches +* head.S would inherit a stale 0x10 from a previous CPU. Flush +* it so the MMU-off secondary sees the reset. +*/ +WRITE_ONCE (__early_cpu_boot_status,0 ); +dcache_clean_inval_poc ((unsigned long )&__early_cpu_boot_status, + (unsigned long )&__early_cpu_boot_status + +sizeof (__early_cpu_boot_status)); +/* Now bring the CPU into our world */ ret =boot_secondary (cpu, idle);if (ret) {@@ -145,10 +156,20 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)switch (status &CPU_BOOT_STATUS_MASK ) {default : +/* +* 0x0 usually means the CPU never ran head.S, but +* treat it as a hint: spin-table secondaries never +* write the marker and the word has no CPU identity. +*/ pr_err ("CPU%u: failed in unknown state : 0x%lx\n" , cpu, status); cpus_stuck_in_kernel++;break ; +case CPU_BOOT_ASM_ENTERED : +pr_err ("CPU%u: entered head.S but not online : 0x%lx\n" , + cpu, status); + cpus_stuck_in_kernel++; +break ;case CPU_KILL_ME :if (!op_cpu_kill (cpu)) {pr_crit ("CPU%u: died during early boot\n" , cpu);base-commit:
290aaf24a551d5a0dce037e3fab30820f9113a10
Quoted 9 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.
Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside head.S ? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.
Still wondering what's the rationale for this change.
Quoted 17 lines · Anshuman Khandual
Quoted 9 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 6 lines
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.Still wondering what's the rationale for this change.
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with 0x0 status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start of head.S and marking itself online at the end of secondary_start_kernel(). There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
Regards,
Naman
Quoted 25 lines · Naman Jain
Quoted 22 lines · Anshuman Khandual
Quoted 9 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.
Quoted 4 lines
Quoted 1 line
Still wondering what's the rationale for this change.
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().
Why that information is useful ? If firmware is sure to have dispatched given CPU then should not the early kernel boot failure be debugged via generally available methods. Is this trying create an alternative ?
Quoted 1 line
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
But wondering if the intent is to further classify failures including entered ASM, but failed else where during boot for ease in debugging why not add more into CPU_STUCK_REASON_* ?
Quoted 28 lines · Anshuman Khandual
Quoted 25 lines · Naman Jain
Quoted 22 lines · Anshuman Khandual
Quoted 9 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Still wondering what's the rationale for this change.
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().Why that information is useful ? If firmware is sure to have dispatched given CPU then should not the early kernel boot failure be debugged via generally available methods. Is this trying create an alternative ?
I am trying to cover the cases, where the kernel does not know that it is stuck, and can thus cannot handle that scenario. Existing CPU_STUCK_IN_KERNEL status with CPU_STUCK_REASON_* are applied only after the kernel detects that something failed.
Generally available methods, like adding breakpoints, debug logs and traces, is not always possible due to production system limitations and limited reproducibility of issues.
Quoted 4 lines
Quoted 1 line
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
But wondering if the intent is to further classify failures including entered ASM, but failed else where during boot for ease in debugging why not add more into CPU_STUCK_REASON_* ?
If we agree on the requirement for this, I can then reuse CPU_STUCK_IN_KERNEL with new CPU_STUCK_REASON_* variables, to proactively mark the CPU as STUCK first and then making checkpoints based on the boot-up progress, and finally changing the status to CPU_BOOT_SUCCESS as usual.
I would be adding these checkpoints only on a few logical blocks of secondary CPU boot flow to narrow down the problem.
Please let me know your thoughts on it.
Regards,
Naman
Will Deacon
maintainerQuoted 22 lines · Naman Jain
Quoted 14 lines · Anshuman Khandual
Quoted 8 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 6 lines
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.Still wondering what's the rationale for this change.
Quoted 12 lines
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
I think this is a game of diminishing returns. There's a lot of stuff that the firmware/hypervisor can get wrong here and trying to detect or handle that in Linux is going to be a real mess. For example, if it enters the kernel at the wrong address, or in the wrong mode, or with the MMU enabled etc. It sounds like you don't have much idea about what happens in the failure case, so it might not even execute the code that you're adding correctly.
It's also going to conflict heavily with the ongoing parallel bringup work, which significantly reworks this code.
I think you need to add the debug to your hypervisor, rather than the kernel.
Will
Quoted 36 lines · Will Deacon
Quoted 22 lines · Naman Jain
Quoted 14 lines · Anshuman Khandual
Quoted 8 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 6 lines
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.Still wondering what's the rationale for this change.
Quoted 12 lines
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.
Quoted 7 lines
I think this is a game of diminishing returns. There's a lot of stuff that the
firmware/hypervisorcan get wrong here and trying to detect or handle that in Linux is going to be a real mess. For example, if it enters the kernel at the wrong address, or in the wrong mode, or with the MMU enabled etc. It sounds like you don't have much idea about what happens in the failure case, so it might not even execute the code that you're adding correctly.
That is true. I agree. While we cannot and should not worry about adding logs for each of these firmware failure points in kernel, do you see any merit in adding any of this information to the kernel to at least narrow down the problem?
Or, can I safely consider 0x0 unknown error in CPU bring-up to be definitely a result of firmware/Hypervisor issues?
Regards,
Naman
Quoted 8 lines
It's also going to conflict heavily with the ongoing parallel bringup work, which significantly reworks this code.
I think you need to add the debug to your hypervisor, rather than the kernel.
Will
Will Deacon
maintainerQuoted 45 lines · Naman Jain
Quoted 36 lines · Will Deacon
Quoted 22 lines · Naman Jain
Quoted 14 lines · Anshuman Khandual
Quoted 8 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 6 lines
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.Still wondering what's the rationale for this change.
Quoted 12 lines
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.Quoted 7 lines
I think this is a game of diminishing returns. There's a lot of stuff that the
firmware/hypervisorcan get wrong here and trying to detect or handle that in Linux is going to be a real mess. For example, if it enters the kernel at the wrong address, or in the wrong mode, or with the MMU enabled etc. It sounds like you don't have much idea about what happens in the failure case, so it might not even execute the code that you're adding correctly.
Quoted 4 lines
That is true. I agree. While we cannot and should not worry about adding logs for each of these firmware failure points in kernel, do you see any merit in adding any of this information to the kernel to at least narrow down the problem?
Possibly, but I'd prefer to land the parallel bringup work first and then see whether this still makes sense.
Will
Quoted 51 lines · Will Deacon
Quoted 45 lines · Naman Jain
Quoted 36 lines · Will Deacon
Quoted 22 lines · Naman Jain
Quoted 14 lines · Anshuman Khandual
Quoted 8 lines · Jinjie Ruan
Quoted 7 lines · Naman Jain
When a secondary CPU fails to come online,
__cpu_up()falls back to__early_cpu_boot_status, but boot status0x0is ambiguous: it cannot distinguish a CPU that never executedhead.S(firmware/hypervisornever dispatched it, so it never ran a single instruction) from one that enteredhead.S, started executing, and then got stuck somewhere in kernel bring-up. Add a change to let us tell those two cases apart, which narrows down where to look when a CPU goes missing during boot.Quoted 4 lines
I previously encountered this issue when debugging the parallel startup of ARM64 secondary cores. It is difficult for the kernel to determine whether the secondary core is hung in the firmware or whether it has not executed a single instruction. So I think this motive is reasonable.
Quoted 6 lines
Why should kernel determine the difference here ? Would not the firmware know if it has started any secondary CPU for the kernel which must have come inside
head.S? If the cpu gets hung inside firmware while starting up then the debug responsibilities belong there instead.Still wondering what's the rationale for this change.
Quoted 12 lines
Hello Anshuman,
This sounds fair to me. Let me elaborate the problem, beyond the scope of this patch. In production, we occasionally see these crashes where one of the CPU fails to bring up online, with0x0status code. Hypervisor may be missing the telemetry, but the problem is that we don't know if the secondary CPU ever started executing the instructions or is stuck somewhere between the start ofhead.Sand marking itself online at the end ofsecondary_start_kernel().
There are couple of places, where we get those other status codes, but not everywhere. If the issue is not easily reproducible, experiments on local setups do not yield anything. That's where I am attempting to add some more information in kernel to debug these issues.Quoted 7 lines
I think this is a game of diminishing returns. There's a lot of stuff that the
firmware/hypervisorcan get wrong here and trying to detect or handle that in Linux is going to be a real mess. For example, if it enters the kernel at the wrong address, or in the wrong mode, or with the MMU enabled etc. It sounds like you don't have much idea about what happens in the failure case, so it might not even execute the code that you're adding correctly.Quoted 4 lines
That is true. I agree. While we cannot and should not worry about adding logs for each of these firmware failure points in kernel, do you see any merit in adding any of this information to the kernel to at least narrow down the problem?
Quoted 4 lines
Possibly, but I'd prefer to land the parallel bringup work first and then see whether this still makes sense.
Will
Acked.
Regards,
Naman