powerpc/970: fix nap return address corruption on async interrupt exit
Subsystems (2)
PATCH 1/1 powerpc/970: fix nap return address corruption on async interrupt exit
On PowerMac G5 (PPC970, CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.
The PPC970 idle path works by setting _TLF_NAPPING in the current thread's local flags before entering the MSR_POW nap loop. When any async interrupt wakes the CPU, nap_adjust_return() is expected to detect _TLF_NAPPING, clear it, and rewrite regs->NIP to power4_idle_nap_return so that the interrupt returns cleanly to the caller of power4_idle_nap() rather than back into the nap spin loop.
DEFINE_INTERRUPT_HANDLER_ASYNC generates the following sequence:
irq_enter_rcu();
____func(regs); /* timer_interrupt / do_IRQ body */
irq_exit_rcu(); /* softirqs run here, irqs re-enabled */
arch_interrupt_async_exit_prepare(regs); /* nap_adjust_return was here */
irqentry_exit(regs, state);irq_exit_rcu() calls invoke_softirq() -> do_softirq_own_stack(), which runs softirqs with hardware interrupts re-enabled. A nested async interrupt can therefore arrive while _TLF_NAPPING is still set. That nested interrupt reaches nap_adjust_return() in its own arch_interrupt_async_exit_prepare() call, finds _TLF_NAPPING set, and redirects *its own* regs->NIP to power4_idle_nap_return. Returning via that blr with an unrelated LR on the softirq stack jumps to a garbage address, causing the observed crashes.
The comment that previously lived in arch_interrupt_async_exit_prepare() even described this exact hazard ("must come before irq_exit()"), but nap_adjust_return() was placed after irq_exit_rcu() in the macro, so the protection was never effective.
Fix this by calling nap_adjust_return() inside DEFINE_INTERRUPT_HANDLER_ASYNC immediately before irq_exit_rcu(), ensuring _TLF_NAPPING is cleared and regs->NIP is adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment into nap_adjust_return() itself and remove it from arch_interrupt_async_exit_prepare().
--- arch/powerpc/include/asm/entry-common.h | 15 +++++++-------- arch/powerpc/include/asm/interrupt.h | 1 + 2 files changed, 8 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/entry-common.h +7 -8
@@ -66,6 +66,13 @@ static inline void srr_regs_clobbered(void)static inline void nap_adjust_return (struct pt_regs *regs) {#ifdef CONFIG_PPC_970_NAP +/* +* Adjust the nap return address before irq_exit_rcu(). irq_exit_rcu() +* may invoke softirqs with interrupts re-enabled, allowing a nested +* async interrupt to arrive. If _TLF_NAPPING is still set at that +* point, the nested interrupt would erroneously redirect its own +* return address to power4_idle_nap_return, corrupting the stack. +*/ if (unlikely (test_thread_local_flags (_TLF_NAPPING))) {/* Can avoid a test-and-clear because NMIs do not call this */ clear_thread_local_flags (_TLF_NAPPING);
@@ -286,14 +293,6 @@ static inline void arch_interrupt_async_enter_prepare(struct pt_regs *regs)static inline void arch_interrupt_async_exit_prepare (struct pt_regs *regs) { -/* -* Adjust at exit so the main handler sees the true NIA. This must -* come before irq_exit() because irq_exit can enable interrupts, and -* if another interrupt is taken before nap_adjust_return has run -* here, then that interrupt would return directly to idle nap return. -*/ -nap_adjust_return (regs); -arch_interrupt_exit_prepare (regs); }
arch/powerpc/include/asm/interrupt.h +1 -0
@@ -246,6 +246,7 @@ interrupt_handler void func(struct pt_regs *regs) \instrumentation_begin (); \irq_enter_rcu (); \ ____##func (regs); \ + nap_adjust_return(regs); \ irq_exit_rcu(); \ instrumentation_end(); \ arch_interrupt_async_exit_prepare(regs); \
Signature · 1 line
2.55.0
Shrikanth Hegde
reviewerHi Mukesh.
Quoted 71 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.The PPC970 idle path works by setting
_TLF_NAPPINGin the current thread's local flags before entering theMSR_POWnap loop. When any async interrupt wakes the CPU,nap_adjust_return()is expected to detect_TLF_NAPPING, clear it, and rewriteregs->NIPtopower4_idle_nap_returnso that the interrupt returns cleanly to the caller ofpower4_idle_nap()rather than back into the nap spin loop.
DEFINE_INTERRUPT_HANDLER_ASYNCgenerates the following sequence:irq_enter_rcu(); ____func(regs); /* timer_interrupt / do_IRQ body */ irq_exit_rcu(); /* softirqs run here, irqs re-enabled */ arch_interrupt_async_exit_prepare(regs); /* nap_adjust_return was here */ irqentry_exit(regs, state);
irq_exit_rcu()callsinvoke_softirq()->do_softirq_own_stack(), which runs softirqs with hardware interrupts re-enabled. A nested async interrupt can therefore arrive while_TLF_NAPPINGis still set. That nested interrupt reachesnap_adjust_return()in its ownarch_interrupt_async_exit_prepare()call, finds_TLF_NAPPINGset, and redirects *its own*regs->NIPtopower4_idle_nap_return. Returning via that blr with an unrelated LR on the softirq stack jumps to a garbage address, causing the observed crashes.The comment that previously lived in
arch_interrupt_async_exit_prepare()even described this exact hazard ("must come beforeirq_exit()"), butnap_adjust_return()was placed afterirq_exit_rcu()in the macro, so the protection was never effective.Fix this by calling
nap_adjust_return()insideDEFINE_INTERRUPT_HANDLER_ASYNCimmediately beforeirq_exit_rcu(), ensuring_TLF_NAPPINGis cleared andregs->NIPis adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment intonap_adjust_return()itself and remove it fromarch_interrupt_async_exit_prepare().Fixes bee25f97ad24 ("powerpc: Enable GENERIC_ENTRY feature")Reported-by Andreas Schwab <[email protected]>Signed-off-by Mukesh Kumar Chaurasiya (IBM) <[email protected]>--- arch/powerpc/include/asm/entry-common.h | 15 +++++++-------- arch/powerpc/include/asm/interrupt.h | 1 + 2 files changed, 8 insertions(+), 8 deletions(-)arch/powerpc/include/asm/entry-common.h +7
-0@@ -66,6 +66,13 @@ static inline void srr_regs_clobbered(void)static inline void nap_adjust_return (struct pt_regs *regs) {#ifdef CONFIG_PPC_970_NAP +/* +* Adjust the nap return address before irq_exit_rcu(). irq_exit_rcu() +* may invoke softirqs with interrupts re-enabled, allowing a nested +* async interrupt to arrive. If _TLF_NAPPING is still set at that +* point, the nested interrupt would erroneously redirect its own +* return address to power4_idle_nap_return, corrupting the stack. +*/ if (unlikely (test_thread_local_flags (_TLF_NAPPING))) {/* Can avoid a test-and-clear because NMIs do not call this */ clear_thread_local_flags (_TLF_NAPPING);@@ -286,14 +293,6 @@ static inline void arch_interrupt_async_enter_prepare(struct pt_regs *regs)static inline void arch_interrupt_async_exit_prepare (struct pt_regs *regs) {
arch/powerpc/include/asm/entry-common.h static inline void arch_interrupt_async_enter_prepare(struct pt_regs *regs) · line 297
Quoted 10 lines
arch/powerpc/include/asm/entry-common.h +0
-8-/* -* Adjust at exit so the main handler sees the true NIA. This must -* come before irq_exit() because irq_exit can enable interrupts, and -* if another interrupt is taken before nap_adjust_return has run -* here, then that interrupt would return directly to idle nap return. -*/ -nap_adjust_return (regs); -arch_interrupt_exit_prepare (regs); }
This makes arch_interrupt_async_exit_prepare same as arch_interrupt_exit_prepare. Maybe remove arch_interrupt_async_exit_prepare?
Quoted 13 lines
arch/powerpc/include/asm/interrupt.h +1
-0@@ -246,6 +246,7 @@ interrupt_handler void func(struct pt_regs *regs) \instrumentation_begin (); \irq_enter_rcu (); \ ____##func (regs); \ + nap_adjust_return(regs); \ irq_exit_rcu(); \ instrumentation_end(); \ arch_interrupt_async_exit_prepare(regs); \
Quoted 3 lines · Shrikanth Hegde
Hi Mukesh.
On 7/7/26 10:54 PM, Mukesh Kumar Chaurasiya (IBM) wrote:
<... snip ...>
Quoted 10 lines
Quoted 10 lines
/* Can avoid a test-and-clear because NMIs do not call this */
clear_thread_local_flags(_TLF_NAPPING);arch/powerpc/include/asm/entry-common.h +0
-5@@ -286,14 +293,6 @@ static inline void arch_interrupt_async_enter_prepare(struct pt_regs *regs)static inline void arch_interrupt_async_exit_prepare (struct pt_regs *regs) { -/* -* Adjust at exit so the main handler sees the true NIA. This must -* come before irq_exit() because irq_exit can enable interrupts, and -* if another interrupt is taken before nap_adjust_return has run -* here, then that interrupt would return directly to idle nap return.
Quoted 9 lines
arch/powerpc/include/asm/entry-common.h static inline void arch_interrupt_async_enter_prepare(struct pt_regs *regs) · line 297
Quoted 5 lines
arch/powerpc/include/asm/entry-common.h +0
-3-*/ -nap_adjust_return (regs); -arch_interrupt_exit_prepare (regs); }This makes
arch_interrupt_async_exit_preparesame asarch_interrupt_exit_prepare. Maybe removearch_interrupt_async_exit_prepare?
Hey,
arch_interrupt_async_exit_prepare and arch_interrupt_exit_prepare, both are marked static inline. So the generated assembly will have no impact. I kept it this way coz it was consistent with how we implemented other macros like DEFINE_INTERRUPT_HANDLER.
Regards,
Mukesh
Quoted 13 lines
Quoted 12 lines
arch/powerpc/include/asm/interrupt.h +1
-0@@ -246,6 +246,7 @@ interrupt_handler void func(struct pt_regs *regs) \instrumentation_begin (); \irq_enter_rcu (); \ ____##func (regs); \ + nap_adjust_return(regs); \ irq_exit_rcu(); \ instrumentation_end(); \ arch_interrupt_async_exit_prepare(regs); \
Quoted 40 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.The PPC970 idle path works by setting
_TLF_NAPPINGin the current thread's local flags before entering theMSR_POWnap loop. When any async interrupt wakes the CPU,nap_adjust_return()is expected to detect_TLF_NAPPING, clear it, and rewriteregs->NIPtopower4_idle_nap_returnso that the interrupt returns cleanly to the caller ofpower4_idle_nap()rather than back into the nap spin loop.
DEFINE_INTERRUPT_HANDLER_ASYNCgenerates the following sequence:irq_enter_rcu(); ____func(regs); /* timer_interrupt / do_IRQ body */ irq_exit_rcu(); /* softirqs run here, irqs re-enabled */ arch_interrupt_async_exit_prepare(regs); /* nap_adjust_return was here */ irqentry_exit(regs, state);
irq_exit_rcu()callsinvoke_softirq()->do_softirq_own_stack(), which runs softirqs with hardware interrupts re-enabled. A nested async interrupt can therefore arrive while_TLF_NAPPINGis still set. That nested interrupt reachesnap_adjust_return()in its ownarch_interrupt_async_exit_prepare()call, finds_TLF_NAPPINGset, and redirects *its own*regs->NIPtopower4_idle_nap_return. Returning via that blr with an unrelated LR on the softirq stack jumps to a garbage address, causing the observed crashes.The comment that previously lived in
arch_interrupt_async_exit_prepare()even described this exact hazard ("must come beforeirq_exit()"), butnap_adjust_return()was placed afterirq_exit_rcu()in the macro, so the protection was never effective.Fix this by calling
nap_adjust_return()insideDEFINE_INTERRUPT_HANDLER_ASYNCimmediately beforeirq_exit_rcu(), ensuring_TLF_NAPPINGis cleared andregs->NIPis adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment intonap_adjust_return()itself and remove it fromarch_interrupt_async_exit_prepare().
Quoted 4 lines
Fixes bee25f97ad24 ("powerpc: Enable GENERIC_ENTRY feature")Reported-by Andreas Schwab <[email protected]>Signed-off-by Mukesh Kumar Chaurasiya (IBM) <[email protected]>
Would be nice to see this in 7.2-rc4
Quoted 41 lines · John Ogness
Quoted 40 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.The PPC970 idle path works by setting
_TLF_NAPPINGin the current thread's local flags before entering theMSR_POWnap loop. When any async interrupt wakes the CPU,nap_adjust_return()is expected to detect_TLF_NAPPING, clear it, and rewriteregs->NIPtopower4_idle_nap_returnso that the interrupt returns cleanly to the caller ofpower4_idle_nap()rather than back into the nap spin loop.
DEFINE_INTERRUPT_HANDLER_ASYNCgenerates the following sequence:irq_enter_rcu(); ____func(regs); /* timer_interrupt / do_IRQ body */ irq_exit_rcu(); /* softirqs run here, irqs re-enabled */ arch_interrupt_async_exit_prepare(regs); /* nap_adjust_return was here */ irqentry_exit(regs, state);
irq_exit_rcu()callsinvoke_softirq()->do_softirq_own_stack(), which runs softirqs with hardware interrupts re-enabled. A nested async interrupt can therefore arrive while_TLF_NAPPINGis still set. That nested interrupt reachesnap_adjust_return()in its ownarch_interrupt_async_exit_prepare()call, finds_TLF_NAPPINGset, and redirects *its own*regs->NIPtopower4_idle_nap_return. Returning via that blr with an unrelated LR on the softirq stack jumps to a garbage address, causing the observed crashes.The comment that previously lived in
arch_interrupt_async_exit_prepare()even described this exact hazard ("must come beforeirq_exit()"), butnap_adjust_return()was placed afterirq_exit_rcu()in the macro, so the protection was never effective.Fix this by calling
nap_adjust_return()insideDEFINE_INTERRUPT_HANDLER_ASYNCimmediately beforeirq_exit_rcu(), ensuring_TLF_NAPPINGis cleared andregs->NIPis adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment intonap_adjust_return()itself and remove it fromarch_interrupt_async_exit_prepare().
Quoted 6 lines
Quoted 4 lines
Fixes bee25f97ad24 ("powerpc: Enable GENERIC_ENTRY feature")Reported-by Andreas Schwab <[email protected]>Signed-off-by Mukesh Kumar Chaurasiya (IBM) <[email protected]>Tested-by John Ogness <[email protected]>
Is there a reason this is not being queued for 7.2? Without this fix, PPC970 machines cannot boot 7.2.
John Ogness
Quoted 18 lines · John Ogness
Quoted 17 lines · John Ogness
Quoted 15 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.
[...]
Fix this by callingnap_adjust_return()insideDEFINE_INTERRUPT_HANDLER_ASYNCimmediately beforeirq_exit_rcu(), ensuring_TLF_NAPPINGis cleared andregs->NIPis adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment intonap_adjust_return()itself and remove it fromarch_interrupt_async_exit_prepare().Fixes bee25f97ad24 ("powerpc: Enable GENERIC_ENTRY feature")Reported-by Andreas Schwab <[email protected]>Signed-off-by Mukesh Kumar Chaurasiya (IBM) <[email protected]>
Quoted 4 lines
Quoted 1 line
Tested-by John Ogness <[email protected]>Is there a reason this is not being queued for 7.2? Without this fix, PPC970 machines cannot boot 7.2.
Wondering the same. Quick reminder: Linus afaik wants all regressions ideally be fixed by -rc6. So it would be good to get this mainlined within the next few days -- unless of course there is a reason why this fix is a bad idea. Is there?
Ciao, Thorsten
Madhavan Srinivasan
maintainerQuoted 18 lines · Thorsten Leemhuis
Quoted 17 lines · John Ogness
Quoted 16 lines · John Ogness
Quoted 15 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.
[...]
Fix this by callingnap_adjust_return()insideDEFINE_INTERRUPT_HANDLER_ASYNCimmediately beforeirq_exit_rcu(), ensuring_TLF_NAPPINGis cleared andregs->NIPis adjusted before any code that can re-enable interrupts or invoke softirqs runs. Move the explanatory comment intonap_adjust_return()itself and remove it fromarch_interrupt_async_exit_prepare().Fixes bee25f97ad24 ("powerpc: Enable GENERIC_ENTRY feature")Reported-by Andreas Schwab <[email protected]>Signed-off-by Mukesh Kumar Chaurasiya (IBM) <[email protected]>
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Tested-by John Ogness <[email protected]>Is there a reason this is not being queued for 7.2? Without this fix, PPC970 machines cannot boot 7.2.
Wondering the same. Quick reminder: Linus afaik wants all regressions ideally be fixed by -rc6. So it would be good to get this mainlined within the next few days -- unless of course there is a reason why this fix is a bad idea. Is there?
I have added this to my fixes-test and will pull this as part of upcoming fixes PR Maddy
Quoted 2 lines
Ciao, Thorsten
Quoted 5 lines · Mukesh Kumar Chaurasiya (IBM)
On PowerMac G5 (PPC970,
CONFIG_PPC_970_NAP) the system panics shortly after boot with symptoms including instruction fetch faults, kernel data access faults, and stack corruption, predominantly on SMP and always somewhere inside softirq processing.
Quoted 8 lines
The PPC970 idle path works by setting
_TLF_NAPPINGin the current thread's local flags before entering theMSR_POWnap loop. When any async interrupt wakes the CPU,nap_adjust_return()is expected to detect_TLF_NAPPING, clear it, and rewriteregs->NIPtopower4_idle_nap_returnso that the interrupt returns cleanly to the caller ofpower4_idle_nap()rather than back into the nap spin loop.[...]
Applied to powerpc/fixes.
[1/1] powerpc/970: fix nap return address corruption on async interrupt exit
https://git.kernel.org/powerpc/c/bddf7540099bf653eaea339e886add6f62555cf3
cheers