linux v5.18.3 fails to boot
[...]
ata3: SATA link down (SStatus 0 SControl 0)
_______________________________
< Your System ate a SPARC! Gah! >
-------------------------------
\ ^__^
(__)\ )\/\
U ||----w |
|| ||
swapper/0 (pid 0): Illegal instruction (code 8)
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000
YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00001000000001000001101100001110 Not tainted
r00-03 0000000008041b0e 000000004e8045b0 0000000010bbda78 000000004e804470 r04-07 0000000010bb5000 0000000000001440 0000000054355000 0000000000001400 r08-11 0000000055000000 000000000000000e 000000000000000f 0000000055002800 r12-15 0000000000000000 0000000055002800 0000000040b668c0 0000000040b668c0 r16-19 0000000000000001 0000000000000001 0000000051b799f0 0000000000000000 r20-23 0000000000000001 fffffffffffff5b9 0000000000000000 0000000000200000 r24-27 000000000000000c 000000000800000e 0000000054355144 0000000040b3e0c0 r28-31 0000000000010001 000000004e804620 000000004e8045b0 0000000040edd040 sr00-03 00000000000a5c00 0000000000000000 0000000000000000 00000000000a5c00 sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
IASQ: 0000000000000000 0000000000000000 IAOQ: 0000000010bbd9e0 0000000010bbd9e4
IIR: 006110c2 ISR: 0000000010240000 IOR: 00000003b76dd048 CPU: 0 CR30: 0000000040edd040 CR31: ffffffffffffffff
ORIG_R28: 0000000000000000
IAOQ[0]: mpt_reply+0x130/0x4f0 [mptbase]
IAOQ[1]: mpt_reply+0x134/0x4f0 [mptbase]
RP(r2): mpt_reply+0x1c8/0x4f0 [mptbase]
Backtrace: [<0000000010bbde24>] mpt_interrupt+0x84/0xe8 [mptbase] [<000000004026dd64>] __handle_irq_event_percpu+0xc4/0x250 [<000000004026df28>] handle_irq_event_percpu+0x38/0xd8 [<00000000402776c4>] handle_percpu_irq+0xb4/0xf0 [<000000004026c90c>] generic_handle_irq+0x5c/0x90 [<00000000401a20e4>] call_on_stack+0x18/0x24 CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000 Backtrace: [<00000000401a8cd8>] show_stack+0x70/0x90 [<0000000040a8e238>] dump_stack_lvl+0xd8/0x128 [<0000000040a8e2bc>] dump_stack+0x34/0x48 [<00000000401a8efc>] die_if_kernel+0x1e4/0x3f8 [<00000000401a9af4>] handle_interruption+0x59c/0xb58 [<00000000401a107c>] intr_check_sig+0x0/0x3c Kernel panic - not syncing: Fatal exception in interrupt
v5.18.2 with similar config is okay. The fault seems consistent. IIR contains illegal instruction.
Attached config.
Dave
Signature · 1 line
John David Anglin [email protected]
I did a regression search. e039c0b5985999b150594126225e1ee51df7b4c9 is the first bad commit
commit e039c0b5985999b150594126225e1ee51df7b4c9
Author: Kuniyuki Iwashima <[email protected]>
Date: Fri Apr 29 14:38:01 2022 -0700
list: fix a data-race around ep->rdllist
[ Upstream commit d679ae94fdd5d3ab00c35078f5af5f37e068b03d ]
ep_poll() first calls ep_events_available() with no lock held and checks if ep->rdllist is empty by list_empty_careful(), which reads rdllist->prev. Thus all accesses to it need some protection to avoid store/load-tearing.
Note INIT_LIST_HEAD_RCU() already has the annotation for both prev and next.
Commit bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.") added the first lockless ep_events_available(), and commit c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()") made some ep_events_available() calls lockless and added single call under a lock, finally commit e59d3c64cba6 ("epoll: eliminate unnecessary lock for zero timeout") made the last ep_events_available() lockless.
BUG: KCSAN: data-race in do_epoll_wait / do_epoll_wait
write to 0xffff88810480c7d8 of 8 bytes by task 1802 on cpu 0:
INIT_LIST_HEAD include/linux/list.h:38 [inline] list_splice_init include/linux/list.h:492 [inline] ep_start_scan fs/eventpoll.c:622 [inline] ep_send_events fs/eventpoll.c:1656 [inline] ep_poll fs/eventpoll.c:1806 [inline] do_epoll_wait+0x4eb/0xf40 fs/eventpoll.c:2234 do_epoll_pwait fs/eventpoll.c:2268 [inline] __do_sys_epoll_pwait fs/eventpoll.c:2281 [inline] __se_sys_epoll_pwait+0x12b/0x240 fs/eventpoll.c:2275 __x64_sys_epoll_pwait+0x74/0x80 fs/eventpoll.c:2275 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x44/0xd0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae
read to 0xffff88810480c7d8 of 8 bytes by task 1799 on cpu 1:
list_empty_careful include/linux/list.h:329 [inline] ep_events_available fs/eventpoll.c:381 [inline] ep_poll fs/eventpoll.c:1797 [inline] do_epoll_wait+0x279/0xf40 fs/eventpoll.c:2234 do_epoll_pwait fs/eventpoll.c:2268 [inline] __do_sys_epoll_pwait fs/eventpoll.c:2281 [inline] __se_sys_epoll_pwait+0x12b/0x240 fs/eventpoll.c:2275 __x64_sys_epoll_pwait+0x74/0x80 fs/eventpoll.c:2275 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x44/0xd0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae
value changed: 0xffff88810480c7d0 -> 0xffff888103c15098
Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 1799 Comm: syz-fuzzer Tainted: G W 5.17.0-rc7-syzkaller-dirty #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
include/linux/list.h | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
Reverting above change fixes v5.18.3 boot.
It seems the most significant byte of the "ldd -10(r3),rp" instruction in mpt_reply has been set to 0:
4084: bf 80 21 18 cmpb,*<> r0,ret0,4118 <mpt_reply+0x1b8> 4088: 08 04 02 5b copy r4,dp
408c: 00 00 04 00 sync
4090: 0c 61 10 c2 ldd -10(r3),rp
See IIR value in crash output.
Quoted 59 lines · John David Anglin
[...]
ata3: SATA link down (SStatus 0 SControl 0)
_______________________________
< Your System ate a SPARC! Gah! >
-------------------------------
\ ^__^
(__)\ )\/\
U ||----w |
|| ||
swapper/0 (pid 0): Illegal instruction (code 8)CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00001000000001000001101100001110 Not tainted
r00-030000000008041b0e000000004e8045b00000000010bbda78000000004e804470r04-070000000010bb50000000000000001440 0000000054355000 0000000000001400 r08-11 0000000055000000000000000000000e000000000000000f0000000055002800 r12-15 0000000000000000 00000000550028000000000040b668c00000000040b668c0r16-19 0000000000000001 00000000000000010000000051b799f00000000000000000 r20-23 0000000000000001fffffffffffff5b90000000000000000 0000000000200000 r24-27000000000000000c000000000800000e00000000543551440000000040b3e0c0r28-31 0000000000010001000000004e804620000000004e8045b00000000040edd040sr00-0300000000000a5c000000000000000000 000000000000000000000000000a5c00sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000IASQ: 0000000000000000 0000000000000000 IAOQ:
0000000010bbd9e00000000010bbd9e4
IIR: 006110c2 ISR: 0000000010240000 IOR:00000003b76dd048CPU: 0 CR30:0000000040edd040CR31: ffffffffffffffff
ORIG_R28: 0000000000000000
IAOQ[0]:mpt_reply+0x130/0x4f0[mptbase]
IAOQ[1]:mpt_reply+0x134/0x4f0[mptbase]
RP(r2):mpt_reply+0x1c8/0x4f0[mptbase]Backtrace: [<0000000010bbde24>] mpt_interrupt+0x84/0xe8 [mptbase] [<000000004026dd64>] __handle_irq_event_percpu+0xc4/0x250 [<000000004026df28>] handle_irq_event_percpu+0x38/0xd8 [<00000000402776c4>] handle_percpu_irq+0xb4/0xf0 [<000000004026c90c>] generic_handle_irq+0x5c/0x90 [<00000000401a20e4>] call_on_stack+0x18/0x24 CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000 Backtrace: [<00000000401a8cd8>] show_stack+0x70/0x90 [<0000000040a8e238>] dump_stack_lvl+0xd8/0x128 [<0000000040a8e2bc>] dump_stack+0x34/0x48 [<00000000401a8efc>] die_if_kernel+0x1e4/0x3f8 [<00000000401a9af4>] handle_interruption+0x59c/0xb58 [<00000000401a107c>] intr_check_sig+0x0/0x3c Kernel panic - not syncing: Fatal exception in interruptv5.18.2 with similar config is okay. The fault seems consistent. IIR contains illegal instruction.
Attached config.
Dave
Signature · 1 line
John David Anglin [email protected]
Hello,
Thanks for heads up!
Date: Fri, 10 Jun 2022 11:06:24 -0400 From: John David Anglin <[email protected]>
Quoted 85 lines
I did a regression search.
e039c0b5985999b150594126225e1ee51df7b4c9is the first bad commitcommit
e039c0b5985999b150594126225e1ee51df7b4c9Author: Kuniyuki Iwashima <[email protected]> Date: Fri Apr 29 14:38:01 2022 -0700list: fix a data-race around
ep->rdllist[ Upstream commit
d679ae94fdd5d3ab00c35078f5af5f37e068b03d]
ep_poll()first callsep_events_available()with no lock held and checks ifep->rdllistis empty bylist_empty_careful(), which readsrdllist->prev. Thus all accesses to it need some protection to avoid store/load-tearing.Note
INIT_LIST_HEAD_RCU()already has the annotation for both prev and next.Commit
bf3b9f6372c4("epoll: Add busy poll support to epoll with socket fds.") added the first locklessep_events_available(), and commitc5a282e9635e("fs/epoll: reduce the scope of wq lock inepoll_wait()") made someep_events_available()calls lockless and added single call under a lock, finally commite59d3c64cba6("epoll: eliminate unnecessary lock for zero timeout") made the lastep_events_available()lockless.BUG: KCSAN: data-race in
do_epoll_wait/do_epoll_waitwrite to
0xffff88810480c7d8of 8 bytes by task 1802 on cpu 0:
INIT_LIST_HEADinclude/linux/list.h:38[inline]list_splice_initinclude/linux/list.h:492[inline]ep_start_scanfs/eventpoll.c:622[inline]ep_send_eventsfs/eventpoll.c:1656[inline]ep_pollfs/eventpoll.c:1806[inline]do_epoll_wait+0x4eb/0xf40fs/eventpoll.c:2234do_epoll_pwaitfs/eventpoll.c:2268[inline]__do_sys_epoll_pwaitfs/eventpoll.c:2281[inline]__se_sys_epoll_pwait+0x12b/0x240fs/eventpoll.c:2275__x64_sys_epoll_pwait+0x74/0x80fs/eventpoll.c:2275do_syscall_x64arch/x86/entry/common.c:50[inline]do_syscall_64+0x44/0xd0arch/x86/entry/common.c:80entry_SYSCALL_64_after_hwframe+0x44/0xaeread to
0xffff88810480c7d8of 8 bytes by task 1799 on cpu 1:
list_empty_carefulinclude/linux/list.h:329[inline]ep_events_availablefs/eventpoll.c:381[inline]ep_pollfs/eventpoll.c:1797[inline]do_epoll_wait+0x279/0xf40fs/eventpoll.c:2234do_epoll_pwaitfs/eventpoll.c:2268[inline]__do_sys_epoll_pwaitfs/eventpoll.c:2281[inline]__se_sys_epoll_pwait+0x12b/0x240fs/eventpoll.c:2275__x64_sys_epoll_pwait+0x74/0x80fs/eventpoll.c:2275do_syscall_x64arch/x86/entry/common.c:50[inline]do_syscall_64+0x44/0xd0arch/x86/entry/common.c:80entry_SYSCALL_64_after_hwframe+0x44/0xaevalue changed:
0xffff88810480c7d0->0xffff888103c15098Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 1799 Comm: syz-fuzzer Tainted: G W 5.17.0-rc7-syzkaller-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011Fixes e59d3c64cba6 ("epoll: eliminate unnecessary lock for zero timeout")Fixes c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()")Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Kuniyuki Iwashima <[email protected]>Reported-by [email protected]Cc Al Viro <[email protected]>, Andrew Morton <[email protected]>Cc Kuniyuki Iwashima <[email protected]>Cc Kuniyuki Iwashima <[email protected]>Cc "Soheil Hassas Yeganeh" <[email protected]>Cc Davidlohr Bueso <[email protected]>Cc "Sridhar Samudrala" <[email protected]>Cc Alexander Duyck <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Sasha Levin <[email protected]>include/linux/list.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)Reverting above change fixes v5.18.3 boot.
It seems the most significant byte of the "ldd -10(r3),rp" instruction in
mpt_replyhas been set to 0:
Quoted 6 lines
4084: bf 80 21 18 cmpb,*<> r0,ret0,4118 <mpt_reply+0x1b8> 4088: 08 04 02 5b copy r4,dp 408c: 00 00 04 00 sync 4090: 0c 61 10 c2 ldd -10(r3),rpSee IIR value in crash output.
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
I'm not familiar with PARISC and this may be a stupid question though, what does ldd exactly do? and which line is it executed in the func/file?
Quoted 42 lines · John David Anglin
Quoted 41 lines
[...]
ata3: SATA link down (SStatus 0 SControl 0)_______________________________ < Your System ate a SPARC! Gah! > ------------------------------- \ ^__^ (__)\ )\/\ U ||----w | || || swapper/0 (pid 0): Illegal instruction (code 8) CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00001000000001000001101100001110 Not taintedr00-03 0000000008041b0e 000000004e8045b0 0000000010bbda78 000000004e804470 r04-07 0000000010bb5000 0000000000001440 0000000054355000 0000000000001400 r08-11 0000000055000000 000000000000000e 000000000000000f 0000000055002800 r12-15 0000000000000000 0000000055002800 0000000040b668c0 0000000040b668c0 r16-19 0000000000000001 0000000000000001 0000000051b799f0 0000000000000000 r20-23 0000000000000001 fffffffffffff5b9 0000000000000000 0000000000200000 r24-27 000000000000000c 000000000800000e 0000000054355144 0000000040b3e0c0 r28-31 0000000000010001 000000004e804620 000000004e8045b0 0000000040edd040sr00-03
00000000000a5c000000000000000000 000000000000000000000000000a5c00sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000IASQ: 0000000000000000 0000000000000000 IAOQ:
0000000010bbd9e00000000010bbd9e4
IIR:006110c2ISR: 0000000010240000 IOR:00000003b76dd048
CPU: 0 CR30:0000000040edd040CR31: ffffffffffffffff
ORIG_R28: 0000000000000000
IAOQ[0]:mpt_reply+0x130/0x4f0[mptbase]
IAOQ[1]:mpt_reply+0x134/0x4f0[mptbase]
RP(r2):mpt_reply+0x1c8/0x4f0[mptbase]Backtrace: [<0000000010bbde24>] mpt_interrupt+0x84/0xe8 [mptbase] [<000000004026dd64>] __handle_irq_event_percpu+0xc4/0x250 [<000000004026df28>] handle_irq_event_percpu+0x38/0xd8 [<00000000402776c4>] handle_percpu_irq+0xb4/0xf0 [<000000004026c90c>] generic_handle_irq+0x5c/0x90 [<00000000401a20e4>] call_on_stack+0x18/0x24
Quoted 9 lines
Quoted 9 lines
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000 Backtrace: [<00000000401a8cd8>] show_stack+0x70/0x90 [<0000000040a8e238>] dump_stack_lvl+0xd8/0x128 [<0000000040a8e2bc>] dump_stack+0x34/0x48 [<00000000401a8efc>] die_if_kernel+0x1e4/0x3f8 [<00000000401a9af4>] handle_interruption+0x59c/0xb58 [<00000000401a107c>] intr_check_sig+0x0/0x3c
Can you decode these stacktraces with ./scripts/decode_stacktrace.sh ?
Quoted 5 lines
Quoted 5 lines
Kernel panic - not syncing: Fatal exception in interruptv5.18.2 with similar config is okay. The fault seems consistent. IIR contains illegal instruction.
Attached config.
Also, can you forward this config to me?
Best regards,
Kuniyuki
Quoted 95 lines · Kuniyuki Iwashima
Hello,
Thanks for heads up!Date: Fri, 10 Jun 2022 11:06:24 -0400 From: John David Anglin <[email protected]>Quoted 90 lines
I did a regression search.
e039c0b5985999b150594126225e1ee51df7b4c9is the first bad commitcommit
e039c0b5985999b150594126225e1ee51df7b4c9Author: Kuniyuki Iwashima <[email protected]> Date: Fri Apr 29 14:38:01 2022 -0700list: fix a data-race around
ep->rdllist[ Upstream commit
d679ae94fdd5d3ab00c35078f5af5f37e068b03d]
ep_poll()first callsep_events_available()with no lock held and checks ifep->rdllistis empty bylist_empty_careful(), which readsrdllist->prev. Thus all accesses to it need some protection to avoid store/load-tearing.Note
INIT_LIST_HEAD_RCU()already has the annotation for both prev and next.Commit
bf3b9f6372c4("epoll: Add busy poll support to epoll with socket fds.") added the first locklessep_events_available(), and commitc5a282e9635e("fs/epoll: reduce the scope of wq lock inepoll_wait()") made someep_events_available()calls lockless and added single call under a lock, finally commite59d3c64cba6("epoll: eliminate unnecessary lock for zero timeout") made the lastep_events_available()lockless.BUG: KCSAN: data-race in
do_epoll_wait/do_epoll_waitwrite to
0xffff88810480c7d8of 8 bytes by task 1802 on cpu 0:
INIT_LIST_HEADinclude/linux/list.h:38[inline]list_splice_initinclude/linux/list.h:492[inline]ep_start_scanfs/eventpoll.c:622[inline]ep_send_eventsfs/eventpoll.c:1656[inline]ep_pollfs/eventpoll.c:1806[inline]do_epoll_wait+0x4eb/0xf40fs/eventpoll.c:2234do_epoll_pwaitfs/eventpoll.c:2268[inline]__do_sys_epoll_pwaitfs/eventpoll.c:2281[inline]__se_sys_epoll_pwait+0x12b/0x240fs/eventpoll.c:2275__x64_sys_epoll_pwait+0x74/0x80fs/eventpoll.c:2275do_syscall_x64arch/x86/entry/common.c:50[inline]do_syscall_64+0x44/0xd0arch/x86/entry/common.c:80entry_SYSCALL_64_after_hwframe+0x44/0xaeread to
0xffff88810480c7d8of 8 bytes by task 1799 on cpu 1:
list_empty_carefulinclude/linux/list.h:329[inline]ep_events_availablefs/eventpoll.c:381[inline]ep_pollfs/eventpoll.c:1797[inline]do_epoll_wait+0x279/0xf40fs/eventpoll.c:2234do_epoll_pwaitfs/eventpoll.c:2268[inline]__do_sys_epoll_pwaitfs/eventpoll.c:2281[inline]__se_sys_epoll_pwait+0x12b/0x240fs/eventpoll.c:2275__x64_sys_epoll_pwait+0x74/0x80fs/eventpoll.c:2275do_syscall_x64arch/x86/entry/common.c:50[inline]do_syscall_64+0x44/0xd0arch/x86/entry/common.c:80entry_SYSCALL_64_after_hwframe+0x44/0xaevalue changed:
0xffff88810480c7d0->0xffff888103c15098Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 1799 Comm: syz-fuzzer Tainted: G W 5.17.0-rc7-syzkaller-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011Fixes e59d3c64cba6 ("epoll: eliminate unnecessary lock for zero timeout")Fixes c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()")Fixes bf3b9f6372c4 ("epoll: Add busy poll support to epoll with socket fds.")Signed-off-by Kuniyuki Iwashima <[email protected]>Reported-by [email protected]Cc Al Viro <[email protected]>, Andrew Morton <[email protected]>Cc Kuniyuki Iwashima <[email protected]>Cc Kuniyuki Iwashima <[email protected]>Cc "Soheil Hassas Yeganeh" <[email protected]>Cc Davidlohr Bueso <[email protected]>Cc "Sridhar Samudrala" <[email protected]>Cc Alexander Duyck <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Sasha Levin <[email protected]>include/linux/list.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)Reverting above change fixes v5.18.3 boot.
It seems the most significant byte of the "ldd -10(r3),rp" instruction in
mpt_replyhas been set to 0:4084: bf 80 21 18 cmpb,*<> r0,ret0,4118 <mpt_reply+0x1b8> 4088: 08 04 02 5b copy r4,dp 408c: 00 00 04 00 sync 4090: 0c 61 10 c2 ldd -10(r3),rp
Quoted 4 lines
Quoted 1 line
See IIR value in crash output.
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
Neither am I.
Quoted 3 lines
I'm not familiar with PARISC and this may be a stupid question though, what does
lddexactly do? and which line is it executed in the func/file?
ldd performs a 64-bit load to register rp (r2). It is part of mpt_reply's epilogue. The prior "sync" instruction corresponds to the "mb()" at the end of mpt_reply.
I would have thought this code should have been write protected. It seems CONFIG_STRICT_MODULE_RWX is not explicitly set in config:
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=yCONFIG_STRICT_KERNEL_RWX=y
I think I need to try enabling CONFIG_STRICT_MODULE_RWX.
Quoted 53 lines
Quoted 44 lines · John David Anglin
Quoted 41 lines
[...]
ata3: SATA link down (SStatus 0 SControl 0)_______________________________ < Your System ate a SPARC! Gah! > ------------------------------- \ ^__^ (__)\ )\/\ U ||----w | || || swapper/0 (pid 0): Illegal instruction (code 8) CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000YZrvWESTHLNXBCVMcbcbcbcbOGFRQPDI
PSW: 00001000000001000001101100001110 Not taintedr00-03 0000000008041b0e 000000004e8045b0 0000000010bbda78 000000004e804470 r04-07 0000000010bb5000 0000000000001440 0000000054355000 0000000000001400 r08-11 0000000055000000 000000000000000e 000000000000000f 0000000055002800 r12-15 0000000000000000 0000000055002800 0000000040b668c0 0000000040b668c0 r16-19 0000000000000001 0000000000000001 0000000051b799f0 0000000000000000 r20-23 0000000000000001 fffffffffffff5b9 0000000000000000 0000000000200000 r24-27 000000000000000c 000000000800000e 0000000054355144 0000000040b3e0c0 r28-31 0000000000010001 000000004e804620 000000004e8045b0 0000000040edd040sr00-03
00000000000a5c000000000000000000 000000000000000000000000000a5c00sr04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000IASQ: 0000000000000000 0000000000000000 IAOQ:
0000000010bbd9e00000000010bbd9e4
IIR:006110c2ISR: 0000000010240000 IOR:00000003b76dd048
CPU: 0 CR30:0000000040edd040CR31: ffffffffffffffff
ORIG_R28: 0000000000000000
IAOQ[0]:mpt_reply+0x130/0x4f0[mptbase]
IAOQ[1]:mpt_reply+0x134/0x4f0[mptbase]
RP(r2):mpt_reply+0x1c8/0x4f0[mptbase]Backtrace: [<0000000010bbde24>] mpt_interrupt+0x84/0xe8 [mptbase] [<000000004026dd64>] __handle_irq_event_percpu+0xc4/0x250 [<000000004026df28>] handle_irq_event_percpu+0x38/0xd8 [<00000000402776c4>] handle_percpu_irq+0xb4/0xf0 [<000000004026c90c>] generic_handle_irq+0x5c/0x90 [<00000000401a20e4>] call_on_stack+0x18/0x24Quoted 9 lines
Quoted 9 lines
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.18.3+ #1 Hardware name: 9000/785/C8000 Backtrace: [<00000000401a8cd8>] show_stack+0x70/0x90 [<0000000040a8e238>] dump_stack_lvl+0xd8/0x128 [<0000000040a8e2bc>] dump_stack+0x34/0x48 [<00000000401a8efc>] die_if_kernel+0x1e4/0x3f8 [<00000000401a9af4>] handle_interruption+0x59c/0xb58 [<00000000401a107c>] intr_check_sig+0x0/0x3c
Quoted 1 line
Can you decode these stacktraces with
./scripts/decode_stacktrace.sh?
I don't think this is helpful as the code in mpt_reply has been corrupted.
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Kernel panic - not syncing: Fatal exception in interruptv5.18.2 with similar config is okay. The fault seems consistent. IIR contains illegal instruction.
Attached config.
Also, can you forward this config to me?
Best regards,
Kuniyuki
Signature · 1 line
John David Anglin [email protected]
Quoted 10 lines · John David Anglin
Quoted 3 lines
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
Neither am I.
Quoted 3 lines
I'm not familiar with PARISC and this may be a stupid question though, what does
lddexactly do? and which line is it executed in the func/file?ldd performs a 64-bit load to register rp (r2). It is part of mpt_reply's epilogue. The prior "sync" instruction corresponds to the "mb()" at the end of mpt_reply.
Quoted 7 lines
I would have thought this code should have been write protected. It seems CONFIG_STRICT_MODULE_RWX is not explicitly set in config:
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=yCONFIG_STRICT_KERNEL_RWX=yI think I need to try enabling CONFIG_STRICT_MODULE_RWX.
With CONFIG_STRICT_MODULE_RWX, the fault went away and the system boots normally.
To enable CONFIG_STRICT_MODULE_RWX, I needed to add attached patch to Kconfig.
As far as I can tell, this only affects patch_map in the parisc backend:
static void __kprobes *patch_map(void *addr, int fixmap, unsigned long *flags, int *need_unmap)
{
unsigned long uintaddr = (uintptr_t) addr;
bool module = !core_kernel_text(uintaddr);
struct page *page;
*need_unmap = 0;
if (module && IS_ENABLED(CONFIG_STRICT_MODULE_RWX)) page = vmalloc_to_page(addr);
else if (!module && IS_ENABLED(CONFIG_STRICT_KERNEL_RWX)) page = virt_to_page(addr);
else
return addr;
Possibly, this might affect Fusion MPT base driver but no patches are applied:
[ 29.971295] mptbase alternatives: applied 0 out of 3 patches [ 29.971295] Fusion MPT base driver 3.04.20
[ 29.971295] Copyright (c) 1999-2008 LSI Corporation [ 29.971295] Fusion MPT SPI Host driver 3.04.20
Dave
Signature · 1 line
John David Anglin [email protected]
arch/parisc/Kconfig +1 -0
@@ -10,6 +10,7 @@ config PARISC select ARCH_WANT_FRAME_POINTERS select ARCH_HAS_ELF_RANDOMIZE select ARCH_HAS_STRICT_KERNEL_RWX + select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_HAS_PTE_SPECIAL select ARCH_NO_SG_CHAIN
Helge Deller
maintainerQuoted 12 lines · John David Anglin
Quoted 10 lines · John David Anglin
Quoted 3 lines
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
Neither am I.
Quoted 3 lines
I'm not familiar with PARISC and this may be a stupid question though, what does
lddexactly do? and which line is it executed in the func/file?ldd performs a 64-bit load to register rp (r2). It is part of mpt_reply's epilogue. The prior "sync" instruction corresponds to the "mb()" at the end of mpt_reply.
Quoted 6 lines
Possibly, this might affect Fusion MPT base driver but no patches are applied:
[ 29.971295] mptbase alternatives: applied 0 out of 3 patches [ 29.971295] Fusion MPT base driver 3.04.20
[ 29.971295] Copyright (c) 1999-2008 LSI Corporation [ 29.971295] Fusion MPT SPI Host driver 3.04.20
To sum it up - this issue war triggered by a few special situations:
The kernel patching code uses the altinstructions table from kernel modules to patch
in alternative assembly instructions.
To read the entries it uses a 32-bit ldw() instruction since the table holds 32-bit values. Because of another issue this table was located at unaligned memory addresses.
That's why then the kernel ldw() emulation jumped in and read the content.
Commit e8aa7b17fe41 ("parisc/unaligned: Rewrite inline assembly of emulate_ldw()")
broke the ldw() emulation and as such invalid 32-bit values were read back.
This then triggered random memory corruption, because the kernel then patched addresses which it shouldn't.
I just sent a patch to the parisc mailing list to fix up the ldw() handler, which finally fixed this issue here too.
Everyone who runs kernel v5.18+ on parisc should better apply the patch I sent:
https://patchwork.kernel.org/project/linux-parisc/patch/[email protected]/
Helge
Quoted 35 lines · Helge Deller
Quoted 12 lines · John David Anglin
Quoted 10 lines · John David Anglin
Quoted 3 lines
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
Neither am I.
Quoted 3 lines
I'm not familiar with PARISC and this may be a stupid question though, what does
lddexactly do? and which line is it executed in the func/file?ldd performs a 64-bit load to register rp (r2). It is part of mpt_reply's epilogue. The prior "sync" instruction corresponds to the "
mb()" at the end ofmpt_reply.Quoted 6 lines
Possibly, this might affect Fusion MPT base driver but no patches are applied:
[ 29.971295] mptbase alternatives: applied 0 out of 3 patches [ 29.971295] Fusion MPT base driver 3.04.20 [ 29.971295] Copyright (c) 1999-2008 LSI Corporation [ 29.971295] Fusion MPT SPI Host driver 3.04.20To sum it up - this issue war triggered by a few special situations:
The kernel patching code uses the altinstructions table from kernel modules to patch
in alternative assembly instructions.
To read the entries it uses a 32-bitldw()instruction since the table holds 32-bit values. Because of another issue this table was located at unaligned memory addresses.
That's why then the kernelldw()emulation jumped in and read the content.
Commite8aa7b17fe41("parisc/unaligned: Rewrite inline assembly ofemulate_ldw()")
broke theldw()emulation and as such invalid 32-bit values were read back.
This then triggered random memory corruption, because the kernel then patched addresses which it shouldn't.
Quoted 6 lines
I just sent a patch to the parisc mailing list to fix up the
ldw()handler, which finally fixed this issue here too.Everyone who runs kernel v5.18+ on parisc should better apply the patch I sent:
https://patchwork.kernel.org/project/linux-parisc/patch/[email protected]/
Appreciate you summarising - I was just wondering about this bug earlier :)
Quoted 1 line
Helge
Best,
sam
From: Helge Deller <[email protected]> Date: Mon, 27 Jun 2022 02:08:29 +0200
Quoted 31 lines · John David Anglin
Quoted 12 lines · John David Anglin
Quoted 10 lines
Quoted 3 lines
The commit was added to prevent compiler optimisation from splitting read/write operations. I think it can lead in a change in opcodes but must be safe. So I'm not sure why the commit causes boot failure for now.
Neither am I.
Quoted 3 lines
I'm not familiar with PARISC and this may be a stupid question though, what does
lddexactly do? and which line is it executed in the func/file?ldd performs a 64-bit load to register rp (r2). It is part of mpt_reply's epilogue. The prior "sync" instruction corresponds to the "
mb()" at the end ofmpt_reply.Quoted 6 lines
Possibly, this might affect Fusion MPT base driver but no patches are applied:
[ 29.971295] mptbase alternatives: applied 0 out of 3 patches [ 29.971295] Fusion MPT base driver 3.04.20 [ 29.971295] Copyright (c) 1999-2008 LSI Corporation [ 29.971295] Fusion MPT SPI Host driver 3.04.20To sum it up - this issue war triggered by a few special situations:
The kernel patching code uses the altinstructions table from kernel modules to patch
in alternative assembly instructions.
To read the entries it uses a 32-bitldw()instruction since the table holds 32-bit values. Because of another issue this table was located at unaligned memory addresses.
That's why then the kernelldw()emulation jumped in and read the content.
Commite8aa7b17fe41("parisc/unaligned: Rewrite inline assembly ofemulate_ldw()")
broke theldw()emulation and as such invalid 32-bit values were read back.
This then triggered random memory corruption, because the kernel then patched addresses which it shouldn't.
Quoted 2 lines
I just sent a patch to the parisc mailing list to fix up the
ldw()handler, which finally fixed this issue here too.
Interesting!
I was wondering enabling CONFIG_STRICT_MODULE_RWX, which was originally off, could have another impact.
I appreciate your summary and fix!
Best regards,
Kuniyuki
Quoted 5 lines
Everyone who runs kernel v5.18+ on parisc should better apply the patch I sent:
https://patchwork.kernel.org/project/linux-parisc/patch/[email protected]/Helge