NMIs reported by console_blast.sh with 6.6.20-rt25
Hello John,
The real-time team at Red Hat is discussing backporting the rt patchset in 6.6 to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with that high cpu count server over uart 8250; which is expected since the patchset is similar.
One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile did not return any NMIs while 6.6 did with that same configuration.
Another aspect I noticed during my testing. I did not set grub to start with the realtime profile at boot for this machine. When I did set it the second (and latter) NMI did not show for fully preemptive (the 3rd set at the bottom of this email).
Caller info was enabled. No modifications to the source code were made.
I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25; with the exception of accidently testing 6.6.10-rt19. 6.6.10 also reported NMIs during this test. If you wish to see these reports please let me know.
Also, please let me know if I can help in any other way or if you want anything else tested. I will be moving to 6.8 and using BUG_ON(system_state == SYSTEM_RUNNING); in the scheduler next for my tests to see if the messages do not show.
Link to lscpu and previous NMI reports:
https://lore.kernel.org/linux-rt-users/ZdUmpblrzX0pcPjz@thinkpad2021/T/#mec258264be95dcba23970def7b248af0c6e942b2
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
-----------------------------
[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 All code
========
0: 0f 1f 00 nopl (%rax) 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx 1f: 89 d0 mov %edx,%eax 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ee out %al,(%dx) 2a:* c3 ret <-- trapping instruction 2b: cc int3 2c: cc int3 2d: cc int3 2e: cc int3 2f: 0f 1f 40 00 nopl 0x0(%rax) 33: 90 nop 34: 90 nop 35: 90 nop 36: 90 nop 37: 90 nop 38: 90 nop 39: 90 nop 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop
Code starting with the faulting instruction
===========================================
0: c3 ret 1: cc int3 2: cc int3 3: cc int3 4: cc int3 5: 0f 1f 40 00 nopl 0x0(%rax) 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop [ C56] RSP: 0018:ffa0000030a9fb28 EFLAGS: 00000002 [ C56] RAX: 000000000000005f RBX: ffffffffa6ac68c0 RCX: 0000000000000000 [ C56] RDX: 00000000000003f8 RSI: 0000000000000000 RDI: ffffffffa6ac68c0 [ C56] RBP: 000000000000005f R08: 303378302f623178 R09: 302b6d73615f6b72 [ C56] R10: 615f6b726f665f6d R11: 6f72665f74657220 R12: ffffffffa6ac68c0 [ C56] R13: ffffffffa69b9434 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] </NMI> [ C56] <TASK>
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? do_dup2 (fs/file.c:1142)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94 [ C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001 [ C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20 [ C56] </TASK> [ C56] NMI backtrace for cpu 56 [ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 All code
========
0: cc int3 1: 0f 1f 40 00 nopl 0x0(%rax) 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ec in (%dx),%al 2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction 2d: c3 ret 2e: cc int3 2f: cc int3 30: cc int3 31: cc int3 32: 0f 1f 00 nopl (%rax) 35: 90 nop 36: 90 nop 37: 90 nop 38: 90 nop 39: 90 nop 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax 3: c3 ret 4: cc int3 5: cc int3 6: cc int3 7: cc int3 8: 0f 1f 00 nopl (%rax) b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop [ C56] RSP: 0018:ffa0000030a9fb00 EFLAGS: 00000002 [ C56] RAX: ffffffffa45cf400 RBX: 0000000000000000 RCX: 0000000000000000 [ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffffa6ac68c0 [ C56] RBP: 0000000000002710 R08: 303578302f643278 R09: 302b6b726f665f6d [ C56] R10: 302b6b726f665f6d R11: 6f72665f74657220 R12: 0000000000000020 [ C56] R13: ffffffffa6ac68c0 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI> [ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? do_dup2 (fs/file.c:1142)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94 [ C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001 [ C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20 [ C56] </TASK>
[ T2614] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2614] Call Trace:
[ T2614] <TASK>
[ T2614] dump_stack_lvl (lib/dump_stack.c:107)
[ T2614] panic (kernel/panic.c:340)
[ T2614] ? _printk (kernel/printk/printk.c:2474)
[ T2614] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T2614] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T2614] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T2614] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T2614] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ T2614] vfs_write (fs/read_write.c:582)
[ T2614] ksys_write (fs/read_write.c:637)
[ T2614] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T2614] ? do_dup2 (fs/file.c:1142)
[ T2614] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T2614] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T2614] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T2614] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T2614] RIP: 0033:0x7f4a81e5fc94
[ T2614] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89
[ T2614] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202
[ T2614] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[ T2614] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[ T2614] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[ T2614] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T2614] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[ T2614] </TASK>
[ T2614] Shutting down cpus with NMI
[ T2614] Kernel Offset: disabled
[ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
[disconnect]
]777;notify;Command completed;KRB5CCNAME=/dev/null console -M conserver-02.hosts.prod.psi.bos.redhat.com intel-eaglestream-spr-15.khw3.lab.eng.bos.redhat.com]777;precmd]0;j@thinkpad2021:~/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25]7;file://thinkpad2021/home/j/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25[?2004h[38;5;166mj[38;5;228m6.6.20-rt25[38;5;43m[38;5;71m
Quoted 1 line
$ (B[mexit
[?2004l ]777;preexecexit
Script done on 2024-03-08 12:25:38-05:00 [COMMAND_EXIT_CODE="0"]
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------
[ T2546] sysrq: Trigger a crash
[ T2546] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 All code
========
0: cc int3 1: 0f 1f 40 00 nopl 0x0(%rax) 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ec in (%dx),%al 2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction 2d: c3 ret 2e: cc int3 2f: cc int3 30: cc int3 31: cc int3 32: 0f 1f 00 nopl (%rax) 35: 90 nop 36: 90 nop 37: 90 nop 38: 90 nop 39: 90 nop 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax 3: c3 ret 4: cc int3 5: cc int3 6: cc int3 7: cc int3 8: 0f 1f 00 nopl (%rax) b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop [ C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002 [ C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000 [ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0 [ C56] RBP: 000000000000270f R08: 6469703e2d727275 R09: 2020202020202020 [ C56] R10: 00000000632e2020 R11: 00000000632e2020 R12: 0000000000000020 [ C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI> [ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94 [ C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001 [ C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20 [ C56] </TASK> [ C56] NMI backtrace for cpu 56 [ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 All code
========
0: cc int3 1: 0f 1f 40 00 nopl 0x0(%rax) 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ec in (%dx),%al 2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction 2d: c3 ret 2e: cc int3 2f: cc int3 30: cc int3 31: cc int3 32: 0f 1f 00 nopl (%rax) 35: 90 nop 36: 90 nop 37: 90 nop 38: 90 nop 39: 90 nop 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax 3: c3 ret 4: cc int3 5: cc int3 6: cc int3 7: cc int3 8: 0f 1f 00 nopl (%rax) b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop [ C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002 [ C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000 [ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0 [ C56] RBP: 000000000000270f R08: 64656c74746f7268 R09: 2020202020202020 [ C56] R10: 00000000742e2020 R11: 00000000742e2020 R12: 0000000000000020 [ C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI> [ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124) [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94 [ C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001 [ C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20 [ C56] </TASK>
[ T2546] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2546] Call Trace:
[ T2546] <TASK>
[ T2546] dump_stack_lvl (lib/dump_stack.c:107)
[ T2546] panic (kernel/panic.c:340)
[ T2546] ? _printk (kernel/printk/printk.c:2474)
[ T2546] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T2546] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T2546] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T2546] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T2546] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ T2546] vfs_write (fs/read_write.c:582)
[ T2546] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ T2546] ksys_write (fs/read_write.c:637)
[ T2546] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T2546] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T2546] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T2546] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ T2546] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ T2546] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T2546] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T2546] RIP: 0033:0x7f4c1740cc94
[ T2546] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89
[ T2546] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202
[ T2546] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94 [ T2546] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001 [ T2546] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001 [ T2546] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ T2546] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20 [ T2546] </TASK>
[ T2546] Shutting down cpus with NMI
[ T2546] Kernel Offset: disabled
[ T2546] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
-----------------------------
NMI Backtrace for 6.6.20-rt25 fully preemptive with tuned realtime profile
-----------------------------
[ T3402] sysrq: Trigger a crash
[ T3406]
[ T3402] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] Code: cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8d 04 37 31 c9 48 89 c2 48 c1 ea 20 66 0f ae f1 <c3> cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 All code
========
0: cc int3 1: 0f 1f 00 nopl (%rax) 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 19: 48 8d 04 37 lea (%rdi,%rsi,1),%rax 1d: 31 c9 xor %ecx,%ecx 1f: 48 89 c2 mov %rax,%rdx 22: 48 c1 ea 20 shr $0x20,%rdx 26: 66 0f ae f1 tpause %ecx 2a:* c3 ret <-- trapping instruction 2b: cc int3 2c: cc int3 2d: cc int3 2e: cc int3 2f: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 34: 90 nop 35: 90 nop 36: 90 nop 37: 90 nop 38: 90 nop 39: 90 nop 3a: 90 nop 3b: 90 nop 3c: 90 nop 3d: 90 nop 3e: 90 nop 3f: 90 nop
Code starting with the faulting instruction
===========================================
0: c3 ret 1: cc int3 2: cc int3 3: cc int3 4: cc int3 5: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 90 nop 10: 90 nop 11: 90 nop 12: 90 nop 13: 90 nop 14: 90 nop 15: 90 nop [ C56] RSP: 0018:ffa0000039cfba80 EFLAGS: 00000002 [ C56] RAX: 00000099a649b2ae RBX: 00000000000006d5 RCX: 0000000000000000 [ C56] RDX: 0000000000000099 RSI: 00000000000006d5 RDI: 00000099a649abd9 [ C56] RBP: 00000099a649abd9 R08: 303178302f303178 R09: 302b646165726874 [ C56] R10: 6461657268746b5f R11: 7866705f5f203f20 R12: 0000000000000020 [ C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] </NMI> [ C56] <TASK>
[ C56] delay_halt.part.0 (arch/x86/lib/delay.c:163)
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94 [ C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001 [ C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20 [ C56] </TASK> [ C56] NMI backtrace for cpu 56 [ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 53 48 89 fb 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 48 89 d5 eb 09 <48> 01 eb 48 89 c5 48 29 c3 48 8b 05 15 bd a0 00 48 89 de 48 89 ef All code
========
0: 90 nop 1: 90 nop 2: 90 nop 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: 90 nop d: 90 nop e: 90 nop f: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 14: 55 push %rbp 15: 53 push %rbx 16: 48 89 fb mov %rdi,%rbx 19: 0f 01 f9 rdtscp 1c: 66 90 xchg %ax,%ax 1e: 48 c1 e2 20 shl $0x20,%rdx 22: 48 09 c2 or %rax,%rdx 25: 48 89 d5 mov %rdx,%rbp 28: eb 09 jmp 0x33 2a:* 48 01 eb add %rbp,%rbx <-- trapping instruction 2d: 48 89 c5 mov %rax,%rbp 30: 48 29 c3 sub %rax,%rbx 33: 48 8b 05 15 bd a0 00 mov 0xa0bd15(%rip),%rax # 0xa0bd4f 3a: 48 89 de mov %rbx,%rsi 3d: 48 89 ef mov %rbp,%rdi
Code starting with the faulting instruction
===========================================
0: 48 01 eb add %rbp,%rbx 3: 48 89 c5 mov %rax,%rbp 6: 48 29 c3 sub %rax,%rbx 9: 48 8b 05 15 bd a0 00 mov 0xa0bd15(%rip),%rax # 0xa0bd25 10: 48 89 de mov %rbx,%rsi 13: 48 89 ef mov %rbp,%rdi [ C56] RSP: 0018:ffa0000039cfba88 EFLAGS: 00000093 [ C56] RAX: 000000ec13deb29f RBX: 000000000000076d RCX: 0000000000004038 [ C56] RDX: 0000000000000050 RSI: 000000000000076d RDI: 000000ec13deb24f [ C56] RBP: 000000ec13deb24f R08: 20203138312f7068 R09: 7570632020202020 [ C56] R10: 38312f7068757063 R11: 2020202020205320 R12: 0000000000000020 [ C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000 [ C56] FS: 00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000 [ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0 [ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 [ C56] PKRU: 55555554 [ C56] Call Trace: [ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1)) [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] </NMI> [ C56] <TASK>
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339) [ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89 [ C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94 [ C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001 [ C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001 [ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20 [ C56] </TASK>
[ T3402] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T3402] Call Trace:
[ T3402] <TASK>
[ T3402] dump_stack_lvl (lib/dump_stack.c:107)
[ T3402] panic (kernel/panic.c:340)
[ T3402] ? _printk (kernel/printk/printk.c:2474)
[ T3402] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T3402] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T3402] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T3402] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T3402] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1)) [ T3402] vfs_write (fs/read_write.c:582)
[ T3402] ksys_write (fs/read_write.c:637)
[ T3402] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T3402] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T3402] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T3402] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T3402] RIP: 0033:0x7f12ca638c94
[ T3402] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89 All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax) 6: b8 ff ff ff ff mov $0xffffffff,%eax b: c3 ret c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1) 13: 00 00 00 16: f3 0f 1e fa endbr64 1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6 21: 74 13 je 0x36 23: b8 01 00 00 00 mov $0x1,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 54 ja 0x86 32: c3 ret 33: 0f 1f 00 nopl (%rax) 36: 55 push %rbp 37: 48 89 e5 mov %rsp,%rbp 3a: 48 83 ec 20 sub $0x20,%rsp 3e: 48 rex.W 3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 54 ja 0x5c 8: c3 ret 9: 0f 1f 00 nopl (%rax) c: 55 push %rbp d: 48 89 e5 mov %rsp,%rbp 10: 48 83 ec 20 sub $0x20,%rsp 14: 48 rex.W 15: 89 .byte 0x89
[ T3402] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202
[ T3402] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94 [ T3402] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001 [ T3402] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001 [ T3402] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002 [ T3402] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20 [ T3402] </TASK>
[ T3402] Kernel Offset: disabled
[ T3402] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
Signature · 3 lines
Sincerly, John Wyatt Software Engineer, Core Kernel
Quoted 1 line · John B. Wyatt IV
Hello John,
Hi John B.,
Quoted 8 lines
The real-time team at Red Hat is discussing backporting the rt patchset in 6.6 to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with
console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with that high cpu count server over uart 8250; which is expected since the patchset is similar.One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile did not return any NMIs while 6.6 did with that same configuration.
The thing is that console_blast.sh does this "show a backtrace on all CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't imagine how you did obtain the backtraces without an NMI. Unless the tuned profile disables this somehow.
Quoted 11 lines
Another aspect I noticed during my testing. I did not set grub to start with the realtime profile at boot for this machine. When I did set it the second (and latter) NMI did not show for fully preemptive (the 3rd set at the bottom of this email).
Caller info was enabled. No modifications to the source code were made.
I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25; with the exception of accidently testing 6.6.10-rt19. 6.6.10 also reported NMIs during this test. If you wish to see these reports please let me know.
This NMI part has nothing todo with printk. If you need this clarified, I would need a reproducer.
…
Quoted 11 lines
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
-----------------------------[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 All code
=======
…
Quoted 9 lines
12: 90 nop
13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx1f: 89 d0 mov %edx,%eax 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ee out %al,(%dx)2a:* c3 ret <-- trapping instruction
where is this output from? The `ret' opcode usually does not cause a trap. My guess is that the machine has been interrupted by an external user at this position.
Side note: This is using early_printk, correct?
…
Quoted 5 lines
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ?nmi_cpu_backtrace(lib/nmi_backtrace.c:115)
[ C56] ?nmi_cpu_backtrace_handler(arch/x86/kernel/apic/hw_nmi.c:47(discriminator 1)) [ C56] ?nmi_handle(arch/x86/kernel/nmi.c:149)
Quoted 8 lines
[ C56] ?
io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?default_do_nmi(arch/x86/kernel/nmi.c:347) [ C56] ?exc_nmi(arch/x86/kernel/nmi.c:538)
[ C56] ?end_repeat_nmi(arch/x86/entry/entry_64.S:1458) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105)
[ C56] </NMI>
This looks okay. The NMI did the backtrace as expected.
Quoted 16 lines
[ C56] <TASK>
[ C56]serial8250_console_putchar(./include/linux/serial_core.h:704drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ?__pfx_serial8250_console_putchar(drivers/tty/serial/8250/8250_port.c:3343)
[ C56]uart_console_write(drivers/tty/serial/serial_core.c:2134)
[ C56]serial8250_console_write_atomic(drivers/tty/serial/8250/8250_port.c:3628)
[ C56]nbcon_emit_next_record(kernel/printk/nbcon.c:940)
[ C56]__nbcon_atomic_flush_all(kernel/printk/nbcon.c:1192(discriminator 1)kernel/printk/nbcon.c:1326(discriminator 1))
[ C56]vprintk_emit(kernel/printk/printk.c:2414)
[ C56]_printk(kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207./arch/x86/include/asm/bitops.h:239./include/asm-generic/bitops/instrumented-non-atomic.h:142kernel/panic.c:528kernel/panic.c:339) [ C56] ?_printk(kernel/printk/printk.c:2474)
[ C56]sysrq_handle_crash(drivers/tty/sysrq.c:154)
[ C56]__handle_sysrq(drivers/tty/sysrq.c:601)
[ C56]write_sysrq_trigger(drivers/tty/sysrq.c:1165)
[ C56]proc_reg_write(fs/proc/inode.c:340fs/proc/inode.c:352)
[ C56] ?preempt_count_add(./include/linux/ftrace.h:974(discriminator 1)kernel/sched/core.c:5847(discriminator 1)kernel/sched/core.c:5844(discriminator 1)kernel/sched/core.c:5872(discriminator 1))
Quoted 8 lines
[ C56]
vfs_write(fs/read_write.c:582)
[ C56]ksys_write(fs/read_write.c:637)
[ C56]do_syscall_64(arch/x86/entry/common.c:51arch/x86/entry/common.c:81)
[ C56] ?do_dup2(fs/file.c:1142)
[ C56] ?syscall_exit_to_user_mode(kernel/entry/common.c:299)
[ C56] ?do_syscall_64(arch/x86/entry/common.c:88)
[ C56] ?exc_page_fault(./arch/x86/include/asm/irqflags.h:37./arch/x86/include/asm/irqflags.h:72arch/x86/mm/fault.c:1513arch/x86/mm/fault.c:1561) [ C56]entry_SYSCALL_64_after_hwframe(arch/x86/entry/entry_64.S:120)
According to this, someone issued a `crash' via sysrq. Why?
…
Quoted 1 line
[ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
yes exactly.
…
Quoted 2 lines
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------
This and the following backtrace shows the same picture: The CPU is crashing due to proc/sysrq request and does CPU-backtraces via NMI and polls in early_printk, waiting for the UART to become idle (probably).
I don't see an issue here so far.
…
Quoted 4 lines
Signature · 3 lines
Sincerly, John Wyatt Software Engineer, Core Kernel
Sebastian
Quoted 5 lines · Sebastian Andrzej Siewior
On 2024-03-15 15:21:19 [-0400], John B. Wyatt IV wrote:
The thing is thatconsole_blast.shdoes this "show a backtrace on all CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't imagine how you did obtain the backtraces without an NMI. Unless the tuned profile disables this somehow.
I may have misapplied it. This is a very manual process right now due to conserver.
Quoted 22 lines
Quoted 11 lines
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
-----------------------------[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 All code
=======…
Quoted 9 lines
12: 90 nop
13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx1f: 89 d0 mov %edx,%eax 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx 25: d3 e6 shl %cl,%esi 27: 01 f2 add %esi,%edx 29: ee out %al,(%dx)2a:* c3 ret <-- trapping instruction
Quoted 3 lines
where is this output from? The `ret' opcode usually does not cause a trap. My guess is that the machine has been interrupted by an external user at this position.
Just before the sysrq that crashes the system.
[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312)
[ T2618] </TASK>
[ T2618] task:kworker/185:0 state:I stack:0 pid:1131 ppid:2 flags:0x00004000
[ T2618] Workqueue: 0x0 (events)
[ T2618] Call Trace:
[ T2618] <TASK>
[ T2618] __schedule (kernel/sched/core.c:5404 kernel/sched/core.c:6718)
[ T2618] schedule (kernel/sched/core.c:6796 kernel/sched/core.c:6810)
[ T2618] worker_thread (kernel/workqueue.c:2734)
[ T2618] ? __pfx_worker_thread (kernel/workqueue.c:2727)
[ T2618] kthread (kernel/kthread.c:388)
[ T2618] ? __pfx_kthread (kernel/kthread.c:341)
[ T2618] ret_from_fork (arch/x86/kernel/process.c:147)
[ T2618] ? __pfx_kthread (kernel/kthread.c:341)
[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312)
[ T2618] </TASK>
[ T2614] sysrq: Trigger a crash
[ T2618] task:kworker/185:0H state:I
[ T2618] stack:0 pid:1132 ppid:2 flags:0x00004000
[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
Quoted 1 line
Side note: This is using
early_printk, correct?
I believe so, but it might be preempted? This is the part it stopped in.
static void io_serial_out(unsigned long addr, int offset, int value)
{
outb(value, addr + offset);
}Quoted 43 lines
…
Quoted 5 lines
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ?nmi_cpu_backtrace(lib/nmi_backtrace.c:115)
[ C56] ?nmi_cpu_backtrace_handler(arch/x86/kernel/apic/hw_nmi.c:47(discriminator 1)) [ C56] ?nmi_handle(arch/x86/kernel/nmi.c:149)Quoted 8 lines
[ C56] ?
io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?default_do_nmi(arch/x86/kernel/nmi.c:347) [ C56] ?exc_nmi(arch/x86/kernel/nmi.c:538)
[ C56] ?end_repeat_nmi(arch/x86/entry/entry_64.S:1458) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105) [ C56] ?io_serial_out(arch/x86/kernel/early_printk.c:105)
[ C56] </NMI>This looks okay. The NMI did the backtrace as expected.
Quoted 16 lines
[ C56] <TASK>
[ C56]serial8250_console_putchar(./include/linux/serial_core.h:704drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ?__pfx_serial8250_console_putchar(drivers/tty/serial/8250/8250_port.c:3343)
[ C56]uart_console_write(drivers/tty/serial/serial_core.c:2134)
[ C56]serial8250_console_write_atomic(drivers/tty/serial/8250/8250_port.c:3628)
[ C56]nbcon_emit_next_record(kernel/printk/nbcon.c:940)
[ C56]__nbcon_atomic_flush_all(kernel/printk/nbcon.c:1192(discriminator 1)kernel/printk/nbcon.c:1326(discriminator 1))
[ C56]vprintk_emit(kernel/printk/printk.c:2414)
[ C56]_printk(kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207./arch/x86/include/asm/bitops.h:239./include/asm-generic/bitops/instrumented-non-atomic.h:142kernel/panic.c:528kernel/panic.c:339) [ C56] ?_printk(kernel/printk/printk.c:2474)
[ C56]sysrq_handle_crash(drivers/tty/sysrq.c:154)
[ C56]__handle_sysrq(drivers/tty/sysrq.c:601)
[ C56]write_sysrq_trigger(drivers/tty/sysrq.c:1165)
[ C56]proc_reg_write(fs/proc/inode.c:340fs/proc/inode.c:352)
[ C56] ?preempt_count_add(./include/linux/ftrace.h:974(discriminator 1)kernel/sched/core.c:5847(discriminator 1)kernel/sched/core.c:5844(discriminator 1)kernel/sched/core.c:5872(discriminator 1))Quoted 8 lines
[ C56]
vfs_write(fs/read_write.c:582)
[ C56]ksys_write(fs/read_write.c:637)
[ C56]do_syscall_64(arch/x86/entry/common.c:51arch/x86/entry/common.c:81)
[ C56] ?do_dup2(fs/file.c:1142)
[ C56] ?syscall_exit_to_user_mode(kernel/entry/common.c:299)
[ C56] ?do_syscall_64(arch/x86/entry/common.c:88)
[ C56] ?exc_page_fault(./arch/x86/include/asm/irqflags.h:37./arch/x86/include/asm/irqflags.h:72arch/x86/mm/fault.c:1513arch/x86/mm/fault.c:1561) [ C56]entry_SYSCALL_64_after_hwframe(arch/x86/entry/entry_64.S:120)
Quoted 2 lines
According to this, someone issued a `crash' via sysrq. Why?
This is part of the console_blast.sh script that John Ogness sent me.
Please see below:
#!/bin/bash
set -e
# This script is intended to be started with no arguments. #
# It then calls itself to create a pinned process for each CPU. Those
# child processes will run in infinite loops of show-task-states via # /proc/sysrq-trigger. This generates lots of contention on the console. # # Finally, it waits 10 seconds before crashing the system.
if [ $# -eq 0 ]; then
cpus=$(($(nproc) - 1)) for i in $(seq 0 $cpus); do $0 $i & done sleep 15 echo c > /proc/sysrq-trigger exit 0
fi
# Setup the CPU mask for the provided CPU number. The real taskset(1)
# could simply use -c, but the busybox(1) variant does not support this. mask=$(printf "0x%x\n" $((1 << $1))) taskset -p $mask $$
# Wait a moment before beginning blast.
sleep 2
while true; do
echo t > /proc/sysrq-trigger
done
Quoted 8 lines
Quoted 2 lines
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------This and the following backtrace shows the same picture: The CPU is crashing due to proc/sysrq request and does CPU-backtraces via NMI and polls in
early_printk, waiting for the UART to become idle (probably).I don't see an issue here so far.
Luis Goncalves discussed it with me after reading your response. Thank you for your help. The NMI was needed to flush the buffers upon the system crashing itself. Does this part about NMI watchdog need to be documented?
Signature · 3 lines
Sincerly, John Wyatt Software Engineer, Core Kernel
Quoted 5 lines · John B. Wyatt IV
Quoted 3 lines
where is this output from? The `ret' opcode usually does not cause a trap. My guess is that the machine has been interrupted by an external user at this position.
Just before the sysrq that crashes the system.
so this is intentional.
…
Quoted 8 lines
Quoted 1 line
Side note: This is using
early_printk, correct?I believe so, but it might be preempted? This is the part it stopped in.
static void
io_serial_out(unsigned long addr, int offset, int value){ outb(value, addr + offset); }
The function is invoked in NMU context so it can't be preempted.
Quoted 6 lines
Quoted 2 lines
According to this, someone issued a `crash' via sysrq. Why?
This is part of the
console_blast.shscript that John Ogness sent me.Please see below:
…
Okay. Then everything works as it should…
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------This and the following backtrace shows the same picture: The CPU is crashing due to proc/sysrq request and does CPU-backtraces via NMI and polls in
early_printk, waiting for the UART to become idle (probably).
Quoted 6 lines
Quoted 1 line
I don't see an issue here so far.
Luis Goncalves discussed it with me after reading your response. Thank you for your help. The NMI was needed to flush the buffers upon the system crashing itself. Does this part about NMI watchdog need to be documented?
Not sure about that one. There is an _a_ _lot_ to be printed from NMI and the NMI watchdog might trigger if nothing is triggering the NMI-watchdog during the print job. Also, the crash was requested.
Sebastian
Quoted 3 lines · Sebastian Andrzej Siewior
Quoted 1 line
Just before the sysrq that crashes the system.
so this is intentional.
Yes.
Quoted 7 lines
Quoted 4 lines
This is part of the
console_blast.shscript that John Ogness sent me.Please see below:
…
Okay. Then everything works as it should…
Correct.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------This and the following backtrace shows the same picture: The CPU is crashing due to proc/sysrq request and does CPU-backtraces via NMI and polls in
early_printk, waiting for the UART to become idle (probably).I don't see an issue here so far.
Quoted 8 lines
Quoted 4 lines
Luis Goncalves discussed it with me after reading your response. Thank you for your help. The NMI was needed to flush the buffers upon the system crashing itself. Does this part about NMI watchdog need to be documented?
Not sure about that one. There is an _a_ _lot_ to be printed from NMI and the NMI watchdog might trigger if nothing is triggering the NMI-watchdog during the print job. Also, the crash was requested.
I reran the 6.6 test and no NMI was reported with fully preemptive and the realtime tuned profile. It was my error; my apologies for that.
I did include more of logs if you want to see here for my 6.8 testing.
https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u
Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat
Quoted 2 lines · John B. Wyatt IV
I reran the 6.6 test and no NMI was reported with fully preemptive and the realtime tuned profile. It was my error; my apologies for that.
no worries.
Quoted 3 lines
I did include more of logs if you want to see here for my 6.8 testing.
https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u
I looked over it and it seems we are good.
Thanks for testing.
Quoted 4 lines
Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat
Sebastian