tty/vt: Touch NMI watchdog in vt_console_print
PATCH 1/1 tty/vt: Touch NMI watchdog in vt_console_print
vt_console_print could trigger NMI watchdog in case writing slow:
[2858736.789664] NMI watchdog: Watchdog detected hard LOCKUP on cpu 23 ... [2858736.790194] CPU: 23 PID: 32504 Comm: tensorflow_mode Not tainted 4.4.131-1.el7.elrepo.x86_64 #1 [2858736.790206] Hardware name: Huawei RH2288 V3/BC11HGSB0, BIOS 3.57 02/26/2017 [2858736.790222] task: ffff881e0a191640 ti: ffff881fd73a8000 task.ti: ffff881fd73a8000 [2858736.790358] RIP: 0010:[<ffffffff810cc06e>] [<ffffffff810cc06e>] native_queued_spin_lock_slowpath+0x15e/0x170 [2858736.790363] RSP: 0018:ffff88203f043db0 EFLAGS: 00000002 [2858736.790365] RAX: 00000000005c0101 RBX: 0000000000000246 RCX: 0000000000000001 ... [2858736.790452] Call Trace: [2858736.790521] <IRQ> [2858736.790521] [<ffffffff8118ab28>] queued_spin_lock_slowpath+0xb/0xf [2858736.790552] [<ffffffff8170eca7>] _raw_spin_lock_irqsave+0x37/0x40 [2858736.790653] [<ffffffff814c80a4>] scsi_end_request+0x104/0x1d0 [2858736.790656] [<ffffffff814c9e13>] scsi_io_completion+0x153/0x650 [2858736.790671] [<ffffffff814c1092>] scsi_finish_command+0xd2/0x120 [2858736.790673] [<ffffffff814c9607>] scsi_softirq_done+0x127/0x150 [2858736.790749] [<ffffffff8131973e>] blk_done_softirq+0x8e/0xc0 [2858736.790811] [<ffffffff810857db>] __do_softirq+0xeb/0x2f0 [2858736.790813] [<ffffffff81085c85>] irq_exit+0xf5/0x100 [2858736.790867] [<ffffffff81051819>] smp_call_function_single_interrupt+0x39/0x40 [2858736.790890] [<ffffffff8171055b>] call_function_single_interrupt+0x9b/0xa0 [2858736.790973] <EOI>
...
PID: 1793 TASK: ffff88103445c2c0 CPU: 32 COMMAND: "java"
#0 [ffff88103fe88e38] crash_nmi_callback at ffffffff810504d7 #1 [ffff88103fe88e48] nmi_handle at ffffffff8101c1f7 #2 [ffff88103fe88ea0] default_do_nmi at ffffffff8101c7d0 #3 [ffff88103fe88ec0] do_nmi at ffffffff8101c901 #4 [ffff88103fe88ee8] end_repeat_nmi at ffffffff8171176a
[exception RIP: cfb_imageblit+1167]
RIP: ffffffff813bdf8f RSP: ffff880006823380 RFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000005
RDX: 000000000000024d RSI: 00000000ff000000 RDI: 0000000000000001
RBP: ffff8800068233f0 R8: ffffffff81785e80 R9: ffffc9000c843168
R10: 0000000000000001 R11: 0000000000000000 R12: ffff882037a831ba
R13: ffff882037a831af R14: ffffc9000c84316c R15: ffffc9000c843000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018--- <NMI exception stack> ---
#5 [ffff880006823380] cfb_imageblit at ffffffff813bdf8f #6 [ffff880006823398] bit_putcs at ffffffff813b2307 #7 [ffff8800068234d0] bit_cursor at ffffffff813b1fc8 #8 [ffff8800068235f0] fbcon_scroll at ffffffff813aebda #9 [ffff880006823650] scrup at ffffffff81442600
...
the cpu23 wait for the same blk queue_lock
--- drivers/tty/vt/vt.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/tty/vt/vt.c +2 -0
@@ -105,6 +105,7 @@#include <linux/ctype.h> #include <linux/bsearch.h> #include <linux/gcd.h> +#include <linux/nmi.h> #define MAX_NR_CON_DRIVER 16
@@ -2978,6 +2979,7 @@ static void vt_console_print(struct console *co, const char *b, unsigned count) vc->vc_pos +=2 ; vc->vc_x++; } +touch_nmi_watchdog (); }if (cnt &&con_is_visible (vc)) vc->vc_sw->con_putcs (vc, start, cnt, vc->vc_y, start_x);
Signature · 1 line
1.9.1
Greg KH
maintainerQuoted 49 lines · Qiujun Huang
vt_console_printcould trigger NMI watchdog in case writing slow:[2858736.789664] NMI watchdog: Watchdog detected hard LOCKUP on cpu 23 ... [2858736.790194] CPU: 23 PID: 32504 Comm: tensorflow_mode Not tainted 4.4.131-1.el7.elrepo.x86_64 #1 [2858736.790206] Hardware name: Huawei RH2288 V3/BC11HGSB0, BIOS 3.57 02/26/2017 [2858736.790222] task: ffff881e0a191640 ti: ffff881fd73a8000 task.ti: ffff881fd73a8000 [2858736.790358] RIP: 0010:[<ffffffff810cc06e>] [<ffffffff810cc06e>] native_queued_spin_lock_slowpath+0x15e/0x170 [2858736.790363] RSP: 0018:ffff88203f043db0 EFLAGS: 00000002 [2858736.790365] RAX: 00000000005c0101 RBX: 0000000000000246 RCX: 0000000000000001 ... [2858736.790452] Call Trace: [2858736.790521] <IRQ> [2858736.790521] [<ffffffff8118ab28>] queued_spin_lock_slowpath+0xb/0xf [2858736.790552] [<ffffffff8170eca7>] _raw_spin_lock_irqsave+0x37/0x40 [2858736.790653] [<ffffffff814c80a4>] scsi_end_request+0x104/0x1d0 [2858736.790656] [<ffffffff814c9e13>] scsi_io_completion+0x153/0x650 [2858736.790671] [<ffffffff814c1092>] scsi_finish_command+0xd2/0x120 [2858736.790673] [<ffffffff814c9607>] scsi_softirq_done+0x127/0x150 [2858736.790749] [<ffffffff8131973e>] blk_done_softirq+0x8e/0xc0 [2858736.790811] [<ffffffff810857db>] __do_softirq+0xeb/0x2f0 [2858736.790813] [<ffffffff81085c85>] irq_exit+0xf5/0x100 [2858736.790867] [<ffffffff81051819>] smp_call_function_single_interrupt+0x39/0x40 [2858736.790890] [<ffffffff8171055b>] call_function_single_interrupt+0x9b/0xa0 [2858736.790973] <EOI>...
PID: 1793 TASK:
ffff88103445c2c0CPU: 32 COMMAND: "java"
#0 [ffff88103fe88e38]crash_nmi_callbackatffffffff810504d7#1 [ffff88103fe88e48]nmi_handleatffffffff8101c1f7#2 [ffff88103fe88ea0]default_do_nmiatffffffff8101c7d0#3 [ffff88103fe88ec0]do_nmiatffffffff8101c901#4 [ffff88103fe88ee8]end_repeat_nmiatffffffff8171176a[exception RIP: cfb_imageblit+1167] RIP: ffffffff813bdf8f RSP: ffff880006823380 RFLAGS: 00000046 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000005 RDX: 000000000000024d RSI: 00000000ff000000 RDI: 0000000000000001 RBP: ffff8800068233f0 R8: ffffffff81785e80 R9: ffffc9000c843168 R10: 0000000000000001 R11: 0000000000000000 R12: ffff882037a831ba R13: ffff882037a831af R14: ffffc9000c84316c R15: ffffc9000c843000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018--- <NMI exception stack> ---
#5 [ffff880006823380]cfb_imageblitatffffffff813bdf8f#6 [ffff880006823398]bit_putcsatffffffff813b2307#7 [ffff8800068234d0]bit_cursoratffffffff813b1fc8#8 [ffff8800068235f0]fbcon_scrollatffffffff813aebda#9 [ffff880006823650] scrup atffffffff81442600
...
Quoted 1 line
the cpu23 wait for the same blk
queue_lock
Why is this acting "slow"?
Do we want to sprinkle this type of call everywhere in the console layer (hint, I doubt it.)
This feels like something odd is wrong with your system, not with the tty layer...
thanks,
greg k-h