Skip to contentSkip to conversation

soft lockup in serial8250_console_write(?)

DISCUSSION Randy.Dunlap 3 participants 5 messages Mar 14, 2006 Scheduler

lore
Share & tools
lore
#0 lore raw

Hi,

(x86_64; 2.6.16-rc6;
serial console configured but nothing connected to the serial port)

I'm seeing an occasional soft lockup, maybe in serial8250_console_write(). (drivers/serial/8250.c)

This function calls wait_for_xmitr() [inline], which in worst case can spin for 1.010 seconds. Could this be the cause of a soft lockup?

[  101.448855] BUG: soft lockup detected on CPU#0!
[  101.448858] CPU 0:
[  101.448860] Modules linked in:
[  101.448863] Pid: 1, comm: swapper Not tainted 2.6.16-rc6 #3
[  101.448866] RIP: 0010:[<ffffffff8013009b>] <ffffffff8013009b>{__do_softirq+75}
[  101.448874] RSP: 0000:ffffffff804dc9a8  EFLAGS: 00000206
[  101.448877] RAX: ffff81007f63dfd8 RBX: ffffffff804dc8f8 RCX: 000000000000000a
[  101.448880] RDX: 0000000000000022 RSI: 0000000000000002 RDI: ffff81007f622740
[  101.448883] RBP: ffffffff8010aef0 R08: 0000000000000083 R09: ffff81007f287638
[  101.448886] R10: 0000000000000002 R11: 0000000000000000 R12: ffffffff804dc9c8
[  101.448889] R13: ffffffff80529d00 R14: 0000000000000022 R15: ffffffff8010d34d
[  101.448892] FS:  0000000000000000(0000) GS:ffffffff80529000(0000) knlGS:0000000000000000
[  101.448895] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[  101.448898] CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
[  101.448900] 
[  101.448900] Call Trace: <IRQ> <ffffffff8010bb92>{call_softirq+30}
[  101.448908]        <ffffffff8010d309>{do_softirq+49} <ffffffff8012fe63>{irq_exit+72}
[  101.448915]        <ffffffff80115ab1>{smp_apic_timer_interrupt+75} <ffffffff8010b536>{apic_timer_interrupt+98} <EOI>
[  101.448923]        <ffffffff8028607e>{serial8250_console_write+516} <ffffffff8012add7>{release_console_sem+335}
[  101.448936]        <ffffffff8012b188>{register_console+433} <ffffffff803134cb>{usb_serial_console_init+51}
[  101.448943]        <ffffffff80312556>{usb_serial_probe+3989} <ffffffff8012b8c3>{vprintk+721}
[  101.448956]        <ffffffff803a3bb7>{_spin_unlock_irq+9} <ffffffff8012b93f>{printk+103}
[  101.448964]        <ffffffff8013a084>{call_usermodehelper_keys+274} <ffffffff802f1dc8>{usb_probe_interface+223}
[  101.448975]        <ffffffff8028f1ff>{driver_probe_device+90} <ffffffff8028f35c>{__driver_attach+135}
[  101.448983]        <ffffffff8028f2d5>{__driver_attach+0} <ffffffff8028e726>{bus_for_each_dev+73}
[  101.448990]        <ffffffff8028f094>{driver_attach+28} <ffffffff8028ea48>{bus_add_driver+114}
[  101.448996]        <ffffffff8028f7f9>{driver_register+180} <ffffffff802f1c7a>{usb_register_driver+134}
[  101.449003]        <ffffffff803112e6>{usb_serial_register+570} <ffffffff805550fc>{belkin_sa_init+39}
[  101.449012]        <ffffffff8010820b>{init+451} <ffffffff8010b842>{child_rip+8}
[  101.449020]        <ffffffff80108048>{init+0} <ffffffff8010b83a>{child_rip+0}

---

#1 lore raw
Quoted 10 lines · Randy.Dunlap

Hi,

(x86_64; 2.6.16-rc6; serial console configured but nothing connected to the serial port)

I'm seeing an occasional soft lockup, maybe in serial8250_console_write(). (drivers/serial/8250.c)

This function calls wait_for_xmitr() [inline], which in worst case can spin for 1.010 seconds. Could this be the cause of a soft lockup?

hm, it shouldnt cause that. Could you try the attached patch [which is the next-gen softlockup detector], do you get the message even with that one applied?

Ingo

below is the latest softdog-rework patch, with all fixes (including Nathan's hotplug CPU fix) merged.

--------
this patch makes the softlockup detector purely timer-interrupt driven, removing softirq-context (timer) dependencies. This means that if the softlockup watchdog triggers, it has truly observed a longer than 10 seconds scheduling delay of a SCHED_FIFO prio 99 task.

(the patch also turns off the softlockup detector during the initial bootup phase and does small style fixes)

----

 include/linux/sched.h |    4 +--
 kernel/softlockup.c   |   55 ++++++++++++++++++++++++++++----------------------
 kernel/timer.c        |    2 -
 3 files changed, 34 insertions(+), 27 deletions(-)
include/linux/sched.h +2 -2
Index: linux-rt.q/include/linux/sched.h
===================================================================
@@ -208,11 +208,11 @@ extern void update_process_times(int use
 extern void scheduler_tick(void);
 
 #ifdef CONFIG_DETECT_SOFTLOCKUP
-extern void softlockup_tick(struct pt_regs *regs);
+extern void softlockup_tick(void);
 extern void spawn_softlockup_task(void);
 extern void touch_softlockup_watchdog(void);
 #else
-static inline void softlockup_tick(struct pt_regs *regs)
+static inline void softlockup_tick(void)
 {
 }
 static inline void spawn_softlockup_task(void)
kernel/softlockup.c +31 -24

121 lines not on this page · raw message

kernel/timer.c +1 -1
Index: linux-rt.q/kernel/timer.c
===================================================================
@@ -914,6 +914,7 @@ static void run_timer_softirq(struct sof
 void run_local_timers(void)
 {
 	raise_softirq(TIMER_SOFTIRQ);
+	softlockup_tick();
 }
 
 /*
@@ -944,7 +945,6 @@ void do_timer(struct pt_regs *regs)
 	/* prevent loading jiffies before storing new jiffies_64 value. */
 	barrier();
 	update_times();
-	softlockup_tick(regs);
 }
 
 #ifdef __ARCH_WANT_SYS_ALARM
#2 lore raw
Quoted 11 lines · On Tue, 14 Mar 2006 22:40:49 +0100 Ingo Molnar wrote:
Quoted 8 lines · Randy.Dunlap

Hi,

(x86_64; 2.6.16-rc6; serial console configured but nothing connected to the serial port)

I'm seeing an occasional soft lockup, maybe in serial8250_console_write(). (drivers/serial/8250.c)

Quoted 6 lines
Quoted 2 lines

This function calls wait_for_xmitr() [inline], which in worst case can spin for 1.010 seconds. Could this be the cause of a soft lockup?

hm, it shouldnt cause that. Could you try the attached patch [which is the next-gen softlockup detector], do you get the message even with that one applied?

5/5 good boots with your new patch.
5/5 soft lockups without it.

Is this scheduled for post-2.6.16 ?

Thanks,
---
~Randy

#3 lore raw
Quoted 11 lines · Randy.Dunlap
Quoted 6 lines
Quoted 2 lines

This function calls wait_for_xmitr() [inline], which in worst case can spin for 1.010 seconds. Could this be the cause of a soft lockup?

hm, it shouldnt cause that. Could you try the attached patch [which is the next-gen softlockup detector], do you get the message even with that one applied?

5/5 good boots with your new patch.
5/5 soft lockups without it.

Is this scheduled for post-2.6.16 ?

yes, in theory. Andrew?

Ingo

#4 lore raw
Quoted 11 lines · Ingo Molnar
Quoted 7 lines · Randy.Dunlap
Quoted 6 lines
Quoted 2 lines

This function calls wait_for_xmitr() [inline], which in worst case can spin for 1.010 seconds. Could this be the cause of a soft lockup?

hm, it shouldnt cause that. Could you try the attached patch [which is the next-gen softlockup detector], do you get the message even with that one applied?

Quoted 7 lines
Quoted 4 lines

5/5 good boots with your new patch.
5/5 soft lockups without it.

Is this scheduled for post-2.6.16 ?

yes, in theory. Andrew?

Which? timer-irq-driven-soft-watchdog-cleanups.patch? Yup.