Spinlock recursion in hvc_poll
Quoted 5 lines · Alex Nixon
Quoted 5 lines
Quoted 5 lines
Well I say fixed - it just means I can proceed to a spinlock recursion
BUG()2 secs into the boot process, but it should be easier to track down with printks and a coherent stack dump at my disposal.
Quoted 7 lines
Quoted 2 lines
What's the backtrace on this?
I just turned
DEBUG_SPINLOCKSback on to try catch this bug again, and it seems to occur (very roughly) 1/10 of the time, with nothing changing between runs.
OK, I added some cc:s.
This is definitely a relatively new bug, because I was running with spinlock debugging on all the time a couple of months ago, with no problems.
Quoted 2 lines
Backtrace attached.
(Should paste it inline so it gets quoted in replies.)
Quoted 10 lines
[ 0.752696] Freeing unused kernel memory: 284k freed [ 0.772757] BUG: spinlock recursion on CPU#0, swapper/1 [ 0.772782] lock: eaca7000, .magic: dead4ead, .owner: swapper/1, .owner_cpu: 0 [ 0.772814] Pid: 1, comm: swapper Not tainted 2.6.27-rc1 #350 [ 0.772843] [<c021f48d>] spin_bug+0x7c/0x87 [ 0.772878] [<c021f5d5>] _raw_spin_lock+0x35/0xda [ 0.772904] [<c030c501>] _spin_lock_irqsave+0x3c/0x45 [ 0.772932] [<c027a394>] ? hvc_poll+0x15/0x1ac [ 0.772964] [<c027a394>] hvc_poll+0x15/0x1ac
OK, I suspect this is a result of Christian Borntraeger's patches to make hvc console cope without interrupts. It's certainly a suggestive set of changes in the right place in the code.
But I have to admit I didn't look at the patches very closely when they went by, so I'm just finger-pointing at the moment. Perhaps it's a pre-existing bug.
J
Quoted 42 lines
[ 0.772990] [<c0166b1f>] ? ftrace_record_ip+0x19f/0x1ee [ 0.773024] [<c027a6a2>] hvc_handle_interrupt+0xf/0x21 [ 0.773051] [<c015f7dc>] handle_IRQ_event+0x1f/0x4f [ 0.773080] [<c0160d1f>] handle_level_irq+0xc4/0xd1 [ 0.773107] [<c010b3ee>] do_IRQ+0x5c/0x7a [ 0.773133] [<c0263573>] xen_evtchn_do_upcall+0xad/0x109 [ 0.773164] [<c01096cb>] xen_do_upcall+0x7/0xc [ 0.773189] [<c0102227>] ? _stext+0x227/0x1000 [ 0.773219] [<c0105be3>] ? xen_force_evtchn_callback+0xf/0x14 [ 0.773253] [<c0106366>] check_events+0x8/0xe [ 0.773278] [<c0106297>] ? xen_irq_enable_direct_end+0x0/0x1 [ 0.773310] [<c018bc07>] ? cache_alloc_refill+0x251/0x4b4 [ 0.773342] [<c0105be3>] ? xen_force_evtchn_callback+0xf/0x14 [ 0.773375] [<c0106366>] ? check_events+0x8/0xe [ 0.773405] [<c018bef8>] __kmalloc+0x8e/0xd6 [ 0.773428] [<c010972b>] ? mcount_call+0x5/0xa [ 0.773457] [<c01c3490>] __proc_create+0x78/0xfb [ 0.773485] [<c01c380a>] proc_mkdir_mode+0x23/0x48 [ 0.773512] [<c01c3843>] proc_mkdir+0x14/0x16 [ 0.773536] [<c016112e>] register_irq_proc+0x70/0xca [ 0.773562] [<c0160013>] setup_irq+0x1bc/0x1f2 [ 0.773587] [<c01600d2>] request_irq+0x89/0xa6 [ 0.773614] [<c027a693>] ? hvc_handle_interrupt+0x0/0x21 [ 0.773645] [<c027a681>] notifier_add_irq+0x2f/0x41 [ 0.773670] [<c027a2aa>] hvc_open+0x6f/0xc9 [ 0.773693] [<c027a23b>] ? hvc_open+0x0/0xc9 [ 0.773720] [<c026b436>] tty_open+0x198/0x299 [ 0.773746] [<c0190fbe>] chrdev_open+0x12c/0x143 [ 0.773772] [<c018d170>] __dentry_open+0x113/0x201 [ 0.773797] [<c018d282>] nameidata_to_filp+0x24/0x38 [ 0.773823] [<c0190e92>] ? chrdev_open+0x0/0x143 [ 0.773852] [<c019845d>] do_filp_open+0x351/0x636 [ 0.773878] [<c010972b>] ? mcount_call+0x5/0xa [ 0.773907] [<c021f5f3>] ? _raw_spin_lock+0x53/0xda [ 0.773937] [<c021f50c>] ? _raw_spin_unlock+0x74/0x78 [ 0.773967] [<c018cf19>] ? get_unused_fd_flags+0xad/0xb7 [ 0.773997] [<c018cf68>] do_sys_open+0x45/0xbb [ 0.774022] [<c018d02a>] sys_open+0x23/0x2b [ 0.774046] [<c0103048>] init_post+0x2a/0xce [ 0.774070] [<c010967b>] ? kernel_thread_helper+0x7/0x10 [ 0.774101] =======================
Am Dienstag, 5. August 2008 schrieb Jeremy Fitzhardinge:
Quoted 3 lines
Quoted 2 lines · Alex Nixon
Quoted 2 lines
Quoted 2 lines
Well I say fixed - it just means I can proceed to a spinlock recursion
BUG()2 secs into the boot process, but it should be easier to track
down
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
with printks and a coherent stack dump at my disposal.
What's the backtrace on this?
I just turned
DEBUG_SPINLOCKSback on to try catch this bug again, and it seems to occur (very roughly) 1/10 of the time, with nothing changing
between
Quoted 1 line
Quoted 1 line
runs.
Ok, this is a guess, I dont fully understand the backtrace.
request_irq tries to call the handler if the IRQ is shared. The irq handler calls hvc_poll and hvc_kill which might take the same spinlock. Can you test it this patch fixes the problem?
--- drivers/char/hvc_console.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/char/hvc_console.c +2 -1
@@ -322,10 +322,11 @@ static int hvc_open(struct tty_struct *t hp->tty = tty; +spin_unlock_irqrestore (&hp->lock, flags); +if (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); -spin_unlock_irqrestore (&hp->lock, flags);/*
Quoted 10 lines · Christian Borntraeger
Am Dienstag, 5. August 2008 schrieb Jeremy Fitzhardinge:
Quoted 3 lines
Quoted 2 lines · Alex Nixon
Quoted 2 lines
Quoted 2 lines
Well I say fixed - it just means I can proceed to a spinlock recursion
BUG()2 secs into the boot processOk, this is a guess, I dont fully understand the backtrace.
request_irqtries to call the handler if the IRQ is shared. The irq handler callshvc_pollandhvc_killwhich might take the same spinlock. Can you test it this patch fixes the problem?
The patch appears to fix things.
- Alex
Quoted 24 lines
Signed-off-by Christian Borntraeger <[email protected]>--- drivers/char/hvc_console.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)Index:
linux-2.6/drivers/char/hvc_console.c
===================================================================drivers/char/hvc_console.c +2
-1@@ -322,10 +322,11 @@ static int hvc_open(struct tty_struct *t hp->tty = tty; +spin_unlock_irqrestore (&hp->lock, flags); +if (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); -spin_unlock_irqrestore (&hp->lock, flags);/*
Am Mittwoch, 6. August 2008 schrieb Alex Nixon:
Quoted 1 line
The patch appears to fix things.
Rusty,
can this patch go into 2.6.27-rc* via your queue?
---snip----
[PATCH] fix spinlock recursion in hvc_console
commit 611e097d7707741a336a0677d9d69bec40f29f3d
Author: Christian Borntraeger <[email protected]>hvc_console: rework setup to replace irq functions with callbacks introduced a spinlock recursion problem.
request_irq tries to call the handler if the IRQ is shared. The irq handler of hvc_console calls hvc_poll and hvc_kill which might take the hvc_struct spinlock. Therefore, we have to call request_irq outside the spinlock.
We can move the notifier_add safely outside the spinlock as ->data must not be changed by the backend. Otherwise, tty_hangup would fail anyway.
--- drivers/char/hvc_console.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/char/hvc_console.c +2 -3
@@ -322,11 +322,10 @@ static int hvc_open(struct tty_struct *t hp->tty = tty; -if (hp->ops->notifier_add) - rc = hp->ops->notifier_add (hp, hp->data); -spin_unlock_irqrestore (&hp->lock, flags); +if (hp->ops->notifier_add) + rc = hp->ops->notifier_add (hp, hp->data);/* * If the notifier fails we return an error. The tty layer
Quoted 6 lines · On Thursday 07 August 2008 17:18:34 Christian Borntraeger wrote:
Am Mittwoch, 6. August 2008 schrieb Alex Nixon:
Quoted 1 line
The patch appears to fix things.
Rusty,
can this patch go into 2.6.27-rc* via your queue?
Done, thanks.
Rusty.