hvc_console WARN() on current upstream
Firing that off here in case I don't get a chance to investigate this week... latest upstream with whatever I'm about to push into powerpc-next (but I doubt it's the pending stuff that's causing it) gives me that on a pseries machine:
------------[ cut here ]------------ Badness at /home/benh/kernels/linux-powerpc/kernel/mutex.c:135 NIP: c0000000004fe0dc LR: c0000000004fe0c0 CTR: c0000000002c4304 REGS: c00000000fffb660 TRAP: 0700 Not tainted (2.6.28-test) MSR: 8000000000021032 <ME,CE,IR,DR> CR: 28000082 XER: 20000003
TASK = c0000000007bd340[0] 'swapper' THREAD: c000000000874000 CPU: 0
GPR00: 0000000000000000 c00000000fffb8e0 c000000000877438 0000000000000001
GPR04: 0000000000000000 c0000000225f0928 0000000000000003 0000000008000001
GPR08: cf3cf3cf3cf3cf3d c000000000e6edc0 c000000000787634 c0000000008d45ac
GPR12: 0000000028000082 c0000000008b7300 c00000002254db20 0000000000000001
GPR16: 0000000000000003 c00000002254d940 c00000002254da70 c0000000225f0828
GPR20: c0000000225f0929 0000000000000000 c00000002254dee8 0000000000000000
GPR24: 0000000000000000 c0000000002c2604 0000000000000003 c0000000007bd340
GPR28: c00000002254dee8 c00000002254d800 c0000000007ef2b8 c00000002254dee8 NIP [c0000000004fe0dc] .mutex_lock_nested+0x90/0x408
LR [c0000000004fe0c0] .mutex_lock_nested+0x74/0x408
Call Trace:
[c00000000fffb9e0] [c0000000002c2604] .echo_set_canon_col+0x28/0x68
[c00000000fffba70] [c0000000002c4db4] .n_tty_receive_buf+0xcbc/0xfe0
[c00000000fffbc10] [c0000000002c80f8] .flush_to_ldisc+0x18c/0x24c
[c00000000fffbcd0] [c0000000002dd02c] .hvc_poll+0x2d0/0x328
[c00000000fffbde0] [c0000000002dd2e4] .hvc_handle_interrupt+0x14/0x3c
[c00000000fffbe50] [c0000000000999a4] .handle_IRQ_event+0x50/0xc0
[c00000000fffbef0] [c00000000009bdec] .handle_fasteoi_irq+0xf8/0x194
[c00000000fffbf90] [c000000000025950] .call_handle_irq+0x1c/0x2c
[c000000000877a30] [c00000000000cd18] .do_IRQ+0x104/0x1dc
[c000000000877ae0] [c000000000004800] hardware_interrupt_entry+0x18/0x98
--- Exception: 501 at .cpu_idle+0x104/0x190
LR = .cpu_idle+0x104/0x190
[c000000000877dd0] [c000000000011e58] .cpu_idle+0xf8/0x190 (unreliable)
[c000000000877e60] [c000000000500b24] .rest_init+0x7c/0x94
[c000000000877ee0] [c000000000715aec] .start_kernel+0x3f4/0x414
[c000000000877f90] [c000000000008368] .start_here_common+0x1c/0x34Instruction dump:
78290464 80090014 5409012f 41820028 4bd5ead5 60000000 2fa30000 419e0018 e93e8008 80090000 2f800000 409e0008 <0fe00000> 38000000 8b8d01da 980d01da
The machine still boots, so it's not a show stopper.
Cheers,
Ben.
Am Donnerstag, 8. Januar 2009 schrieb Benjamin Herrenschmidt:
Quoted 33 lines
------------[ cut here ]------------ Badness at /home/benh/kernels/linux-powerpc/kernel/mutex.c:135 NIP: c0000000004fe0dc LR: c0000000004fe0c0 CTR: c0000000002c4304 REGS: c00000000fffb660 TRAP: 0700 Not tainted (2.6.28-test) MSR: 8000000000021032 <ME,CE,IR,DR> CR: 28000082 XER: 20000003TASK = c0000000007bd340[0] 'swapper' THREAD:
c000000000874000CPU: 0
GPR00: 0000000000000000c00000000fffb8e0c0000000008774380000000000000001
GPR04: 0000000000000000c0000000225f09280000000000000003 0000000008000001
GPR08:cf3cf3cf3cf3cf3dc000000000e6edc0c000000000787634c0000000008d45ac
GPR12: 0000000028000082c0000000008b7300c00000002254db200000000000000001
GPR16: 0000000000000003c00000002254d940c00000002254da70c0000000225f0828
GPR20:c0000000225f09290000000000000000c00000002254dee80000000000000000
GPR24: 0000000000000000c0000000002c26040000000000000003c0000000007bd340
GPR28:c00000002254dee8c00000002254d800c0000000007ef2b8c00000002254dee8NIP [c0000000004fe0dc] .mutex_lock_nested+0x90/0x408
LR [c0000000004fe0c0] .mutex_lock_nested+0x74/0x408Call Trace: [c00000000fffb9e0] [c0000000002c2604] .echo_set_canon_col+0x28/0x68 [c00000000fffba70] [c0000000002c4db4] .n_tty_receive_buf+0xcbc/0xfe0 [c00000000fffbc10] [c0000000002c80f8] .flush_to_ldisc+0x18c/0x24c [c00000000fffbcd0] [c0000000002dd02c] .hvc_poll+0x2d0/0x328 [c00000000fffbde0] [c0000000002dd2e4] .hvc_handle_interrupt+0x14/0x3c [c00000000fffbe50] [c0000000000999a4] .handle_IRQ_event+0x50/0xc0 [c00000000fffbef0] [c00000000009bdec] .handle_fasteoi_irq+0xf8/0x194 [c00000000fffbf90] [c000000000025950] .call_handle_irq+0x1c/0x2c [c000000000877a30] [c00000000000cd18] .do_IRQ+0x104/0x1dc [c000000000877ae0] [c000000000004800] hardware_interrupt_entry+0x18/0x98 --- Exception: 501 at .cpu_idle+0x104/0x190 LR = .cpu_idle+0x104/0x190 [c000000000877dd0] [c000000000011e58] .cpu_idle+0xf8/0x190 (unreliable) [c000000000877e60] [c000000000500b24] .rest_init+0x7c/0x94 [c000000000877ee0] [c000000000715aec] .start_kernel+0x3f4/0x414 [c000000000877f90] [c000000000008368] .start_here_common+0x1c/0x34
Quoted 5 lines
Instruction dump:
78290464 800900145409012f418200284bd5ead5600000002fa30000419e0018e93e8008800900002f800000409e0008<0fe00000> 380000008b8d01da980d01daThe machine still boots, so it's not a show stopper.
Hmmm,
Seems that we are in interrupt, doing hvc_poll, which does tty_flip_buffer_push
which callsflush_to_ldisc
doing an_tty_receive_buf
and we will finally take a mutex in echo_set_canon_col
my first guess is that the following patch triggered the Badness:
git show a88a69c9
commit a88a69c91256418c5907c2f1f8a0ec0a36f9e6cc
Author: Joe Peterson <[email protected]> Date: Fri Jan 2 13:40:53 2009 +0000
n_tty: Fix loss of echoed characters and remove bkl from n_tty
I have no idea how to fix this.
Joe Peterson CCed.
Christian
Also including Alan, he's the current tty guru
Quoted 65 lines · Christian Borntraeger
Am Donnerstag, 8. Januar 2009 schrieb Benjamin Herrenschmidt:
Quoted 33 lines
------------[ cut here ]------------ Badness at /home/benh/kernels/linux-powerpc/kernel/mutex.c:135 NIP: c0000000004fe0dc LR: c0000000004fe0c0 CTR: c0000000002c4304 REGS: c00000000fffb660 TRAP: 0700 Not tainted (2.6.28-test) MSR: 8000000000021032 <ME,CE,IR,DR> CR: 28000082 XER: 20000003TASK = c0000000007bd340[0] 'swapper' THREAD:
c000000000874000CPU: 0
GPR00: 0000000000000000c00000000fffb8e0c0000000008774380000000000000001
GPR04: 0000000000000000c0000000225f09280000000000000003 0000000008000001
GPR08:cf3cf3cf3cf3cf3dc000000000e6edc0c000000000787634c0000000008d45ac
GPR12: 0000000028000082c0000000008b7300c00000002254db200000000000000001
GPR16: 0000000000000003c00000002254d940c00000002254da70c0000000225f0828
GPR20:c0000000225f09290000000000000000c00000002254dee80000000000000000
GPR24: 0000000000000000c0000000002c26040000000000000003c0000000007bd340
GPR28:c00000002254dee8c00000002254d800c0000000007ef2b8c00000002254dee8NIP [c0000000004fe0dc] .mutex_lock_nested+0x90/0x408
LR [c0000000004fe0c0] .mutex_lock_nested+0x74/0x408Call Trace: [c00000000fffb9e0] [c0000000002c2604] .echo_set_canon_col+0x28/0x68 [c00000000fffba70] [c0000000002c4db4] .n_tty_receive_buf+0xcbc/0xfe0 [c00000000fffbc10] [c0000000002c80f8] .flush_to_ldisc+0x18c/0x24c [c00000000fffbcd0] [c0000000002dd02c] .hvc_poll+0x2d0/0x328 [c00000000fffbde0] [c0000000002dd2e4] .hvc_handle_interrupt+0x14/0x3c [c00000000fffbe50] [c0000000000999a4] .handle_IRQ_event+0x50/0xc0 [c00000000fffbef0] [c00000000009bdec] .handle_fasteoi_irq+0xf8/0x194 [c00000000fffbf90] [c000000000025950] .call_handle_irq+0x1c/0x2c [c000000000877a30] [c00000000000cd18] .do_IRQ+0x104/0x1dc [c000000000877ae0] [c000000000004800] hardware_interrupt_entry+0x18/0x98 --- Exception: 501 at .cpu_idle+0x104/0x190 LR = .cpu_idle+0x104/0x190 [c000000000877dd0] [c000000000011e58] .cpu_idle+0xf8/0x190 (unreliable) [c000000000877e60] [c000000000500b24] .rest_init+0x7c/0x94 [c000000000877ee0] [c000000000715aec] .start_kernel+0x3f4/0x414 [c000000000877f90] [c000000000008368] .start_here_common+0x1c/0x34Quoted 5 lines
Instruction dump:
78290464 800900145409012f418200284bd5ead5600000002fa30000419e0018e93e8008800900002f800000409e0008<0fe00000> 380000008b8d01da980d01daThe machine still boots, so it's not a show stopper.
Hmmm,
Seems that we are in interrupt, doing
hvc_poll, which doestty_flip_buffer_push
which callsflush_to_ldisc
doing an_tty_receive_buf
and we will finally take a mutex inecho_set_canon_colmy first guess is that the following patch triggered the Badness:
git showa88a69c9
commita88a69c91256418c5907c2f1f8a0ec0a36f9e6ccAuthor: Joe Peterson <[email protected]> Date: Fri Jan 2 13:40:53 2009 +0000
n_tty: Fix loss of echoed characters and remove bkl fromn_ttyI have no idea how to fix this.
Joe Peterson CCed.Christian
Quoted 2 lines
Quoted 2 lines
Seems that we are in interrupt, doing
hvc_poll, which doestty_flip_buffer_push
Which means that someone has tty->low_latency set and is calling tty_flip_buffer_push in an IRQ. That has never been allowed or safe, and now it hurts ;)
/**
* tty_flip_buffer_push - terminal
* @tty: tty to push
*
* Queue a push of the terminal flip buffers to the line discipline.
This
* function must not be called from IRQ context if tty->low_latency
is set *
* In the event of the queue being busy for flipping the work will be
* held off and retried later.
*
* Locking: tty buffer lock. Driver locks in low latency mode.
*/
That comment has been there for some years in varying formats
cover hvc_console updates was Re: [BUG] hvc_console WARN() on current upstream
Quoted 23 lines · Alan Cox
Quoted 2 lines · Benjamin Herrenschmidt
Seems that we are in interrupt, doing
hvc_poll, which doestty_flip_buffer_pushWhich means that someone has
tty->low_latencyset and is callingtty_flip_buffer_pushin an IRQ. That has never been allowed or safe, and now it hurts ;)/**
*tty_flip_buffer_push- terminal
* @tty: tty to push
*
* Queue a push of the terminal flip buffers to the line discipline.
This
* function must not be called from IRQ context iftty->low_latency
is set *
* In the event of the queue being busy for flipping the work will be
* held off and retried later.
*
* Locking: tty buffer lock. Driver locks in low latency mode.
*/
Quoted 2 lines
That comment has been there for some years in varying formats
I actually was preparing a patch for this problem after I had encountered
the a deadlock due to this. That is in the first patch. I then found
and made a few more cleanups, although I might have reordered the rest.
The history for setting low_latency is in the changelog of the first patch..
milton
PATCH 1/4 hvc_console: do not set low_latency
hvc_console is setting low_latency unconditionally, but some clients are interrupt driven and will call hvc_poll from irq context. This will cause tty_flip_buffer_push to be called from irq context, and it very clearly states it must not be called from IRQ when low_latency is specified.
Looking back through history:
v2.6.16-rc1 via 33f0f88f1c51ae5c2d593d26960c760ea154c2e2
[PATCH] TTY layer buffering revamp
added this new api.
v2.6.16-rc3 via 8977d929e49021d9a6e031310aab01fa72f849c2
[PATCH] tty buffering stall fix
claims to fix a stall discovered with hvc_console
v2.6.16-rc5 via fb5c594c2acc441f0d2d8f457484a0e0e9285db3
[PATCH] Fix race condition in hvc console.
said set this flag to avoid a stall problem, and was merged through the powerpc arch tree.
Without searching for email discussions, it would appear to be an overlapping "fix", but one that did not consider all users.
---
This version continues to set low_latency if irqs are not flagged to
be in use, as requested by paulus. Do all hvc drivers identify the
interrupt this way? (A quick look at hvc_iucv says they lock to bh and are not irq driven, the rest would have used the irq before that patch).
Having the flag set for purely polled drivers will save delaying the work when receiving input for 1 jiffie.
drivers/char/hvc_console.c +2 -1
@@ -318,7 +318,8 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ hp->tty = tty;
Re: [PATCH 1/4] hvc_console: do not set low_latency
Quoted 4 lines
v2.6.16-rc1 via
33f0f88f1c51ae5c2d593d26960c760ea154c2e2
[PATCH] TTY layer buffering revampadded this new api.
No - tty_flip_buffer_push is from 2.1.66 and with the same constraints from the day it was added.
Quoted 9 lines
Having the flag set for purely polled drivers will save delaying the work when receiving input for 1 jiffie.
Index:
work.git/drivers/char/hvc_console.c
===================================================================
drivers/char/hvc_console.c static int hvc_open(struct tty_struct *t · line 324
Quoted 8 lines
drivers/char/hvc_console.c +2
-1@@ -318,7 +318,8 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ hp->tty = tty;
Looks good to me
Re: [PATCH 1/4] hvc_console: do not set low_latency
Quoted 8 lines · Alan Cox
Quoted 4 lines
v2.6.16-rc1 via
33f0f88f1c51ae5c2d593d26960c760ea154c2e2
[PATCH] TTY layer buffering revampadded this new api.
No -
tty_flip_buffer_pushis from 2.1.66 and with the same constraints from the day it was added.
Yes but wrappers were added and this this and many ohter drivers were converted to use them:
And they were slightly buggy leading to the buggy workaround.
Quoted 14 lines
Quoted 14 lines
Having the flag set for purely polled drivers will save delaying the work when receiving input for 1 jiffie.
Index:
work.git/drivers/char/hvc_console.c
===================================================================
---work.git.orig/drivers/char/hvc_console.c2009-01-08
03:01:24.000000000 -0600
+++work.git/drivers/char/hvc_console.c2009-01-08 03:01:51.000000000
-0600drivers/char/hvc_console.c +0
-0@@ -318,7 +318,8 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */
Quoted 8 lines
drivers/char/hvc_console.c static int hvc_open(struct tty_struct *t · line 324
Quoted 6 lines
drivers/char/hvc_console.c +2
-1tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ hp->tty = tty;Looks good to me
Thanks, I guess that is an Ack?
milton
Re: [PATCH 1/4] hvc_console: do not set low_latency
In reply to Milton MillerAm Donnerstag 08 Januar 2009 schrieb Milton Miller:
Quoted 32 lines
hvc_consoleis settinglow_latencyunconditionally, but some clients are interrupt driven and will callhvc_pollfrom irq context. This will causetty_flip_buffer_pushto be called from irq context, and it very clearly states it must not be called from IRQ whenlow_latencyis specified.Looking back through history:
v2.6.16-rc1 via33f0f88f1c51ae5c2d593d26960c760ea154c2e2
[PATCH] TTY layer buffering revampadded this new api.
v2.6.16-rc3 via
8977d929e49021d9a6e031310aab01fa72f849c2
[PATCH] tty buffering stall fixclaims to fix a stall discovered with
hvc_consolev2.6.16-rc5 via
fb5c594c2acc441f0d2d8f457484a0e0e9285db3
[PATCH] Fix race condition in hvc console.said set this flag to avoid a stall problem, and was merged through the powerpc arch tree.
Without searching for email discussions, it would appear to be an overlapping "fix", but one that did not consider all users.
---
This version continues to setlow_latencyif irqs are not flagged to
be in use, as requested by paulus. Do all hvc drivers identify the
interrupt this way? (A quick look athvc_iucvsays they lock to bh and are not irq driven, the rest would have used the irq before that patch).
Quoted 7 lines
Having the flag set for purely polled drivers will save delaying the work when receiving input for 1 jiffie.
Index:
work.git/drivers/char/hvc_console.c
===================================================================
---work.git.orig/drivers/char/hvc_console.c2009-01-08
03:01:24.000000000 -0600
Review static int hvc_open(struct tty_struct *t · line 325
Quoted 12 lines
+++
work.git/drivers/char/hvc_console.c2009-01-08 03:01:51.000000000 -0600(quoted diff) +2
-1@@ -318,7 +318,8 @@ static int hvc_open(struct tty_struct *t } /* else count == 0 */ tty->driver_data = hp; - tty->low_latency = 1; /* Makes flushes to ldisc synchronous. */ + if (!hp->irq_requested) + tty->low_latency = 1; /* Makes flushes to ldisc synchronous. */ hp->tty = tty;
This wont work, since the call to notifier_add is done later:
What about:
--- drivers/char/hvc_console.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/char/hvc_console.c +3 -2
@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ - hp->tty = tty;spin_unlock_irqrestore (&hp->lock, flags);
@@ -327,6 +325,9 @@ static int hvc_open(struct tty_struct *tif (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +/* * If the notifier fails we return an error. The tty layer * will call hvc_close() after a failed open but we don't want to clean
Re: [PATCH 1/4] hvc_console: do not set low_latency
Am Dienstag 13 Januar 2009 schrieb Christian Borntraeger:
Quoted 19 lines
drivers/char/hvc_console.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/drivers/char/hvc_console.c
===================================================================drivers/char/hvc_console.c +0
-2@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ - hp->tty = tty;spin_unlock_irqrestore (&hp->lock, flags);
drivers/char/hvc_console.c static int hvc_open(struct tty_struct *t · line 333
Quoted 9 lines
drivers/char/hvc_console.c +3
-0@@ -327,6 +325,9 @@ static int hvc_open(struct tty_struct *tif (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +/* * If the notifier fails we return an error. The tty layer * will call hvc_close() after a failed open but we don't want to clean
Just in case:
Hendrik Brueckner
Acked-byRe: [PATCH 1/4] hvc_console: do not set low_latency
In reply to Christian BorntraegerHere is yet another bug trace related to the low_latency issue, that is experienced using the hvc_iucv backend. The hvc_iucv backend now also uses irq notifiers.
The bug trace below appears if hvc_kick() is called.
Quoted 5 lines · Christian Borntraeger
Am Donnerstag 08 Januar 2009 schrieb Milton Miller:
Quoted 4 lines
hvc_consoleis settinglow_latencyunconditionally, but some clients are interrupt driven and will callhvc_pollfrom irq context. This will causetty_flip_buffer_pushto be called from irq context, and it very clearly states it must not be called from IRQ whenlow_latencyis specified.
It seems that if low_latency is set, tty_flip_buffer_push() should also not be called within an atomic context, because echo_char_raw() and other echo* calls might sleep.
Christian's patch solves this problem for irq driven backends. However, there might be still a problem with polled backends since the khvcd() thread calls hvc_poll() while hvc_structs_lock is held.
I think the hvc_udbg backend is based on polling.
David, could you check if you experience any problems with your hvc_udbg backend on latest git.
BUG: sleeping function called from invalid context at /root/cvs/linux-2.6.git/kernel/mutex.c:207
in_atomic(): 1, irqs_disabled(): 0, pid: 748, name: khvcd
1 lock held by khvcd/748:
#0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12c
CPU: 0 Not tainted 2.6.29-rc1git #29
Process khvcd (pid: 748, task: 000000002fb9a480, ksp: 000000002f66bd78)
070000000000000a 000000002f66ba00 0000000000000002 (null)
000000002f66baa0 000000002f66ba18 000000002f66ba18 0000000000104f08
ffffffffffffc000 000000002f66bd78 (null) (null)
000000002f66ba00 000000000000000c 000000002f66ba00 000000002f66ba70
0000000000466af8 0000000000104f08 000000002f66ba00 000000002f66ba50
Call Trace:
([<0000000000104e7c>] show_trace+0x138/0x158)
[<0000000000104f62>] show_stack+0xc6/0xf8
[<0000000000105740>] dump_stack+0xb0/0xc0
[<000000000013144a>] __might_sleep+0x14e/0x17c
[<000000000045e226>] mutex_lock_nested+0x42/0x3b4
[<00000000002c443e>] echo_char_raw+0x3a/0x9c
[<00000000002c688c>] n_tty_receive_buf+0x1154/0x1208
[<00000000002ca0a2>] flush_to_ldisc+0x152/0x220
[<00000000002ca1da>] tty_flip_buffer_push+0x6a/0x90
[<00000000002cea74>] hvc_poll+0x244/0x2c8
[<00000000002ceb68>] khvcd+0x70/0x12c
[<000000000015bbd0>] kthread+0x68/0xa0
[<0000000000109d5a>] kernel_thread_starter+0x6/0xc
[<0000000000109d54>] kernel_thread_starter+0x0/0xc
1 lock held by khvcd/748:
#0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12cQuoted 20 lines
This wont work, since the call to
notifier_addis done later:
What about:--- drivers/char/hvc_console.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/drivers/char/hvc_console.c
===================================================================drivers/char/hvc_console.c +0
-2@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ - hp->tty = tty;spin_unlock_irqrestore (&hp->lock, flags);
drivers/char/hvc_console.c static int hvc_open(struct tty_struct *t · line 333
Quoted 9 lines
drivers/char/hvc_console.c +3
-0@@ -327,6 +325,9 @@ static int hvc_open(struct tty_struct *tif (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +/* * If the notifier fails we return an error. The tty layer * will call hvc_close() after a failed open but we don't want to clean
Re: [PATCH 1/4] hvc_console: do not set low_latency
Quoted 15 lines · Hendrik Brueckner
Here is yet another bug trace related to the
low_latencyissue, that is experienced using thehvc_iucvbackend. Thehvc_iucvbackend now also uses irq notifiers.
The bug trace below appears ifhvc_kick()is called.Quoted 5 lines · Christian Borntraeger
Am Donnerstag 08 Januar 2009 schrieb Milton Miller:
Quoted 4 lines
hvc_consoleis settinglow_latencyunconditionally, but some clients are interrupt driven and will callhvc_pollfrom irq context. This will causetty_flip_buffer_pushto be called from irq context, and it very clearly states it must not be called from IRQ whenlow_latencyis specified.It seems that if
low_latencyis set,tty_flip_buffer_push()should also not be called within an atomic context, becauseecho_char_raw()and other echo* calls might sleep.
Quoted 8 lines
Christian's patch solves this problem for irq driven backends. However, there might be still a problem with polled backends since the
khvcd()thread callshvc_poll()whilehvc_structs_lockis held.I think the
hvc_udbgbackend is based on polling.
David, could you check if you experience any problems with yourhvc_udbgbackend on latest git.
So the simplest is to never set it.
milton
Quoted 67 lines
BUG: sleeping function called from invalid context at /root/cvs/linux-2.6.git/kernel/mutex.c:207 in_atomic(): 1, irqs_disabled(): 0, pid: 748, name: khvcd 1 lock held by khvcd/748: #0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12c CPU: 0 Not tainted 2.6.29-rc1git #29 Process khvcd (pid: 748, task: 000000002fb9a480, ksp: 000000002f66bd78) 070000000000000a 000000002f66ba00 0000000000000002 (null) 000000002f66baa0 000000002f66ba18 000000002f66ba18 0000000000104f08 ffffffffffffc000 000000002f66bd78 (null) (null) 000000002f66ba00 000000000000000c 000000002f66ba00 000000002f66ba70 0000000000466af8 0000000000104f08 000000002f66ba00 000000002f66ba50 Call Trace: ([<0000000000104e7c>] show_trace+0x138/0x158) [<0000000000104f62>] show_stack+0xc6/0xf8 [<0000000000105740>] dump_stack+0xb0/0xc0 [<000000000013144a>] __might_sleep+0x14e/0x17c [<000000000045e226>] mutex_lock_nested+0x42/0x3b4 [<00000000002c443e>] echo_char_raw+0x3a/0x9c [<00000000002c688c>] n_tty_receive_buf+0x1154/0x1208 [<00000000002ca0a2>] flush_to_ldisc+0x152/0x220 [<00000000002ca1da>] tty_flip_buffer_push+0x6a/0x90 [<00000000002cea74>] hvc_poll+0x244/0x2c8 [<00000000002ceb68>] khvcd+0x70/0x12c [<000000000015bbd0>] kthread+0x68/0xa0 [<0000000000109d5a>] kernel_thread_starter+0x6/0xc [<0000000000109d54>] kernel_thread_starter+0x0/0xc 1 lock held by khvcd/748: #0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12cQuoted 29 lines
This wont work, since the call to
notifier_addis done later:
What about:--- drivers/char/hvc_console.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/drivers/char/hvc_console.c
===================================================================drivers/char/hvc_console.c +3
-2@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ - hp->tty = tty;spin_unlock_irqrestore (&hp->lock, flags);@@ -327,6 +325,9 @@ static int hvc_open(struct tty_struct *tif (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +/* * If the notifier fails we return an error. The tty layer * will call hvc_close() after a failed open but we don't want to Quoted 1 line
clean
Re: [PATCH 1/4] hvc_console: do not set low_latency
Quoted 2 lines
So the simplest is to never set it.
I already applied your previous patch. Please send an incremental fix.
Cheers,
Ben.
Quoted 70 lines
milton
Quoted 68 lines
BUG: sleeping function called from invalid context at
/root/cvs/linux-2.6.git/kernel/mutex.c:207in_atomic(): 1,irqs_disabled(): 0, pid: 748, name: khvcd 1 lock held by khvcd/748:
#0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>]khvcd+0x58/0x12c CPU: 0 Not tainted 2.6.29-rc1git #29 Process khvcd (pid: 748, task: 000000002fb9a480, ksp: 000000002f66bd78) 070000000000000a 000000002f66ba00 0000000000000002 (null) 000000002f66baa0 000000002f66ba18 000000002f66ba18 0000000000104f08 ffffffffffffc000 000000002f66bd78 (null) (null) 000000002f66ba00 000000000000000c 000000002f66ba00 000000002f66ba70 0000000000466af8 0000000000104f08 000000002f66ba00 000000002f66ba50 Call Trace: ([<0000000000104e7c>] show_trace+0x138/0x158) [<0000000000104f62>] show_stack+0xc6/0xf8 [<0000000000105740>] dump_stack+0xb0/0xc0 [<000000000013144a>] __might_sleep+0x14e/0x17c [<000000000045e226>] mutex_lock_nested+0x42/0x3b4 [<00000000002c443e>] echo_char_raw+0x3a/0x9c [<00000000002c688c>] n_tty_receive_buf+0x1154/0x1208 [<00000000002ca0a2>] flush_to_ldisc+0x152/0x220 [<00000000002ca1da>] tty_flip_buffer_push+0x6a/0x90 [<00000000002cea74>] hvc_poll+0x244/0x2c8 [<00000000002ceb68>] khvcd+0x70/0x12c [<000000000015bbd0>] kthread+0x68/0xa0 [<0000000000109d5a>] kernel_thread_starter+0x6/0xc [<0000000000109d54>] kernel_thread_starter+0x0/0xc 1 lock held by khvcd/748: #0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12cQuoted 29 lines
This wont work, since the call to
notifier_addis done later:
What about:--- drivers/char/hvc_console.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)Index:
linux-2.6/drivers/char/hvc_console.c
===================================================================drivers/char/hvc_console.c +3
-2@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ - hp->tty = tty;spin_unlock_irqrestore (&hp->lock, flags);@@ -327,6 +325,9 @@ static int hvc_open(struct tty_struct *tif (hp->ops->notifier_add) rc = hp->ops->notifier_add (hp, hp->data); +if (!hp->irq_requested) + tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ +/* * If the notifier fails we return an error. The tty layer * will call hvc_close() after a failed open but we don't want to Quoted 1 line
clean
Re: [PATCH 1/4 v2] hvc_console: remove tty->low_latency
Quoted 3 lines · Benjamin Herrenschmidt
Quoted 1 line
So the simplest is to never set it.
I already applied your previous patch. Please send an incremental fix.
Here is the incremental patch for your powerpc tree along with a summary as patch description.
Regards,
Hendrik
--
This patch removes the tty->low_latency setting.
For irq based hvc_console backends the tty->low_latency must be set to 0, because the tty_flip_buffer_push() function must not be called from IRQ context (see drivers/char/tty_buffer.c).
For polled backends, the low_latency setting causes the bug trace below, because tty_flip_buffer_push() is called within an atomic context and subsequent calls might sleep due to mutex_lock.
BUG: sleeping function called from invalid context at /root/cvs/linux-2.6.git/kernel/mutex.c:207
in_atomic(): 1, irqs_disabled(): 0, pid: 748, name: khvcd
1 lock held by khvcd/748:
#0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12c
CPU: 0 Not tainted 2.6.29-rc1git #29
Process khvcd (pid: 748, task: 000000002fb9a480, ksp: 000000002f66bd78)
070000000000000a 000000002f66ba00 0000000000000002 (null)
000000002f66baa0 000000002f66ba18 000000002f66ba18 0000000000104f08
ffffffffffffc000 000000002f66bd78 (null) (null)
000000002f66ba00 000000000000000c 000000002f66ba00 000000002f66ba70
0000000000466af8 0000000000104f08 000000002f66ba00 000000002f66ba50
Call Trace:
([<0000000000104e7c>] show_trace+0x138/0x158)
[<0000000000104f62>] show_stack+0xc6/0xf8
[<0000000000105740>] dump_stack+0xb0/0xc0
[<000000000013144a>] __might_sleep+0x14e/0x17c
[<000000000045e226>] mutex_lock_nested+0x42/0x3b4
[<00000000002c443e>] echo_char_raw+0x3a/0x9c
[<00000000002c688c>] n_tty_receive_buf+0x1154/0x1208
[<00000000002ca0a2>] flush_to_ldisc+0x152/0x220
[<00000000002ca1da>] tty_flip_buffer_push+0x6a/0x90
[<00000000002cea74>] hvc_poll+0x244/0x2c8
[<00000000002ceb68>] khvcd+0x70/0x12c
[<000000000015bbd0>] kthread+0x68/0xa0
[<0000000000109d5a>] kernel_thread_starter+0x6/0xc
[<0000000000109d54>] kernel_thread_starter+0x0/0xc
1 lock held by khvcd/748:
#0: (hvc_structs_lock){--..}, at: [<00000000002ceb50>] khvcd+0x58/0x12c--- drivers/char/hvc_console.c | 2 -- 1 file changed, 2 deletions(-)
drivers/char/hvc_console.c +0 -2
@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; -if (!hp->irq_requested) - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ hp->tty = tty;
Christian Borntraeger
Acked-byRe: [PATCH 1/4 v2] hvc_console: remove tty->low_latency
Quoted 9 lines
Signed-off-by Hendrik Brueckner <[email protected]>--- drivers/char/hvc_console.c | 2 -- 1 file changed, 2 deletions(-)
drivers/char/hvc_console.c static int hvc_open(struct tty_struct *t · line 323
Quoted 8 lines
drivers/char/hvc_console.c +0
-2@@ -318,8 +318,6 @@ static int hvc_open(struct tty_struct *t }/* else count == 0 */ tty->driver_data = hp; -if (!hp->irq_requested) - tty->low_latency =1 ;/* Makes flushes to ldisc synchronous. */ hp->tty = tty;
PATCH 2/4 hvc_console: use kzalloc
In reply to Milton MillerReplace kmalloc + memset with kzalloc.
drivers/char/hvc_console.c +1 -3
@@ -765,13 +765,11 @@ struct hvc_struct __devinit *hvc_alloc(ureturn ERR_PTR (err); } - hp =kmalloc (ALIGN (sizeof (*hp),sizeof (long )) + outbuf_size, + hp =kzalloc (ALIGN (sizeof (*hp),sizeof (long )) + outbuf_size,GFP_KERNEL );if (!hp)return ERR_PTR (-ENOMEM ); -memset (hp,0x00 ,sizeof (*hp)); - hp->vtermno = vtermno; hp->data = data; hp->ops = ops;
PATCH 3/4 hvc_console: free_irq only if request_irq was successful
In reply to Milton MillerOnly call free_irq if we marked the request_irq has having succeeded instead of whenever the the sub-driver identified the interrupt to use.
---
Appears to be a bugfix but might use a bit of time in -next.
This code was created in 2.6.28 to allow s390 to build without adding ifdefs. Many hvc-console sub-drivers have a single channel and are not modules.
drivers/char/hvc_irq.c +1 -1
@@ -37,7 +37,7 @@ int notifier_add_irq(struct hvc_struct *void notifier_del_irq (struct hvc_struct *hp,int irq) { -if (!irq) +if (!hp->irq_requested)return ;free_irq (irq, hp); hp->irq_requested =0 ;
Re: [PATCH 3/4] hvc_console: free_irq only if request_irq was successful
Am Donnerstag, 8. Januar 2009 schrieb Milton Miller:
Quoted 7 lines
Only call
free_irqif we marked therequest_irqhas having succeeded instead of whenever the the sub-driver identified the interrupt to use.Signed-off-by Milton Miller <[email protected]>---
Appears to be a bugfix but might use a bit of time in -next. This code was created in 2.6.28 to allow s390 to build without adding
ifdefs.
Quoted 8 lines
Many hvc-console sub-drivers have a single channel and are not modules.
Index:
work.git/drivers/char/hvc_irq.c
===================================================================
drivers/char/hvc_irq.c int notifier_add_irq(struct hvc_struct * · line 43
Quoted 8 lines
drivers/char/hvc_irq.c +1
-1@@ -37,7 +37,7 @@ int notifier_add_irq(struct hvc_struct *void notifier_del_irq (struct hvc_struct *hp,int irq) { -if (!irq) +if (!hp->irq_requested)return ;free_irq (irq, hp); hp->irq_requested =0 ;
Looks sane.
PATCH 4/4 hvc_console: comment mb and make it an smp_ one
In reply to Milton MillerI remember some history on this barrier. There was a race between open via /dev/console and the tty being fully setup. Its also why there is a temporary variable and the global is assigned at the end of the function.
drivers/char/hvc_console.c +5 -2
@@ -875,8 +875,11 @@ static int hvc_init(void) goto stop_thread; } -/* FIXME: This mb() seems completely random. Remove it. */ -mb (); +/* +* Make sure tty is fully registered before allowing it to be +* found by hvc_console_device. +*/ +smp_mb (); hvc_driver = drv;return 0 ;
Quoted 6 lines · Alan Cox
Quoted 2 lines
Quoted 2 lines
Seems that we are in interrupt, doing
hvc_poll, which doestty_flip_buffer_pushWhich means that someone has
tty->low_latencyset and is callingtty_flip_buffer_pushin an IRQ. That has never been allowed or safe, and now it hurts ;)
Heh, allright :-) I'll see where that flag is set and clear it when using IRQs.
Quoted 1 line
That comment has been there for some years in varying formats
Possibly, I didn't write that code, thanks for pointing me to the problem.
Cheers,
Ben.