2.6.31 regression: system hang after pptp connection established
Hi.
After pptp connection is established my 2.6.31 system freezes while 2.6.30 works as expected. Bissecting gave me the following result:
commit ac89a9174decf343de049a06fad75681f71890eb
Author: Linus Torvalds <[email protected]> Date: Sat Sep 5 13:27:10 2009 -0700
pty: don't limit the writes to 'pty_space()' inside 'pty_write()'
and looks like reverting this patch from 2.6.31 fixes the problem.
Other observations: It's hard to say when exactly system hangs - it hangs not immediatly, sometimes when I try to send some traffic, sometimes when I switch ppp connection off. Hang is not complete: mouse cursor keeps moving in Xorg, but every click gets no respond, I'm unable to start new programs, in open xconsoles it's possible to input something but after I press enter consoles hang too. Also there is no way out of X (ctrl+alt+FN combo does not work) and connected to this computer ssh consoles hang too (again, it's possible to put ls there but after that it hangs). No new ssh connections possible due to time out.
In hope to get any oops I've started netconsole but at hang no new ouput was there. I've managed to gather some information with SysRq (it's gzipped in attachment) but I'm not sure how useful it is.
I've tried to establish pptp connection both over wireless and wired connections and system hanged with both, so it looks like networking drivers are not the reason here. BTW, I'm using networkmanager to establish connection.
gzipped kernel config is in attachment.
Is this problem known? Does anybody experience same problem? Do you have a fix? :)
Signature · 1 line
Peter.
Quoted 11 lines · Peter Volkov
After pptp connection is established my 2.6.31 system freezes while 2.6.30 works as expected. Bissecting gave me the following result:
commit
ac89a9174decf343de049a06fad75681f71890ebAuthor: Linus Torvalds <[email protected]> Date: Sat Sep 5 13:27:10 2009 -0700pty: don't limit the writes to '
pty_space()' inside 'pty_write()'and looks like reverting this patch from 2.6.31 fixes the problem.
Hmm. The only thing it should cause is that pty_write() will effectively allow a larger buffer for writes (limited to ~64kB rather than 8kB).
But considering how fragile ppp has been, I guess I shouldn't be surprised that this can cause a hang in itself.
Quoted 3 lines
In hope to get any oops I've started netconsole but at hang no new ouput was there. I've managed to gather some information with SysRq (it's gzipped in attachment) but I'm not sure how useful it is.
It's interesting, but I don't know how _useful_ it is.
What's interesting about it is that it shows a problem, but the problem it shows would seem to have nothing at all to do with ppp or networking or pty's. The problem seems to be processes stuck in disk-wait:
events/0 D ffff88007d0c7b50 0 7 2 0x00000000 events/0 D ffff88007d0c7b50 0 7 2 0x00000000 kacpi_notify D ffff88007d2ffbe8 0 170 2 0x00000000 khubd D ffff88007d211ae8 0 260 2 0x00000000 pdflush D ffff88007d26bd40 0 326 2 0x00000000 kjournald D ffff88007b2f3df8 0 3361 2 0x00000000 kjournald D ffff88007c65ddf8 0 3362 2 0x00000000 reiserfs/0 D [<ffffffff810725f7>] ? delayacct_end+0x81/0x8c events/0 D ffff88007d0c7b50 0 7 2 0x00000000 kacpi_notify D ffff88007d2ffbe8 0 170 2 0x00000000 khubd D ffff88007d211ae8 0 260 2 0x00000000 pdflush D ffff88007d26bd40 0 326 2 0x00000000 kjournald D ffff88007b2f3df8 0 3361 2 0x00000000 kjournald D ffff88007c65ddf8 0 3362 2 0x00000000
which explains your symptoms - hung X (with just cursor moving) and ssh's hanging.
It's just that while it all explains your symptoms, none of the above should have anything what-so-ever to do with pty's!
pdflush, for example, seems to be stuck waiting for &jl->j_commit_mutex in reiserfs. Odd. It really looks like you have something stuck waiting for IO.
But your CPU 1 backtrace looks relevant, and seems hung on a spinlock in tty_buffer_request_room() and has that pty_write() thing there. I'm not seeing why the 'D' states above happen, though.
Quoted 9 lines
I've tried to establish pptp connection both over wireless and wired connections and system hanged with both, so it looks like networking drivers are not the reason here. BTW, I'm using networkmanager to establish connection.
gzipped kernel config is in attachment.
Is this problem known? Does anybody experience same problem? Do you have a fix? :)
Not a known problem, but it's entirely possible that there is some bug in the "tty buffer out of memory" handling - that nobody has ever seen because in practice everybody always hit other limits first.
Let me look at it a bit, and see if I can come up with test patches for you.
Linus
Quoted 4 lines · Linus Torvalds
What's interesting about it is that it shows a problem, but the problem it shows would seem to have nothing at all to do with ppp or networking or pty's. The problem seems to be processes stuck in disk-wait:
Ahh. I think I see what may be going on.
Somebody got a filesystem mutex, and then went to sleep due to IO. Then pptp comes in, and seems to be stuck in a loop in kernel space, and it seems to be stuck with preemption off.
So one CPU is stuck, and the thing that we want to run is on the same run-queue, and not preempting. An looking at your CPU#1 trace, it's likely looping in ppp_async_push().
And that whole loop is insane (and very prone to infinite loops), but it also depends on that tty wakeup() thing.
Does this patch make a difference? Make sure to _not_ try to do the whole wakeup thing if we couldn't actually insert anything into the tty buffers.
Linus
--- drivers/char/pty.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
drivers/char/pty.c +4 -2
@@ -120,8 +120,10 @@ static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)/* Stuff the data into the input queue of the other end */ c =tty_insert_flip_string (to, buf, c);/* And shovel */ -tty_flip_buffer_push (to); -tty_wakeup (tty); +if (c) { +tty_flip_buffer_push (to); +tty_wakeup (tty); + } }return c; }
The patch fixes the problem here. Thank you very much.
--
Peter.
В Чтв, 17/09/2009 в 14:12 -0700, Linus Torvalds пишет:
Quoted 45 lines · Linus Torvalds
Quoted 4 lines
What's interesting about it is that it shows a problem, but the problem it shows would seem to have nothing at all to do with ppp or networking or pty's. The problem seems to be processes stuck in disk-wait:
Ahh. I think I see what may be going on.
Somebody got a filesystem mutex, and then went to sleep due to IO. Then pptp comes in, and seems to be stuck in a loop in kernel space, and it seems to be stuck with preemption off.
So one CPU is stuck, and the thing that we want to run is on the same run-queue, and not preempting. An looking at your CPU#1 trace, it's likely looping in
ppp_async_push().And that whole loop is insane (and very prone to infinite loops), but it also depends on that tty
wakeup()thing.Does this patch make a difference? Make sure to _not_ try to do the whole wakeup thing if we couldn't actually insert anything into the tty buffers.
Linus
--- drivers/char/pty.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)drivers/char/pty.c +4
-2@@ -120,8 +120,10 @@ static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)/* Stuff the data into the input queue of the other end */ c =tty_insert_flip_string (to, buf, c);/* And shovel */ -tty_flip_buffer_push (to); -tty_wakeup (tty); +if (c) { +tty_flip_buffer_push (to); +tty_wakeup (tty); + } }return c; }
Quoted 2 lines · Peter Volkov
The patch fixes the problem here. Thank you very much.
Hey, thank _you_ for the sysrq output, that made it quite debuggable.
Committed as 202c4675c, and I cc'd stable.
Linus
Quoted 6 lines · Andrey Rahmatullin
Quoted 1 line · Peter Volkov
Is this problem known?
Yes, it's described at http://bugzilla.kernel.org/show_bug.cgi?id=14179
since Tuesday.
Quoted 7 lines
Quoted 3 lines · Linus Torvalds
Quoted 1 line
The patch fixes the problem here. Thank you very much.
Hey, thank _you_ for the sysrq output, that made it quite debuggable. Committed as
202c4675c, and I cc'd stable.Thanks for the fix, but should I send bugreports directly here next time instead of filing a bug in bugzilla.kernel.org and waiting for response that will never come?
Bugzilla is great, but you should _also_ target the maintainers directly and let them know. And especially if you have bisected things, always cc everybody that is listed in the commit.
Otherwise, what happens is that other people not directly involved will eventually look at the regression list, and see it - but that generally happens much later. So things will get fixed from just the bugzilla report too, but you'll have a much longer latency than required.
In fact, if you can bisect it to a single commit (especially a small one like this), then bugzilla is the secondary, rather than the primary place. Bugzilla is great for keeping track of things and trying to avoid losing reports, but that comes at the expense of not being very convenient for short-term stuff.
So if you have a very targeted bugreport, and know who to send a report to, try the direct route first. Then, if nothing happens immediately, open a bugzilla (or open the bugzilla immediately, just in case, but see it as a "fallback" thing).
Linus
Quoted 1 line · Peter Volkov
Is this problem known?
Yes, it's described at http://bugzilla.kernel.org/show_bug.cgi?id=14179
since Tuesday.
Quoted 3 lines · Linus Torvalds
Quoted 1 line
The patch fixes the problem here. Thank you very much.
Hey, thank _you_ for the sysrq output, that made it quite debuggable. Committed as
202c4675c, and I cc'd stable.
Thanks for the fix, but should I send bugreports directly here next time instead of filing a bug in bugzilla.kernel.org and waiting for response that will never come?
Signature · 1 line
WBR, wRAR (ALT Linux Team)