Skip to contentSkip to conversation

kernel oops!

DISCUSSION ierdnah 4 participants 16 messages Jan 22, 2005 – Jan 28, 2005 Char and misc drivers

lore
Share & tools
lore

ierdnah

author
#0 lore raw
Jan 22 13:27:59 warsheep Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 22 13:27:59 warsheep printing eip:
Jan 22 13:27:59 warsheep 00000000
Jan 22 13:27:59 warsheep *pgd = cde9ddb400000000
Jan 22 13:27:59 warsheep *pmd = cde9ddb400000000
Jan 22 13:27:59 warsheep Oops: 0000 [#1]
Jan 22 13:27:59 warsheep SMP
Jan 22 13:27:59 warsheep CPU:    0
Jan 22 13:27:59 warsheep EIP:    0060:[<00000000>]    Not tainted VLI
Jan 22 13:27:59 warsheep EFLAGS: 00010282   (2.6.10-hardened-r2-warsheep62)
Jan 22 13:27:59 warsheep EIP is at 0x0
Jan 22 13:27:59 warsheep eax: 00000000   ebx: de455000   ecx: c02c60e0   edx: c6b41000
Jan 22 13:27:59 warsheep esi: de455000   edi: 00000000   ebp: dd0a2680   esp: cde9de9c
Jan 22 13:27:59 warsheep ds: 007b   es: 007b   ss: 0068
Jan 22 13:27:59 warsheep Process pptpctrl (pid: 16689, threadinfo=cde9c000 task=d112ca20)
Jan 22 13:27:59 warsheep Stack: c02c97bc c6b41000 00000000 c02c895c de455000 04949168 c03d0106 de455000
Jan 22 13:27:59 warsheep de45500c dd0a2680 00000000 c02c4141 de455000 dd0a2680 00000000 c01c7d49
Jan 22 13:27:59 warsheep dd0a2680 00000020 00000005 00000005 c01da72f dd0a2680 00000000 00000000
Jan 22 13:27:59 warsheep Call Trace:
Jan 22 13:27:59 warsheep [<c02c97bc>] pty_chars_in_buffer+0x2c/0x50
Jan 22 13:27:59 warsheep [<c02c895c>] normal_poll+0xfc/0x16b
Jan 22 13:27:59 warsheep [<c03d0106>] schedule_timeout+0x76/0xc0
Jan 22 13:27:59 warsheep [<c02c4141>] tty_poll+0xa1/0xc0
Jan 22 13:27:59 warsheep [<c01c7d49>] fget+0x49/0x60
Jan 22 13:27:59 warsheep [<c01da72f>] do_select+0x26f/0x2e0
Jan 22 13:27:59 warsheep [<c01da2f0>] __pollwait+0x0/0xd0
Jan 22 13:27:59 warsheep [<c01daabb>] sys_select+0x2db/0x4f0
Jan 22 13:27:59 warsheep [<c0173049>] sysenter_past_esp+0x52/0x79
Jan 22 13:27:59 warsheep Code:  Bad EIP value.

The oops ocures only when the kernel is build with SMP and HT support, in UP mode the oops doesn't occur! I have a 2.6.10 kernel with SMP and HT compiled kernel, I have a P4 3GHz with HT
a have a VPN server with pppd and pptpd(poptop) and and average of 130
simultanious connections, the oops doesn't occur at a particular number
of simulationus VPN connection.I can build a kernel with debugging enabled or something to help to track th source of the problem. Please CC as I am not subscribed to this mailing list.

Signature · 1 line
ierdnah <[email protected]>
#1 lore raw

Interesting. That last call trace entry is the call in pty_chars_in_buffer() to

        /* The ldisc must report 0 if no characters available to be read */
        count = to->ldisc.chars_in_buffer(to);

and it looks like it has jumped to address zero.

However, we _just_ compared the fn pointer to zero immediately before, and while there could certainly have been a race that cleared it in between the test and the call, normally we wouldn't even have re-loaded the value at all, but kept it in a register instead.

That said, it does act like a race. Somebody clearing the ldisc and racing with somebody using it?

Can you do a

gdb vmlinux

disassemble pty_chars_in_buffer

to show what it looks like (whether it reloads the value, and what the registers are - it looks like either %eax or %edi is all zeroes, but I'd like to verify that it matches your code generation).

Alan? Any ideas? The tty_select() path seems to take a ldisc reference, but does that guarantee that the ldisc won't _change_? What happens if the line discipline is reset from ppp to regular (or set to ppp) asymchronously? You've been deep in this area lately..

Linus

Quoted 47 lines · ierdnah
Jan 22 13:27:59 warsheep Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 22 13:27:59 warsheep printing eip:
Jan 22 13:27:59 warsheep 00000000
Jan 22 13:27:59 warsheep *pgd = cde9ddb400000000
Jan 22 13:27:59 warsheep *pmd = cde9ddb400000000
Jan 22 13:27:59 warsheep Oops: 0000 [#1]
Jan 22 13:27:59 warsheep SMP
Jan 22 13:27:59 warsheep CPU:    0
Jan 22 13:27:59 warsheep EIP:    0060:[<00000000>]    Not tainted VLI
Jan 22 13:27:59 warsheep EFLAGS: 00010282   (2.6.10-hardened-r2-warsheep62)
Jan 22 13:27:59 warsheep EIP is at 0x0
Jan 22 13:27:59 warsheep eax: 00000000   ebx: de455000   ecx: c02c60e0   edx: c6b41000
Jan 22 13:27:59 warsheep esi: de455000   edi: 00000000   ebp: dd0a2680   esp: cde9de9c
Jan 22 13:27:59 warsheep ds: 007b   es: 007b   ss: 0068
Jan 22 13:27:59 warsheep Process pptpctrl (pid: 16689, threadinfo=cde9c000 task=d112ca20)
Jan 22 13:27:59 warsheep Stack: c02c97bc c6b41000 00000000 c02c895c de455000 04949168 c03d0106 de455000
Jan 22 13:27:59 warsheep de45500c dd0a2680 00000000 c02c4141 de455000 dd0a2680 00000000 c01c7d49
Jan 22 13:27:59 warsheep dd0a2680 00000020 00000005 00000005 c01da72f dd0a2680 00000000 00000000
Jan 22 13:27:59 warsheep Call Trace:
Jan 22 13:27:59 warsheep [<c02c97bc>] pty_chars_in_buffer+0x2c/0x50
Jan 22 13:27:59 warsheep [<c02c895c>] normal_poll+0xfc/0x16b
Jan 22 13:27:59 warsheep [<c03d0106>] schedule_timeout+0x76/0xc0
Jan 22 13:27:59 warsheep [<c02c4141>] tty_poll+0xa1/0xc0
Jan 22 13:27:59 warsheep [<c01c7d49>] fget+0x49/0x60
Jan 22 13:27:59 warsheep [<c01da72f>] do_select+0x26f/0x2e0
Jan 22 13:27:59 warsheep [<c01da2f0>] __pollwait+0x0/0xd0
Jan 22 13:27:59 warsheep [<c01daabb>] sys_select+0x2db/0x4f0
Jan 22 13:27:59 warsheep [<c0173049>] sysenter_past_esp+0x52/0x79
Jan 22 13:27:59 warsheep Code:  Bad EIP value.

The oops ocures only when the kernel is build with SMP and HT support, in UP mode the oops doesn't occur! I have a 2.6.10 kernel with SMP and HT compiled kernel, I have a P4 3GHz with HT
a have a VPN server with pppd and pptpd(poptop) and and average of 130
simultanious connections, the oops doesn't occur at a particular number
of simulationus VPN connection.I can build a kernel with debugging enabled or something to help to track th source of the problem. Please CC as I am not subscribed to this mailing list.

Signature · 7 lines
ierdnah <[email protected]>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
author
#2 lore raw

(gdb) disassemble pty_chars_in_buffer
Dump of assembler code for function pty_chars_in_buffer:

0xc02c9790 <pty_chars_in_buffer+0>:     sub    $0x8,%esp
0xc02c9793 <pty_chars_in_buffer+3>:     xor    %eax,%eax
0xc02c9795 <pty_chars_in_buffer+5>:     mov    %ebx,0x4(%esp,1)
0xc02c9799 <pty_chars_in_buffer+9>:     mov    0xc(%esp,1),%ebx
0xc02c979d <pty_chars_in_buffer+13>:    mov    0xd0(%ebx),%edx
0xc02c97a3 <pty_chars_in_buffer+19>:    test   %edx,%edx
0xc02c97a5 <pty_chars_in_buffer+21>:    je     0xc02c97ae
<pty_chars_in_buffer+30>
0xc02c97a7 <pty_chars_in_buffer+23>:    mov    0x28(%edx),%ecx
0xc02c97aa <pty_chars_in_buffer+26>:    test   %ecx,%ecx
0xc02c97ac <pty_chars_in_buffer+28>:    jne    0xc02c97b6
<pty_chars_in_buffer+38>
0xc02c97ae <pty_chars_in_buffer+30>:    mov    0x4(%esp,1),%ebx
0xc02c97b2 <pty_chars_in_buffer+34>:    add    $0x8,%esp
0xc02c97b5 <pty_chars_in_buffer+37>:    ret
0xc02c97b6 <pty_chars_in_buffer+38>:    mov    %edx,(%esp,1)
0xc02c97b9 <pty_chars_in_buffer+41>:    call   *0x28(%edx)
0xc02c97bc <pty_chars_in_buffer+44>:    mov    %eax,%edx
0xc02c97be <pty_chars_in_buffer+46>:    mov    0x4(%ebx),%eax
0xc02c97c1 <pty_chars_in_buffer+49>:    cmpw   $0x2,0x76(%eax)
0xc02c97c6 <pty_chars_in_buffer+54>:    je     0xc02c97d5
<pty_chars_in_buffer+69>
0xc02c97c8 <pty_chars_in_buffer+56>:    xor    %eax,%eax
0xc02c97ca <pty_chars_in_buffer+58>:    cmp    $0x800,%edx
0xc02c97d0 <pty_chars_in_buffer+64>:    cmovge %edx,%eax
0xc02c97d3 <pty_chars_in_buffer+67>:    jmp    0xc02c97ae
<pty_chars_in_buffer+30>
0xc02c97d5 <pty_chars_in_buffer+69>:    mov    %edx,%eax
0xc02c97d7 <pty_chars_in_buffer+71>:    jmp    0xc02c97ae

<pty_chars_in_buffer+30>
0xc02c97d9 <pty_chars_in_buffer+73>: lea 0x0(%esi,1),%esi
End of assembler dump.

this is another compiled kernel, but is compiled with the same .config file and same gcc version...because I only have the bzImage, how do I convert it to vmlinux?

Quoted 82 lines · Linus Torvalds

Interesting. That last call trace entry is the call in pty_chars_in_buffer() to

        /* The ldisc must report 0 if no characters available to be read */
        count = to->ldisc.chars_in_buffer(to);

and it looks like it has jumped to address zero.

However, we _just_ compared the fn pointer to zero immediately before, and while there could certainly have been a race that cleared it in between the test and the call, normally we wouldn't even have re-loaded the value at all, but kept it in a register instead.

That said, it does act like a race. Somebody clearing the ldisc and racing with somebody using it?

Can you do a

gdb vmlinux

disassemble pty_chars_in_buffer

to show what it looks like (whether it reloads the value, and what the registers are - it looks like either %eax or %edi is all zeroes, but I'd like to verify that it matches your code generation).

Alan? Any ideas? The tty_select() path seems to take a ldisc reference, but does that guarantee that the ldisc won't _change_? What happens if the line discipline is reset from ppp to regular (or set to ppp) asymchronously? You've been deep in this area lately..

Linus

Quoted 47 lines · ierdnah
Jan 22 13:27:59 warsheep Unable to handle kernel NULL pointer dereference at virtual address 00000000
Jan 22 13:27:59 warsheep printing eip:
Jan 22 13:27:59 warsheep 00000000
Jan 22 13:27:59 warsheep *pgd = cde9ddb400000000
Jan 22 13:27:59 warsheep *pmd = cde9ddb400000000
Jan 22 13:27:59 warsheep Oops: 0000 [#1]
Jan 22 13:27:59 warsheep SMP
Jan 22 13:27:59 warsheep CPU:    0
Jan 22 13:27:59 warsheep EIP:    0060:[<00000000>]    Not tainted VLI
Jan 22 13:27:59 warsheep EFLAGS: 00010282   (2.6.10-hardened-r2-warsheep62)
Jan 22 13:27:59 warsheep EIP is at 0x0
Jan 22 13:27:59 warsheep eax: 00000000   ebx: de455000   ecx: c02c60e0   edx: c6b41000
Jan 22 13:27:59 warsheep esi: de455000   edi: 00000000   ebp: dd0a2680   esp: cde9de9c
Jan 22 13:27:59 warsheep ds: 007b   es: 007b   ss: 0068
Jan 22 13:27:59 warsheep Process pptpctrl (pid: 16689, threadinfo=cde9c000 task=d112ca20)
Jan 22 13:27:59 warsheep Stack: c02c97bc c6b41000 00000000 c02c895c de455000 04949168 c03d0106 de455000
Jan 22 13:27:59 warsheep de45500c dd0a2680 00000000 c02c4141 de455000 dd0a2680 00000000 c01c7d49
Jan 22 13:27:59 warsheep dd0a2680 00000020 00000005 00000005 c01da72f dd0a2680 00000000 00000000
Jan 22 13:27:59 warsheep Call Trace:
Jan 22 13:27:59 warsheep [<c02c97bc>] pty_chars_in_buffer+0x2c/0x50
Jan 22 13:27:59 warsheep [<c02c895c>] normal_poll+0xfc/0x16b
Jan 22 13:27:59 warsheep [<c03d0106>] schedule_timeout+0x76/0xc0
Jan 22 13:27:59 warsheep [<c02c4141>] tty_poll+0xa1/0xc0
Jan 22 13:27:59 warsheep [<c01c7d49>] fget+0x49/0x60
Jan 22 13:27:59 warsheep [<c01da72f>] do_select+0x26f/0x2e0
Jan 22 13:27:59 warsheep [<c01da2f0>] __pollwait+0x0/0xd0
Jan 22 13:27:59 warsheep [<c01daabb>] sys_select+0x2db/0x4f0
Jan 22 13:27:59 warsheep [<c0173049>] sysenter_past_esp+0x52/0x79
Jan 22 13:27:59 warsheep Code:  Bad EIP value.

The oops ocures only when the kernel is build with SMP and HT support, in UP mode the oops doesn't occur! I have a 2.6.10 kernel with SMP and HT compiled kernel, I have a P4 3GHz with HT
a have a VPN server with pppd and pptpd(poptop) and and average of 130
simultanious connections, the oops doesn't occur at a particular number
of simulationus VPN connection.I can build a kernel with debugging enabled or something to help to track th source of the problem. Please CC as I am not subscribed to this mailing list.

Signature · 7 lines
ierdnah <[email protected]>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Signature · 1 line
ierdnah <[email protected]>
#3 lore raw
Quoted 2 lines · ierdnah

(gdb) disassemble pty_chars_in_buffer

...

Quoted 8 lines
0xc02c97a7 <pty_chars_in_buffer+23>:    mov    0x28(%edx),%ecx **
0xc02c97aa <pty_chars_in_buffer+26>:    test   %ecx,%ecx
0xc02c97ac <pty_chars_in_buffer+28>:    jne    0xc02c97b6
0xc02c97ae <pty_chars_in_buffer+30>:    mov    0x4(%esp,1),%ebx
0xc02c97b2 <pty_chars_in_buffer+34>:    add    $0x8,%esp
0xc02c97b5 <pty_chars_in_buffer+37>:    ret
0xc02c97b6 <pty_chars_in_buffer+38>:    mov    %edx,(%esp,1)
0xc02c97b9 <pty_chars_in_buffer+41>:    call   *0x28(%edx) **

Ahh, indeed. When I compiled this function, it kept 0x28(%edx) in a register. Your config/compiler combination does not, so there actually is a race condition if somebody changes the function pointer.

Quoted 3 lines

this is another compiled kernel, but is compiled with the same .config file and same gcc version...because I only have the bzImage, how do I convert it to vmlinux?

Don't worry, it clearly shows that it's at least possible, and worth looking at.

For testing, a patch like this might get rid of the problem by hiding the race (but for all I know, gcc ends up re-loading it anyway) - you might want to check the disassembly.

However, this patch is just for testing, to verify that your problem really is that particular race. It's not a proper fix.

I suspect that pty's should always lock each others line disciplines too, not just their "own" side.

Linus

----

drivers/char/pty.c +4 -2
@@ -149,13 +149,15 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
 	struct tty_struct *to = tty->link;
+	ssize_t (*chars_in_buffer)(struct tty_struct *);
 	int count;
 
-	if (!to || !to->ldisc.chars_in_buffer)
+	/* We should get the line discipline lock for "tty->link" */
+	if (!to || !(chars_in_buffer = to->ldisc.chars_in_buffer))
 		return 0;
 
 	/* The ldisc must report 0 if no characters available to be read */
-	count = to->ldisc.chars_in_buffer(to);
+	count = chars_in_buffer(to);
 
 	if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
 
author
#4 lore raw

On Sun, 2005-01-23 at 09:51 -0800, Linus Torvalds wrote:

I applied the patch and the uptime is 1 day and no oops, i will wait 2 more days too see if the oops appears, then I will compile the kernel with CONFIG_PREEMPT. Before, when PREEMPT was enabled, the oops appeared immediatly after booting and more often.

Quoted 22 lines

----

drivers/char/pty.c +4 -2
@@ -149,13 +149,15 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
 	struct tty_struct *to = tty->link;
+	ssize_t (*chars_in_buffer)(struct tty_struct *);
 	int count;
 
-	if (!to || !to->ldisc.chars_in_buffer)
+	/* We should get the line discipline lock for "tty->link" */
+	if (!to || !(chars_in_buffer = to->ldisc.chars_in_buffer))
 		return 0;
 
 	/* The ldisc must report 0 if no characters available to be read */
-	count = to->ldisc.chars_in_buffer(to);
+	count = chars_in_buffer(to);
 
 	if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
 
Signature · 1 line
ierdnah <[email protected]>
author
#5 lore raw
In reply to Linus Torvalds

On Sun, 2005-01-23 at 09:51 -0800, Linus Torvalds wrote:

with this patch the oops is gone(also tested with PREEMPT and no oops)

Quoted 12 lines

----

drivers/char/pty.c +1 -1
@@ -149,13 +149,15 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
 	struct tty_struct *to = tty->link;
+	ssize_t (*chars_in_buffer)(struct tty_struct *);
 	int count;
 
-	if (!to || !to->ldisc.chars_in_buffer)

drivers/char/pty.c line 163

Quoted 10 lines
drivers/char/pty.c +3 -1
+	/* We should get the line discipline lock for "tty->link" */
+	if (!to || !(chars_in_buffer = to->ldisc.chars_in_buffer))
 		return 0;
 
 	/* The ldisc must report 0 if no characters available to be read */
-	count = to->ldisc.chars_in_buffer(to);
+	count = chars_in_buffer(to);
 
 	if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
 

is this patch better? should i test this too?

drivers/char/pty.c 2005-01-23 10:21:04 -08:00 renamed from drivers/char/pty.c 2005-01-10 17:29:36 -08:00 +1 -5
@@ -149,13 +149,17 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
        struct tty_struct *to = tty->link;
-       int count;
+       int count = 0;
 
-       if (!to || !to->ldisc.chars_in_buffer)
-               return 0;
-
-       /* The ldisc must report 0 if no characters available to be read

*/
- count = to->ldisc.chars_in_buffer(to);
+ if (to) {
+ struct tty_ldisc *ld = tty_ldisc_ref(to);
+ if (ld) {
+ if (ld->chars_in_buffer) {
+ count = ld->chars_in_buffer(to);
+ tty_ldisc_deref(ld);
+ }
+ }
+ }

        if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
Signature · 1 line
ierdnah <[email protected]>
#6 lore raw
Quoted 2 lines · ierdnah

is this patch better? should i test this too?

You probably should. The patch you've tested is really ugly, and not a fix at all - it's really just depending on the compiler generating a specific code sequence that will hide the race. As such, it's a patch I would only accept in the standard kernel as an absolute last resort.

In contrast, the second patch I tested may actually _fix_ the race.

The fact that the first patch makes the oops go away is a good thing,
though: it shows that your oops really was due to that small race window, and as such it helps validate that it wasn't anything else.

Thanks,

Linus

author
#7 lore raw

On Thu, 2005-01-27 at 15:35 -0800, Linus Torvalds wrote:

the last patch works, but the load increases very much (normally with 200 VPN connections I have a load of maximum 10, with this patch I have a load of 50-100 - after 30 min of uptime)

Quoted 10 lines

You probably should. The patch you've tested is really ugly, and not a fix at all - it's really just depending on the compiler generating a specific code sequence that will hide the race. As such, it's a patch I would only accept in the standard kernel as an absolute last resort.

In contrast, the second patch I tested may actually _fix_ the race.

The fact that the first patch makes the oops go away is a good thing,
though: it shows that your oops really was due to that small race window, and as such it helps validate that it wasn't anything else.

drivers/char/pty.c 2005-01-23 10:21:04 -08:00 renamed from drivers/char/pty.c 2005-01-10 17:29:36 -08:00 +1 -5
@@ -149,13 +149,17 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
        struct tty_struct *to = tty->link;
-       int count;
+       int count = 0;
 
-       if (!to || !to->ldisc.chars_in_buffer)
-               return 0;
-
-       /* The ldisc must report 0 if no characters available to be read

*/
- count = to->ldisc.chars_in_buffer(to);
+ if (to) {
+ struct tty_ldisc *ld = tty_ldisc_ref(to);
+ if (ld) {
+ if (ld->chars_in_buffer) {
+ count = ld->chars_in_buffer(to);
+ tty_ldisc_deref(ld);
+ }
+ }
+ }

       if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
Signature · 1 line
ierdnah <[email protected]>
#8 lore raw
Quoted 4 lines · ierdnah

the last patch works, but the load increases very much (normally with 200 VPN connections I have a load of maximum 10, with this patch I have a load of 50-100 - after 30 min of uptime)

Yeah, that "tty_ldisc_try()" is pretty expensive, and it really would be worth trying to get both sides of the tty at the same time, since with my patch in place it has to get that "tty_ldisc_lock" four times (and disable interrupts etc - twice for getting the lock, twice for releasing it).

I'm surprised that it makes _that_ much of a difference, but it sounds like you used to be borderline on CPU usage before, and this just made it much worse.

One option is to instead of locking both master and slave on use in the pty side, lock both when _changing_ the ldisc instead. That's the rare case. Much more complex, though.

Alan, any clever ideas?

Linus

author
#9 lore raw
Quoted 3 lines · Linus Torvalds

I'm surprised that it makes _that_ much of a difference, but it sounds like you used to be borderline on CPU usage before, and this just made it much worse.

it's musch worst, I had a load of 5 with 250 VPN connections, and now, I have a load of 200 with 150 connections

Signature · 1 line
ierdnah <[email protected]>
#10 lore raw
In reply to Linus Torvalds
Quoted 20 lines · On Sat, 22 Jan 2005 22:43:50 -0800 (PST) Linus Torvalds wrote:

Interesting. That last call trace entry is the call in pty_chars_in_buffer() to

        /* The ldisc must report 0 if no characters available to be read */
        count = to->ldisc.chars_in_buffer(to);

and it looks like it has jumped to address zero.

However, we _just_ compared the fn pointer to zero immediately before, and while there could certainly have been a race that cleared it in between the test and the call, normally we wouldn't even have re-loaded the value at all, but kept it in a register instead.

That said, it does act like a race. Somebody clearing the ldisc and racing with somebody using it?

Can you do a

gdb vmlinux

Quoted 8 lines

disassemble pty_chars_in_buffer

to show what it looks like (whether it reloads the value, and what the registers are - it looks like either %eax or %edi is all zeroes, but I'd like to verify that it matches your code generation).

Alan? Any ideas? The tty_select() path seems to take a ldisc reference, but does that guarantee that the ldisc won't _change_?

tty_poll() grabs ldisc reference for the tty it was called with; however, in this case pty_chars_in_buffer() accesses another ldisc (tty->link->ldisc) without grabbing a reference to it. BTW, many other pty_* functions do the same thing.

Is calling tty_ldisc_ref(tty->link) safe here? There is a comment warning about possible deadlocks before pty_write().

#11 lore raw
Quoted 5 lines · Sergey Vlasov

tty_poll() grabs ldisc reference for the tty it was called with; however, in this case pty_chars_in_buffer() accesses another ldisc (tty->link->ldisc) without grabbing a reference to it. BTW, many other pty_* functions do the same thing.

Yes, I think you put the finger on it.

Quoted 2 lines

Is calling tty_ldisc_ref(tty->link) safe here? There is a comment warning about possible deadlocks before pty_write().

I think it's only the tty_ldisc_ref_wait() thing that can deadlock (and even that is likely safe if you just specify an order - "masters first" or something). Adding a nonblocking "tty_ldisc_ref()" looks safe, ie something like the appended.

This has the problem (apart from the fact that it's obviously totally untested ;) that it looks like every single pty function would need to do it, so it would be nicer if "tty_ldisc_ref_wait()" would just always get both references (ie do the ordering). Alan?

Linus
----

drivers/char/pty.c +10 -6
@@ -149,13 +149,17 @@
 static int pty_chars_in_buffer(struct tty_struct *tty)
 {
 	struct tty_struct *to = tty->link;
-	int count;
+	int count = 0;
 
-	if (!to || !to->ldisc.chars_in_buffer)
-		return 0;
-
-	/* The ldisc must report 0 if no characters available to be read */
-	count = to->ldisc.chars_in_buffer(to);
+	if (to) {
+		struct tty_ldisc *ld = tty_ldisc_ref(to);
+		if (ld) {
+			if (ld->chars_in_buffer) {
+				count = ld->chars_in_buffer(to);
+				tty_ldisc_deref(ld);
+			}
+		}
+	}
 
 	if (tty->driver->subtype == PTY_TYPE_SLAVE) return count;
 
#12 lore raw
Quoted 4 lines · Linus Torvalds

I think it's only the tty_ldisc_ref_wait() thing that can deadlock (and even that is likely safe if you just specify an order - "masters first" or something). Adding a nonblocking "tty_ldisc_ref()" looks safe, ie something like the appended.

Yes.

Quoted 4 lines

This has the problem (apart from the fact that it's obviously totally untested ;) that it looks like every single pty function would need to do it, so it would be nicer if "tty_ldisc_ref_wait()" would just always get both references (ie do the ordering). Alan?

Almost every user of tty_ldisc_ref_* doesn't need to lock two objects and
the code at that layer has no knowledge of pty/tty pairs. The needed info is exposed however in order to do this since the tty knows if its a tty/pty pair. I'll take a look - chances are it can be buried in tty_ldisc_ref.

I'm dubious this is the actual bug although vhangup on a pty might trigger it I guess.

Alan

#13 lore raw
Quoted 3 lines · Alan Cox

Almost every user of tty_ldisc_ref_* doesn't need to lock two objects and the code at that layer has no knowledge of pty/tty pairs.

Hmm.. It looks different to me - almost all users of the xxx_ref_wait() that act on pty's _do_ seem to use tty->link->ldisc.*()". The one counter- example seems to be pty_set_termios().

But I only did a quick grep in drivers/char, so there might be other users out there I didn't even think of.

While doing that, I did notice that all the other tty ops do take the kernel lock still - both read() and write() do. I wonder if we should just make poll() consistent, regardless of other issues (obviously, the nicest way to make it consistent would be to remove the kernel lock from the read/write paths, but I assume nobody wants to go through the locking).

Quoted 2 lines

I'm dubious this is the actual bug although vhangup on a pty might trigger it I guess.

ierdnah - can you verify whether either of my silly patches seems to have made any difference?

And I don't think it can be vhangup - that wouldn't clear the chars_in_buffer function pointer (it might change it to the n_tty_chars_in_buffer or something by resetting to N_TTY).

I'd assume that it's when the other end is changed to using the "ppp_[sync_]ldisc", both of which do actually have a NULL "chars_in_buffer". (Alternatively, maybe "chars_in_buffer" isn't actually NULL at all - the backtrace might be confused by either some really strange memory corruption or a tail-call to a NULL function pointer by some other line discipline thing, although quite frankly I think that sounds extremely unlikely).

So another potential fix is to just make all tty line disciplines always have a valid "chars_in_buffer" pointer. We could even do automatically in "tty_set_operations()", ie just do a

	driver->chars_in_buffer = op->chars_in_buffer ? : default_chars_in_buffer;

(where the default just returns zero) thing or something.

That would even speed up the normal cases (no need to test the pointer for NULL).

Linus

#14 lore raw
Quoted 6 lines · Linus Torvalds

On Mon, 24 Jan 2005, Alan Cox wrote:
While doing that, I did notice that all the other tty ops do take the kernel lock still - both read() and write() do. I wonder if we should just make poll() consistent, regardless of other issues (obviously, the nicest way to make it consistent would be to remove the kernel lock from the read/write paths, but I assume nobody wants to go through the locking).

The lock is needed for the console at least and for some driver level bits.

Quoted 3 lines

And I don't think it can be vhangup - that wouldn't clear the chars_in_buffer function pointer (it might change it to the n_tty_chars_in_buffer or something by resetting to N_TTY).

True - that would need an ldisc.close and the vhangup no longer does that.

Quoted 3 lines

So another potential fix is to just make all tty line disciplines always have a valid "chars_in_buffer" pointer. We could even do automatically in "tty_set_operations()", ie just do a

Not really. I'm in n_tty ->chars_in_buffer and the race occurs -> I'm dead
as I'm already running the wrong chars_in_buffer function on CPU #0 when CPU #1
comes along and flips ldisc. Holding the right locks matters here. We also
need both locks holding really for tty/pty pairs because pty_write wants to
output to the ldisc of the other side. Treating no ldisc as no characters
seems very sane however and is easy to code up - if the ldisc_get fails we
can sleep on the ldisc level wait queue then retry. Ugly enough to want to
hide the contents in tty_io.c but doable. (When I get time - likely to be
a couple of weeks if nobody does it first)

Alan

#15 lore raw
In reply to Sergey Vlasov
Quoted 5 lines · Sergey Vlasov

On Sat, 22 Jan 2005 22:43:50 -0800 (PST) Linus Torvalds wrote:
tty_poll() grabs ldisc reference for the tty it was called with; however, in this case pty_chars_in_buffer() accesses another ldisc (tty->link->ldisc) without grabbing a reference to it. BTW, many other pty_* functions do the same thing.

Correct - the pty code still relies on all sorts of "lock_kernel" assumptions so its probably very very racy. It can't however easily use tty_ldisc_()
functions that wait because the lock ordering is indeterminate.