Skip to contentSkip to conversation

general protection fault in n_tty_receive_buf_common

DISCUSSION Dmitry Vyukov 2 participants 5 messages Oct 27, 2017 1 message not in the archive TTY layer and serial drivers

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 8 lines · syzbot

Hello,

syzkaller hit the following crash on
623ce3456671ea842c0ebda79c38655c8c04af74
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

A more recent report is on upstream 0787643a5f6aad1f0cdeb305f7fe492b71943ea4

kasan: GPF could be caused by NULL-ptr deref or user memory access

general protection fault: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 1 PID: 59 Comm: kworker/u4:2 Not tainted 4.14.0-rc5+ #142
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Workqueue: events_unbound flush_to_ldisc
task: ffff8801d9aa6700 task.stack: ffff8801d9ab0000
RIP: 0010:__read_once_size include/linux/compiler.h:276 [inline]
RIP: 0010:n_tty_receive_buf_common+0x154/0x2520 drivers/tty/n_tty.c:1690
RSP: 0018:ffff8801d9ab7130 EFLAGS: 00010202
RAX: 000000000000044c RBX: ffff8801d4cb0400 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801d4cb07a8 RDI: 0000000000002260
RBP: ffff8801d9ab72d8 R08: ffffffff82721563 R09: 0000000000000005
R10: ffff8801d9ab6f68 R11: 0000000000000002 R12: ffff8801d28ed27f
R13: 0000000000000000 R14: dffffc0000000000 R15: ffff8801d28ed27f
FS:  0000000000000000(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f97bb6e6330 CR3: 00000001c5959000 CR4: 00000000001426e0
Call Trace:
 n_tty_receive_buf2+0x33/0x40 drivers/tty/n_tty.c:1746
 tty_ldisc_receive_buf+0xa7/0x180 drivers/tty/tty_buffer.c:455
 tty_port_default_receive_buf+0x106/0x160 drivers/tty/tty_port.c:37
 receive_buf drivers/tty/tty_buffer.c:474 [inline]
 flush_to_ldisc+0x3c4/0x590 drivers/tty/tty_buffer.c:523
 process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2119
 worker_thread+0x223/0x1860 kernel/workqueue.c:2253
 kthread+0x35e/0x430 kernel/kthread.c:231
 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431
Code: 89 85 18 ff ff ff 48 8d 45 98 48 89 85 e0 fe ff ff e8 61 a8 fb
fe 48 8b 85 18 ff ff ff c6 00 00 48 8b 85 a0 fe ff ff 48 c1 e8 03 <42>
80 3c 30 00 0f 85 6d 1e 00 00 48 8b 85 28 ff ff ff 4c 8b a8
RIP: __read_once_size include/linux/compiler.h:276 [inline] RSP:
ffff8801d9ab7130
RIP: n_tty_receive_buf_common+0x154/0x2520 drivers/tty/n_tty.c:1690
RSP: ffff8801d9ab7130
---[ end trace b62a8a78df2eb9d9 ]---
Quoted 75 lines

kasan: CONFIG_KASAN_INLINE enabled
kasan: GPF could be caused by NULL-ptr deref or user memory access

general protection fault: 0000 [#1] SMP KASAN

Dumping ftrace buffer:
SELinux: unrecognized netlink message: protocol=4 nlmsg_type=32 sclass=netlink_tcpdiag_socket pig=18822 comm=syz-executor4
(ftrace buffer empty)

Modules linked in:
CPU: 0 PID: 3797 Comm: kworker/u4:5 Not tainted 4.13.0-rc4+ #24
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: events_unbound flush_to_ldisc
task: ffff8801c6abc340 task.stack: ffff8801c6a08000
RIP: 0010:__read_once_size include/linux/compiler.h:250 [inline]
RIP: 0010:n_tty_receive_buf_common+0x154/0x2520 drivers/tty/n_tty.c:1690
RSP: 0018:ffff8801c6a0f130 EFLAGS: 00010202
RAX: 000000000000044c RBX: ffff8801c8c0f680 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801c8c0fa28 RDI: 0000000000002260
RBP: ffff8801c6a0f2d8 R08: 0000000000000004 R09: 0000000000000005
R10: ffff8801c6a0ef68 R11: ffffffff8266dad3 R12: ffff8801c73d5b82
R13: ffff8801c73d5c82 R14: dffffc0000000000 R15: ffff8801c73d5b82
SELinux: unrecognized netlink message: protocol=4 nlmsg_type=32
sclass=netlink_tcpdiag_socket pig=18822 comm=syz-executor4
FS:  0000000000000000(0000) GS:ffff8801dc000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000000 CR3: 0000000005021000 CR4: 00000000001406f0
DR0: 0000000020000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 n_tty_receive_buf2+0x33/0x40 drivers/tty/n_tty.c:1746
 tty_ldisc_receive_buf+0xa7/0x180 drivers/tty/tty_buffer.c:429
 tty_port_default_receive_buf+0x106/0x160 drivers/tty/tty_port.c:37
 receive_buf drivers/tty/tty_buffer.c:448 [inline]
 flush_to_ldisc+0x3c4/0x590 drivers/tty/tty_buffer.c:497
 process_one_work+0xbf3/0x1bc0 kernel/workqueue.c:2097
 worker_thread+0x223/0x1860 kernel/workqueue.c:2231
 kthread+0x35e/0x430 kernel/kthread.c:231
 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:425
Code: 89 85 18 ff ff ff 48 8d 45 98 48 89 85 e0 fe ff ff e8 71 cb 06 ff 48
8b 85 18 ff ff ff c6 00 00 48 8b 85 a0 fe ff ff 48 c1 e8 03 <42> 80 3c 30 00
0f 85 6d 1e 00 00 48 8b 85 28 ff ff ff 4c 8b a8
RIP: __read_once_size include/linux/compiler.h:250 [inline] RSP:
ffff8801c6a0f130
RIP: n_tty_receive_buf_common+0x154/0x2520 drivers/tty/n_tty.c:1690 RSP:
ffff8801c6a0f130
---[ end trace 1e2f55861d14ef37 ]---
Kernel panic - not syncing: Fatal exception

Dumping ftrace buffer:
(ftrace buffer empty)
Kernel Offset: disabled

---
This bug is generated by a dumb bot. It may contain errors. See https://goo.gl/tpsmEJ for details.
Direct all questions to [email protected].

syzbot will keep track of this bug report.
Once a fix for this bug is committed, please reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug report.

--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/syzkaller-bugs/001a113a9e244584f1055c83d598%40google.com.
For more options, visit https://groups.google.com/d/optout.

#2 lore raw
Quoted 5 lines · Dmitry Vyukov
Quoted 2 lines · syzbot

Hello,

Quoted 8 lines
Quoted 6 lines

syzkaller hit the following crash on
623ce3456671ea842c0ebda79c38655c8c04af74
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

A more recent report is on upstream 0787643a5f6aad1f0cdeb305f7fe492b71943ea4

Bad. Do we have a reproducer?

thanks,

Signature · 2 lines
js
suse labs
#3 lore raw
Quoted 13 lines · Jiri Slaby
Quoted 12 lines · Dmitry Vyukov
Quoted 8 lines · syzbot

Hello,

syzkaller hit the following crash on
623ce3456671ea842c0ebda79c38655c8c04af74
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

Quoted 3 lines
Quoted 1 line

A more recent report is on upstream 0787643a5f6aad1f0cdeb305f7fe492b71943ea4

Bad. Do we have a reproducer?

Please refer to this section of the referenced doc:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-reproducer-at-all

In this case we don't even know what program from the log triggered it, because the crash happened in the background thread. Probably one of the programs doing openat$ptmx.
And we only got 30 of such crashes overall, so it seems to be notoriously difficult to reproduce.

#4 lore raw
Quoted 2 lines · Dmitry Vyukov

And we only got 30 of such crashes overall, so it seems to be notoriously difficult to reproduce.

So this is a bug lasting for years, right?

thanks,

Signature · 2 lines
js
suse labs
#5 lore raw
In reply to Dmitry Vyukov
Quoted 46 lines · Dmitry Vyukov
Quoted 8 lines · syzbot

Hello,

syzkaller hit the following crash on
623ce3456671ea842c0ebda79c38655c8c04af74
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

A more recent report is on upstream 0787643a5f6aad1f0cdeb305f7fe492b71943ea4

kasan: GPF could be caused by NULL-ptr deref or user memory access

general protection fault: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in:
CPU: 1 PID: 59 Comm: kworker/u4:2 Not tainted 4.14.0-rc5+ #142
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Workqueue: events_unbound flush_to_ldisc
task: ffff8801d9aa6700 task.stack: ffff8801d9ab0000
RIP: 0010:__read_once_size include/linux/compiler.h:276 [inline]
RIP: 0010:n_tty_receive_buf_common+0x154/0x2520 drivers/tty/n_tty.c:1690
RSP: 0018:ffff8801d9ab7130 EFLAGS: 00010202
RAX: 000000000000044c RBX: ffff8801d4cb0400 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801d4cb07a8 RDI: 0000000000002260
RBP: ffff8801d9ab72d8 R08: ffffffff82721563 R09: 0000000000000005
R10: ffff8801d9ab6f68 R11: 0000000000000002 R12: ffff8801d28ed27f
R13: 0000000000000000 R14: dffffc0000000000 R15: ffff8801d28ed27f
FS:  0000000000000000(0000) GS:ffff8801db300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f97bb6e6330 CR3: 00000001c5959000 CR4: 00000000001426e0
Call Trace:
 n_tty_receive_buf2+0x33/0x40 drivers/tty/n_tty.c:1746
 tty_ldisc_receive_buf+0xa7/0x180 drivers/tty/tty_buffer.c:455
 tty_port_default_receive_buf+0x106/0x160 drivers/tty/tty_port.c:37
 receive_buf drivers/tty/tty_buffer.c:474 [inline]
 flush_to_ldisc+0x3c4/0x590 drivers/tty/tty_buffer.c:523
 process_one_work+0xbf0/0x1bc0 kernel/workqueue.c:2119
 worker_thread+0x223/0x1860 kernel/workqueue.c:2253
 kthread+0x35e/0x430 kernel/kthread.c:231
 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431
Code: 89 85 18 ff ff ff 48 8d 45 98 48 89 85 e0 fe ff ff e8 61 a8 fb
Quoted 2 lines

fe 48 8b 85 18 ff ff ff c6 00 00 48 8b 85 a0 fe ff ff 48 c1 e8 03 <42> 80 3c 30 00 0f 85 6d 1e 00 00 48 8b 85 28 ff ff ff 4c 8b a8

This decodes as:

   0:   89 85 18 ff ff ff       mov    %eax,-0xe8(%rbp)
   6:   48 8d 45 98             lea    -0x68(%rbp),%rax
   a:   48 89 85 e0 fe ff ff    mov    %rax,-0x120(%rbp)
  11:   e8 61 a8 fb fe          callq  0xfffffffffefba877
  16:   48 8b 85 18 ff ff ff    mov    -0xe8(%rbp),%rax
  1d:   c6 00 00                movb   $0x0,(%rax)
  20:   48 8b 85 a0 fe ff ff    mov    -0x160(%rbp),%rax
  27:   48 c1 e8 03             shr    $0x3,%rax
  2b:*  42 80 3c 30 00          cmpb   $0x0,(%rax,%r14,1)
<-- trapping instruction
  30:   0f 85 6d 1e 00 00       jne    0x1ea3
  36:   48 8b 85 28 ff ff ff    mov    -0xd8(%rbp),%rax

So KASAN is checking 0x44c in the shadow. This is 0x2260 in normal memory. 0x2260 is the offset of read_tail in struct n_tty_data aka ldata in this function. So ldata (i.e. tty->ldisc_data) is NULL.

This means ->receive_buf was called before n_tty_open proceeded (or finished).

Coincidently, Yuhong Tao is seeing a similar issue and tried to fix this in his "tty: fix flush_to_ldisc() oops before tty_open is done" the day before yesterday. The patch is not correct though.

At this point I am curious, what is the driver behind the failing tty? And why and who queued flush_to_ldics at this point? May I ask you to apply the patch from the attachment? It is only for debugging, but I am running my debug kernels with it for quite some time already (almost 7 years, apparently 8-)).

thanks,

Signature · 2 lines
js
suse labs
Quoted 1 line

From 8b3902b4304cbc86822c88cf476faf483b0368d3 Mon Sep 17 00:00:00 2001

From: Jiri Slaby <[email protected]>
Date: Sun, 28 Oct 2012 21:42:02 +0100
Subject: [PATCH 1/1] tty BUF DEBUG

---
 drivers/tty/pty.c         | 1 +
 drivers/tty/tty_buffer.c  | 1 +
 drivers/tty/tty_io.c      | 4 ++--
 drivers/tty/tty_ldisc.c   | 1 +
 drivers/tty/tty_port.c    | 4 ++--
 drivers/tty/vt/keyboard.c | 2 +-
 6 files changed, 8 insertions(+), 5 deletions(-)
drivers/tty/pty.c +1 -0
index 26dcb3b60fb9..644db78bb035 100644
@@ -445,6 +445,7 @@ static int pty_common_install(struct tty_driver *driver, struct tty_struct *tty,
 
 static void pty_cleanup(struct tty_struct *tty)
 {
+	tty->port->itty = ERR_PTR(-2);
 	tty_port_put(tty->port);
 }
 
drivers/tty/tty_buffer.c +1 -0
index f8eba1c5412f..642a7fc50aa5 100644
@@ -404,6 +404,7 @@ void tty_schedule_flip(struct tty_port *port)
 	 * flush_to_ldisc() sees buffer data.
 	 */
 	smp_store_release(&buf->tail->commit, buf->tail->used);
+	WARN_RATELIMIT(IS_ERR_OR_NULL(port->itty), "HUH %p\n", port->itty);
 	queue_work(system_unbound_wq, &buf->work);
 }
 EXPORT_SYMBOL(tty_schedule_flip);
drivers/tty/tty_io.c +2 -2
index 94cccb6efa32..7dd81d3e0917 100644
@@ -1471,9 +1471,9 @@ static void release_tty(struct tty_struct *tty, int idx)
 		tty->ops->shutdown(tty);
 	tty_free_termios(tty);
 	tty_driver_remove_tty(tty->driver, tty);
-	tty->port->itty = NULL;
+	tty->port->itty = ERR_PTR(-1);
 	if (tty->link)
-		tty->link->port->itty = NULL;
+		tty->link->port->itty = ERR_PTR(-1);
 	tty_buffer_cancel_work(tty->port);
 
 	tty_kref_put(tty->link);
drivers/tty/tty_ldisc.c +1 -0
index 84a8ac2a779f..8b2ac1216205 100644
@@ -615,6 +615,7 @@ int tty_set_ldisc(struct tty_struct *tty, int disc)
 
 	/* Restart the work queue in case no characters kick it off. Safe if
 	   already running */
+	WARN_ON(IS_ERR_OR_NULL(tty->port->itty));
 	tty_buffer_restart_work(tty->port);
 err:
 	tty_ldisc_put(new_ldisc);	/* drop the extra reference */
drivers/tty/tty_port.c +2 -2
index 1286f2478bce..70661417dc17 100644
@@ -27,7 +27,7 @@ static int tty_port_default_receive_buf(struct tty_port *port,
 	struct tty_ldisc *disc;
 
 	tty = READ_ONCE(port->itty);
-	if (!tty)
+	if (IS_ERR_OR_NULL(tty))
 		return 0;
 
 	disc = tty_ldisc_ref(tty);
@@ -253,7 +253,7 @@ static void tty_port_destructor(struct kref *kref)
 	struct tty_port *port = container_of(kref, struct tty_port, kref);
 
 	/* check if last port ref was dropped before tty release */
-	if (WARN_ON(port->itty))
+	if (WARN_ON(!IS_ERR_OR_NULL(port->itty)))
 		return;
 	if (port->xmit_buf)
 		free_page((unsigned long)port->xmit_buf);
drivers/tty/vt/keyboard.c +1 -1
index f974d6340d04..f0e3e6869908 100644
@@ -1325,7 +1325,7 @@ static void kbd_keycode(unsigned int keycode, int down, int hw_raw)
 
 	tty = vc->port.tty;
 
-	if (tty && (!tty->driver_data)) {
+	if (WARN_ON(tty && (!tty->driver_data))) {
 		/* No driver data? Strange. Okay we fix it then. */
 		tty->driver_data = vc;
 	}
Signature · 1 line
2.14.3