NULL pointer dereference / tty subsystem? / 2.6.31.1/ still persists, in stable .1 release too
Kernel 2.6.31.1
x86 , 32-bit
gcc 4.4.1
Workload - busy pppoe server, but probably it is not related. CPU Core 2 Quad
Here is panic message, but it is mixed with syslog (seems), syslog message starting from "kernel:"
Seems triggered by busybox login process?
[64130.463884] BUG: unable to handle kernel NULL pointer dereference at 00000003 [64130.463942] IP: [<c025080c>] add_echo_byte+0x94/0x9b [64130.463988] *pdpt = 0000000029004001 *pde = 0000000000000000 Sep 29 08:58:10 194.146.155.70 [64130.464044] Oops: 0002 [#1] SMP Sep 29 08:58:10 194.146.155.70 [64130.464084] last sysfs file: /sys/devices/virtual/vc/vcsa3/dev [64130.464127] Modules linked in:
netconsole
configfsact_skbeditsch_ingresssch_priocls_flowcls_u32em_metacls_basicxt_dscpxt_DSCPipt_REJECTts_bmxt_stringxt_hl
ifbcls_fwsch_tbfsch_htbact_iptact_mirredxt_MARK
slhcxt_TCPMSSxt_markxt_tcpudpiptable_mangleiptable_natnf_natnf_conntrack_ipv4nf_conntrackrtc_cmosrtc_corertc_libnf_defrag_ipv4iptable_filterip_tablesx_tables
8021q
garp
stp
llc
loopusb_storage
mtdblockmtd_blkdevs
mtd
e1000eiTCO_wdtiTCO_vendor_supportpata_acpiata_piixata_generic
libatasr_mod
cdrom
tulip
r8169
sky2via_velocityvia_rhine
sis900ne2k_pci
8390
skge
tg3
libphy
8139too
e1000
e100
usbhidohci_hcduhci_hcdehci_hcd
usbcorenls_base
Sep 29 08:58:10 194.146.155.70 [64130.464757] [64130.464790] Pid: 22690, comm: login Not tainted (2.6.31.1-build-0047 #8) [64130.464818] EIP: 0060:[<c025080c>] EFLAGS: 00010206 CPU: 1 [64130.464818] EIP is at add_echo_byte+0x94/0x9b [64130.464818] EAX: 00000000 EBX: eeca1069 ECX: 00000003 EDX: 00000069 [64130.464818] ESI: 00000069 EDI: 00000069 EBP: f49e2d6c ESP: f49e2d64 [64130.464818] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [64130.464818] Process login (pid: 22690, ti=f49e2000 task=e93b2b80 task.ti=f49e2000) [64130.464818] Stack: [64130.464818] eeca1069
eeca1000f49e2d84c0250d09eeca13dceeca1000
00000000eeca1000
Sep 29 08:58:10 194.146.155.70 [64130.464818] <0>
f49e2e94c0252181eeca100026fc795ec502a9e0eeca13f8e93b2b80f4ab341c
Sep 29 08:58:10 194.146.155.70 [64130.464818] <0>
9e59f74600002f6df4ab341ef4ab351f00002f6d
00000042c01231e1eeca1120
Sep 29 08:58:10 194.146.155.70 [64130.464818] Call Trace: [64130.464818] [<c0250d09>] ? echo_char+0x5b/0x69 [64130.464818] [<c0252181>] ? n_tty_receive_buf+0x41b/0x10a5 [64130.464818] [<c01231e1>] ? dequeue_task_fair+0x184/0x18c [64130.464818] [<c02fbcc2>] ? schedule+0x75d/0x7c7 [64130.464818] [<c0170734>] ? filemap_fault+0x69/0x2e5 [64130.464818] [<c0253dc3>] ? tty_ldisc_try+0x36/0x3c [64130.464818] [<c0254958>] ? flush_to_ldisc+0xf1/0x17f [64130.464818] [<c0254a44>] ? tty_flush_to_ldisc+0xd/0xf [64130.464818] [<c0251930>] ? n_tty_read+0x2ac/0x5ad [64130.464818] [<c01293e5>] ? default_wake_function+0x0/0xd [64130.464818] [<c0251684>] ? n_tty_read+0x0/0x5ad [64130.464818] [<c024e212>] ? tty_read+0x62/0x99 [64130.464818] [<c024e1b0>] ? tty_read+0x0/0x99 [64130.464818] [<c0190e4a>] ? vfs_read+0x87/0x110 [64130.464818] [<c0190f6c>] ? sys_read+0x3b/0x60 [64130.464818] [<c0102975>] ? syscall_call+0x7/0xb [64130.464818] Code:
00
ff
0f
00
00
80
8a
4c
01
00
00
40
eb
15
89
c1
40
03
8a
8c
03
00
00
89
82
90
03
00
00
81
e1
ff
0f
00
00
8b
82
88
03
00
00
89
f2
Sep 29 08:58:10 194.146.155.70
14
08
5b
5e
5d
c3
55
89
c1
83
b8
b4
00
00
00
00
89
e5
74
26
Sep 29 08:58:10 194.146.155.70 [64130.464818] EIP: [<c025080c>] add_echo_byte+0x94/0x9b SS:ESP 0068:f49e2d64 [64130.464818] CR2: 0000000000000003
kernel: [64130.463884] BUG: unable to handle kernel NULL pointer dereference at 00000003
kernel: [64130.463942] IP: [<c025080c>] add_echo_byte+0x94/0x9b
kernel: [64130.463988] *pdpt = 0000000029004001 *pde = 0000000000000000
kernel: [64130.464044] Oops: 0002 [#1] SMP
kernel: [64130.464084] last sysfs file: /sys/devices/virtual/vc/vcsa3/dev
kernel: [64130.464127] Modules linked in: netconsole configfs act_skbedit sch_ingress sch_prio cls_flow cls_u32 em_meta cls_basic xt_dscp xt_DSCP ipt_REJECT ts_bm xt_strin
g xt_hl ifb cls_fw sch_tbf sch_htb act_ipt act_mirred xt_MARK pp
kernel: [64130.464757]
kernel: [64130.464790] Pid: 22690, comm: login Not tainted (2.6.31.1-build-0047 #8)
kernel: [64130.464818] EIP: 0060:[<c025080c>] EFLAGS: 00010206 CPU: 1
kernel: [64130.464818] EIP is at add_echo_byte+0x94/0x9b
kernel: [64130.464818] EAX: 00000000 EBX: eeca1069 ECX: 00000003 EDX: 00000069
[64130.467376] ---[ end trace 2aeb2797ef9a99d4 ]---
kernel: [64130.464818] ESI: 00000069 EDI: 00000069 EBP: f49e2d6c ESP: f49e2d64
kernel: [64130.464818] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
kernel: [64130.464818] Process login (pid: 22690, ti=f49e2000 task=e93b2b80 task.ti=f49e2000)
kernel: [64130.464818] Stack:
kernel: [64130.464818] eeca1069 eeca1000 f49e2d84 c0250d09 eeca13dc eeca1000 00000000 eeca1000
kernel: [64130.464818] <0> f49e2e94 c0252181 eeca1000 26fc795e c502a9e0 eeca13f8 e93b2b80 f4ab341c
kernel: [64130.464818] <0> 9e59f746 00002f6d f4ab341e f4ab351f 00002f6d 00000042 c01231e1 eeca1120
kernel: [64130.464818] Call Trace:
kernel: [64130.464818] [<c0250d09>] ? echo_char+0x5b/0x69
kernel: [64130.464818] [<c0252181>] ? n_tty_receive_buf+0x41b/0x10a5
kernel: [64130.464818] [<c01231e1>] ? dequeue_task_fair+0x184/0x18c
[64130.467477] Kernel panic - not syncing: Fatal exception
kernel: [64130.464818] [<c02fbcc2>] ? schedule+0x75d/0x7c7
kernel: [64130.464818] [<c0170734>] ? filemap_fault+0x69/0x2e5
kernel: [64130.464818] [<c0253dc3>] ? tty_ldisc_try+0x36/0x3c
kernel: [64130.464818] [<c0254958>] ? flush_to_ldisc+0xf1/0x17f
kernel: [64130.464818] [<c0254a44>] ? tty_flush_to_ldisc+0xd/0xf
kernel: [64130.464818] [<c0251930>] ? n_tty_read+0x2ac/0x5ad
kernel: [64130.464818] [<c01293e5>] ? default_wake_function+0x0/0xd
kernel: [64130.464818] [<c0251684>] ? n_tty_read+0x0/0x5ad
kernel: [64130.464818] [<c024e212>] ? tty_read+0x62/0x99
kernel: [64130.464818] [<c024e1b0>] ? tty_read+0x0/0x99
kernel: [64130.464818] [<c0190e4a>] ? vfs_read+0x87/0x110
[64130.467554] Pid: 22690, comm: login Tainted: G D
2.6.31.1-build-0047 #8
kernel: [64130.464818] [<c0190f6c>] ? sys_read+0x3b/0x60
kernel: [64130.464818] [<c0102975>] ? syscall_call+0x7/0xb
kernel: [64130.464818] Code: 00 ff 0f 00 00 80 8a 4c 01 00 00 40 eb 15 89 c1 40 03 8a 8c 03 00 00 89 82 90 03 00 00 81 e1 ff 0f 00 00 8b 82 88 03 00 00 89 f2 <88> 14 08 5b
5e 5d c3 55 89 c1 83 b8 b4 00 00 00 00 89 e5 74 26
kernel: [64130.464818] EIP: [<c025080c>] add_echo_byte+0x94/0x9b SS:ESP 0068:f49e2d64
kernel: [64130.464818] CR2: 0000000000000003
kernel: [64130.467376] ---[ end trace 2aeb2797ef9a99d4 ]---
kernel: [64130.467477] Kernel panic - not syncing: Fatal exception
kernel: [64130.467554] Pid: 22690, comm: login Tainted: G D
2.6.31.1-build-0047 #8
kernel: [64130.467671] Call Trace:
[64130.467671] Call Trace: [64130.467772] [<c02fb2f0>] ? printk+0xf/0x17 [64130.467889] [<c02fb241>] panic+0x39/0xd9 [64130.467945] [<c01059b7>] oops_end+0x8b/0x9a [64130.468003] [<c0118f49>] no_context+0x13d/0x147 [64130.468076] [<c0119066>] __bad_area_nosemaphore+0x113/0x11b
kernel: [64130.467772] [<c02fb2f0>] ? printk+0xf/0x17
kernel: [64130.467889] [<c02fb241>] panic+0x39/0xd9
kernel: [64130.467945] [<c01059b7>] oops_end+0x8b/0x9a
kernel: [64130.468003] [<c0118f49>] no_context+0x13d/0x147
kernel: [64130.468076] [<c0119066>] __bad_area_nosemaphore+0x113/0x11b
kernel: [64130.468144] [<c024d417>] ? extract_buf+0x78/0xc8
[64130.468144] [<c024d417>] ? extract_buf+0x78/0xc8 [64130.468281] [<c01190ae>] bad_area+0x30/0x39 [64130.468340] [<c0119334>] do_page_fault+0x16b/0x26f [64130.468419] [<c01191c9>] ? do_page_fault+0x0/0x26f [64130.468507] [<c02fd346>] error_code+0x66/0x6c [64130.468569] [<c01191c9>] ? do_page_fault+0x0/0x26f [64130.468631] [<c025080c>] ? add_echo_byte+0x94/0x9b [64130.468687] [<c0250d09>] echo_char+0x5b/0x69 [64130.468760] [<c0252181>] n_tty_receive_buf+0x41b/0x10a5 [64130.468825] [<c01231e1>] ? dequeue_task_fair+0x184/0x18c [64130.468890] [<c02fbcc2>] ? schedule+0x75d/0x7c7 [64130.468948] [<c0170734>] ? filemap_fault+0x69/0x2e5 [64130.469013] [<c0253dc3>] ? tty_ldisc_try+0x36/0x3c [64130.469074] [<c0254958>] flush_to_ldisc+0xf1/0x17f [64130.469141] [<c0254a44>] tty_flush_to_ldisc+0xd/0xf
kernel: [64130.468281] [<c01190ae>] bad_area+0x30/0x39
kernel: [64130.468340] [<c0119334>] do_page_fault+0x16b/0x26f
kernel: [64130.468419] [<c01191c9>] ? do_page_fault+0x0/0x26f
kernel: [64130.468507] [<c02fd346>] error_code+0x66/0x6c
kernel: [64130.468569] [<c01191c9>] ? do_page_fault+0x0/0x26f
kernel: [64130.468631] [<c025080c>] ? add_echo_byte+0x94/0x9b
kernel: [64130.468687] [<c0250d09>] echo_char+0x5b/0x69
kernel: [64130.468760] [<c0252181>] n_tty_receive_buf+0x41b/0x10a5
kernel: [64130.468825] [<c01231e1>] ? dequeue_task_fair+0x184/0x18c
kernel: [64130.468890] [<c02fbcc2>] ? schedule+0x75d/0x7c7
kernel: [64130.468948] [<c0170734>] ? filemap_fault+0x69/0x2e5
[64130.469212] [<c0251930>] n_tty_read+0x2ac/0x5ad
kernel: [64130.469013] [<c0253dc3>] ? tty_ldisc_try+0x36/0x3c
kernel: [64130.469074] [<c0254958>] flush_to_ldisc+0xf1/0x17f
kernel: [64130.469141] [<c0254a44>] tty_flush_to_ldisc+0xd/0xf
kernel: [64130.469212] [<c0251930>] n_tty_read+0x2ac/0x5ad
[64130.469295] [<c01293e5>] ? default_wake_function+0x0/0xd [64130.469365] [<c0251684>] ? n_tty_read+0x0/0x5ad [64130.469420] [<c024e212>] tty_read+0x62/0x99 [64130.469483] [<c024e1b0>] ? tty_read+0x0/0x99 [64130.469562] [<c0190e4a>] vfs_read+0x87/0x110 [64130.469620] [<c0190f6c>] sys_read+0x3b/0x60 [64130.469678] [<c0102975>] syscall_call+0x7/0xb [64130.469741] Rebooting in 5 seconds..
Quoted 1 line · Denys Fedoryschenko
Kernel 2.6.31.1
2.6.31.1 still has a number of tty problems. I'm working through all of the patches right now to hopefully have 2.6.31.2 fixed in this area.
thanks,
greg k-h