proc_pid_stat crashes in 2.6.2 (was 2.6.0-test11: Crash in ps axH)
Hi,
I already reported this crash twice on 2.6.0-test11, and now I
reproduced it on something more fresh - on 2.6.2-bk-something like it was actual on Feb 4. It is really annoying, as any user can do 'ps axH' and crash system after some uptime :-( Crash is identical to one I got with 2.6.0-test11, so it looks like that I'll have to go back to 2.4.x on publicly accessible machines. After about 14 days uptime.
I've got no replies to previous reports.
Thanks,
Petr Vandrovec [email protected]
Unable to handle kernel NULL pointer dereference at virtual address 00000df9 printing eip: c018ea16 *pde = 00000000 Oops: 0000 [#1] CPU: 0 EIP: 0060:[<c018ea16>] Not tainted EFLAGS: 00010286
EIP is at proc_pid_stat+0xa6/0x510
eax: 000203ea ebx: 00000d9d ecx: d6662000 edx: edc1e4bc
esi: ed3c2ce0 edi: edc1e4bc ebp: 00000000 esp: e6745e3c
ds: 007b es: 007b ss: 0068
Process ps (pid: 19007, threadinfo=e6744000 task=ed3c3940)
Stack: edc1e4bc f3acf3a8 c0401150 ed3c2ce0 e81c74fc ed3c2ce0 0000001d c018c2fc
ed3c2ce0 4034fdab 28e18b6e c0401150 f3acf42c c0401150 c0401150 c018c82e
f3acf3a8 e81c74fc 0000001d 00000004 ed3c2ce0 ffffffea fffffff4 e81c76e0
Call Trace:
[<c018c2fc>] proc_pid_make_inode+0x9c/0xd0
[<c018c82e>] proc_pident_lookup+0xfe/0x260
[<c016cf75>] real_lookup+0xd5/0x100
[<c01433c4>] buffered_rmqueue+0xd4/0x180
[<c0145e3b>] check_poison_obj+0x2b/0x1a0
[<c015fc48>] get_empty_filp+0x68/0xe0
[<c018b7d4>] proc_info_read+0x54/0x150
[<c015df48>] filp_open+0x68/0x70
[<c015ed78>] vfs_read+0xb8/0x130
[<c015e3be>] sys_open+0x7e/0x90
[<c015f022>] sys_read+0x42/0x70
[<c010939f>] syscall_call+0x7/0xb
Code: 0f bf 43 5c 0f bf 5b 5e c1 e0 14 09 d8 8b 59 08 01 d8 89 c1Linux usermap 2.6.2-c1549 #1 SMP Wed Feb 4 23:42:14 CET 2004 i686 GNU/Linux
----- Forwarded message from Petr Vandrovec <[email protected]> -----
Date: Wed, 4 Feb 2004 18:07:48 +0100
From: Petr Vandrovec <[email protected]>
To: [email protected]
Subject: 2.6.0-test11: Crash in ps axH
Hi,
maybe that this problem is already fixed in 2.6.2-rc3, but just
in case that somebody has a clue what's going on, after about month of uptime, running couple of multithreaded apps on 2.6.0-test11 (of which slapd was running for this month without restart), today I decided to use 'ps axH' to find what server is doing.
In the middle of 'ps axH' output I was awarded by 'Segmentation
fault', and following oops appeared in kernel log. It died in proc_pid_stat when doing tty_devnum(task->tty) because task->tty->driver was 0x269BD005 (see %ebx). I'm not able to explain what happened, but as it happened second time on this system, and once on other one (also running 2.6.0-test11), there is certainly something wrong... It may be also interesting that task->tty->pgrp was 0x4E70F005 (%eax), so it looks like that task->tty points just to the garbage (task was 0xF7C12080 (%esi), and its address looks safe. task->tty was 0xC82B2000 (%ecx), and it also looks like possible value for tty).
Kernel is SMP, PIII, non-preempt, with DEBUG_SLAB enabled, 4GB
config. Machine is dual PIII/933MHz, with 1.25GB of memory.
I'm now building latest 2.6.2 from bk, but as it usually takes
about one month to get these oopses, it may take a while until I can report success/failure.
Best regards,
Petr Vandrovec
P.S.: Sorry, I noticed only after crash that Debian still does not run klogd with -x, so numbers were converted to symbolics in the trace.
Feb 4 17:18:06 usermap kernel: Unable to handle kernel paging request at virtual address 269bd061 Feb 4 17:18:06 usermap kernel: printing eip: Feb 4 17:18:06 usermap kernel: c018cbc6 Feb 4 17:18:06 usermap kernel: *pde = 00000000 Feb 4 17:18:06 usermap kernel: Oops: 0000 [#1] Feb 4 17:18:06 usermap kernel: CPU: 1 Feb 4 17:18:06 usermap kernel: EIP: 0060:[proc_pid_stat+166/1296] Not tainted Feb 4 17:18:06 usermap kernel: EFLAGS: 00010286 Feb 4 17:18:06 usermap kernel: EIP is at proc_pid_stat+0xa6/0x510 Feb 4 17:18:06 usermap kernel: eax: 4e70f005 ebx: 269bd005 ecx: c82b2000 edx: db55f2a4 Feb 4 17:18:06 usermap kernel: esi: f7c12080 edi: db55f2a4 ebp: 00000000 esp: e82e7e3c Feb 4 17:18:06 usermap kernel: ds: 007b es: 007b ss: 0068 Feb 4 17:18:06 usermap kernel: Process ps (pid: 6437, threadinfo=e82e6000 task=d72346b0) Feb 4 17:18:06 usermap kernel: Stack: db55f2a4 c1cd3104 d26d729c c03fa790 f7c12080 db8194fc f7c12080 0000001d Feb 4 17:18:06 usermap kernel: c018a53c f7c12080 40211b3e 16ff7c54 c03fa790 d26d7320 c03fa790 c03fa790 Feb 4 17:18:06 usermap kernel: c018aa6e d26d729c db8194fc 0000001d 00000004 f7c12080 ffffffea fffffff4 Feb 4 17:18:06 usermap kernel: Call Trace: Feb 4 17:18:06 usermap kernel: [proc_pid_make_inode+156/208] proc_pid_make_inode+0x9c/0xd0 Feb 4 17:18:06 usermap kernel: [proc_pident_lookup+254/608] proc_pident_lookup+0xfe/0x260 Feb 4 17:18:06 usermap kernel: [real_lookup+213/256] real_lookup+0xd5/0x100 Feb 4 17:18:06 usermap kernel: [buffered_rmqueue+212/384] buffered_rmqueue+0xd4/0x180 Feb 4 17:18:06 usermap kernel: [check_poison_obj+43/416] check_poison_obj+0x2b/0x1a0 Feb 4 17:18:06 usermap kernel: [__alloc_pages+176/832] __alloc_pages+0xb0/0x340 Feb 4 17:18:06 usermap kernel: [get_empty_filp+104/224] get_empty_filp+0x68/0xe0 Feb 4 17:18:06 usermap kernel: [proc_info_read+84/336] proc_info_read+0x54/0x150 Feb 4 17:18:06 usermap kernel: [filp_open+104/112] filp_open+0x68/0x70 Feb 4 17:18:06 usermap kernel: [vfs_read+184/304] vfs_read+0xb8/0x130 Feb 4 17:18:06 usermap kernel: [sys_open+126/144] sys_open+0x7e/0x90 Feb 4 17:18:06 usermap kernel: [sys_read+66/112] sys_read+0x42/0x70 Feb 4 17:18:06 usermap kernel: [syscall_call+7/11] syscall_call+0x7/0xb Feb 4 17:18:06 usermap kernel: Feb 4 17:18:06 usermap kernel: Code: 0f bf 43 5c 0f bf 5b 5e c1 e0 14 09 d8 8b 59 08 01 d8 89 c1
----- End forwarded message -----
Quoted 8 lines
Hi,
I already reported this crash twice on 2.6.0-test11, and now I
reproduced it on something more fresh - on 2.6.2-bk-something like it was actual on Feb 4. It is really annoying, as any user can do 'ps axH' and crash system after some uptime :-( Crash is identical to one I got with 2.6.0-test11, so it looks like that I'll have to go back to 2.4.x on publicly accessible machines. After about 14 days uptime.
Quoted 4 lines
I've got no replies to previous reports.
Thanks,
Petr Vandrovec [email protected]
what version of procps?
Quoted 8 lines · Kieran
Quoted 8 lines
Hi,
I already reported this crash twice on 2.6.0-test11, and now I
reproduced it on something more fresh - on 2.6.2-bk-something like it was actual on Feb 4. It is really annoying, as any user can do 'ps axH' and crash system after some uptime :-( Crash is identical to one I got with 2.6.0-test11, so it looks like that I'll have to go back to 2.4.x on publicly accessible machines. After about 14 days uptime.
Quoted 6 lines
Quoted 4 lines
I've got no replies to previous reports.
Thanks,
Petr Vandrovec [email protected]what version of procps?
Not that it should matter, as walking through /proc/* could do same, but... 3.1.15-3. I have no idea what procps were installed at the beginning of February or in the December. Until crash 'ps ax' does not cause problem. Only 'axH', which descends to /task/ subdirectories, crashes.
usermap:~# dpkg -l procps Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-==========================-==========================-==================================================================== ii procps 3.1.15-3 The /proc file system utilities usermap:~#
Petr Vandrovec
Quoted 14 lines · Petr Vandrovec
Quoted 13 lines · Kieran
Quoted 12 lines
Hi,
I already reported this crash twice on 2.6.0-test11, and now I
reproduced it on something more fresh - on 2.6.2-bk-something like it was actual on Feb 4. It is really annoying, as any user can do 'ps axH' and crash system after some uptime :-( Crash is identical to one I got with 2.6.0-test11, so it looks like that I'll have to go back to 2.4.x on publicly accessible machines. After about 14 days uptime.I've got no replies to previous reports.
Thanks,
Petr Vandrovec [email protected]
Quoted 7 lines
Quoted 1 line
what version of procps?
Not that it should matter, as walking through
/proc/*could do same, but... 3.1.15-3. I have no idea what procps were installed at the beginning of February or in the December. Until crash 'ps ax' does not cause problem. Only 'axH', which descends to /task/ subdirectories, crashes.
With Kieran questions I was able to get quickly reproducible case:
Connect through ssh.
Stop both openldap's I have running (/etc/init.d/slapd-perl stop; /etc/init.d/slapd stop) Start them again (/etc/init.d/slapd start; /etc/init.d/slapd-perl start) Disconnect.
Connect back.
Stop ssh (/etc/init.d/ssh stop)
Do 'ps axH'. Kaboom... Well, it is a bit different, but...
Petr Vandrovec
Unable to handle kernel paging request at virtual address 6962738b printing eip: c018ea16 *pde = 00000000 Oops: 0000 [#1] CPU: 0 EIP: 0060:[<c018ea16>] Not tainted EFLAGS: 00010286
EIP is at proc_pid_stat+0xa6/0x510
eax: 31203639 ebx: 6962732f ecx: ea43c000 edx: ea5c12a0
esi: ea5f1940 edi: ea5c12a0 ebp: 00000000 esp: f3263e3c
ds: 007b es: 007b ss: 0068
Process ps (pid: 537, threadinfo=f3262000 task=ea6fd310)
Stack: ea5c12a0 ea62c000 f3262000 f3263e9f c025d6b7 ea62c000 f3263ea0 00000000
00000000 0000002f 6c726570 7061646c 6374652f f7c3ed14 c1ccc854 f7c3ed14
c0176991 00000000 f7e4b648 f7c3ed14 c1ccc854 f7c3ed14 c0176991 00000000
Call Trace:
[<c025d6b7>] pty_write+0x1c7/0x1d0
[<c0176991>] dput+0x31/0x1b0
[<c0176991>] dput+0x31/0x1b0
[<c018c371>] pid_revalidate+0x41/0xd0
[<c01433c4>] buffered_rmqueue+0xd4/0x180
[<c0145e3b>] check_poison_obj+0x2b/0x1a0
[<c015fc48>] get_empty_filp+0x68/0xe0
[<c018b7d4>] proc_info_read+0x54/0x150
[<c015df48>] filp_open+0x68/0x70
[<c015ed78>] vfs_read+0xb8/0x130
[<c015e3be>] sys_open+0x7e/0x90
[<c015f022>] sys_read+0x42/0x70
[<c010939f>] syscall_call+0x7/0xb
Code: 0f bf 43 5c 0f bf 5b 5e c1 e0 14 09 d8 8b 59 08 01 d8 89 c1