2.6.24 regression: pan hanging unkilleable and un-straceable
Subsystems (2)
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task 0 8063 1 Jan 21 21:45:01 Anastacia kernel: ssh S 0000000000000000 0 8323 6809 Jan 21 21:45:01 Anastacia kernel: ffff81000a51f9c8 0000000000000082 ffff81000ed6dc00 ffffffff8045ad6f Jan 21 21:45:01 Anastacia kernel: ffffffff805875b8 ffffffff80623980 ffffffff80623980 ffffffff80623980 Jan 21 21:45:01 Anastacia kernel: ffffffff8061fe80 ffffffff80623980 ffff81003941b8a8 ffffffff8043142b Jan 21 21:45:01 Anastacia kernel: Call Trace: Jan 21 21:45:01 Anastacia kernel: [arp_bind_neighbour+143/208] arp_bind_neighbour+0x8f/0xd0 Jan 21 21:45:01 Anastacia kernel: [rt_intern_hash+955/1056] rt_intern_hash+0x3bb/0x420 Jan 21 21:45:01 Anastacia kernel: [nommu_map_single+56/96] nommu_map_single+0x38/0x60 Jan 21 21:45:01 Anastacia kernel: [schedule_timeout+149/208] schedule_timeout+0x95/0xd0 Jan 21 21:45:01 Anastacia kernel: [tty_ldisc_deref+82/128] tty_ldisc_deref+0x52/0x80 Jan 21 21:45:01 Anastacia kernel: [tty_poll+145/160] tty_poll+0x91/0xa0 Jan 21 21:45:01 Anastacia kernel: [do_select+1128/1376] do_select+0x468/0x560 Jan 21 21:45:01 Anastacia kernel: [__pollwait+0/304] __pollwait+0x0/0x130 Jan 21 21:45:01 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 21 21:45:01 Anastacia kernel:last message repeated 2 times Jan 21 21:45:01 Anastacia kernel: [enqueue_task+19/48] enqueue_task+0x13/0x30 Jan 21 21:45:01 Anastacia kernel: [try_to_wake_up+98/720] try_to_wake_up+0x62/0x2d0 Jan 21 21:45:01 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 21 21:45:01 Anastacia kernel: [tcp_recvmsg+1463/3360] tcp_recvmsg+0x5b7/0xd20 Jan 21 21:45:01 Anastacia kernel: [__wake_up_common+90/144] __wake_up_common+0x5a/0x90 Jan 21 21:45:01 Anastacia kernel: [__wake_up+67/112] __wake_up+0x43/0x70 Jan 21 21:45:01 Anastacia kernel: [n_tty_receive_buf+821/3888] n_tty_receive_buf+0x335/0xf30 Jan 21 21:45:01 Anastacia kernel: [sock_aio_read+349/368] sock_aio_read+0x15d/0x170 Jan 21 21:45:01 Anastacia kernel: [core_sys_select+521/768] core_sys_select+0x209/0x300 Jan 21 21:45:01 Anastacia kernel: [remove_wait_queue+25/96] remove_wait_queue+0x19/0x60 Jan 21 21:45:01 Anastacia kernel: [__wake_up+67/112] __wake_up+0x43/0x70 Jan 21 21:45:01 Anastacia kernel: [tty_ldisc_deref+82/128] tty_ldisc_deref+0x52/0x80 Jan 21 21:45:01 Anastacia kernel: [tty_write+569/592] tty_write+0x239/0x250 Jan 21 21:45:01 Anastacia kernel: [sys_select+68/448] sys_select+0x44/0x1c0 Jan 21 21:45:01 Anastacia kernel: [sys_write+83/144] sys_write+0x53/0x90 Jan 21 21:45:01 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83
What could be causing this problem?
Signature · 1 line
Frederik Himpe <[email protected]>
Quoted 8 lines · Frederik Himpe
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task 0
Well I've twice tried to submit a patch to print stacks for running tasks as well, but nobody seems interested. It would at least give a chance to see something.
Can you post a few Sysrq+P traces?
Quoted 7 lines · Nick Piggin
Quoted 6 lines · Frederik Himpe
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 6 lines
Quoted 2 lines
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task 0Well I've twice tried to submit a patch to print stacks for running tasks as well, but nobody seems interested. It would at least give a chance to see something.
I've hit same twice recently (not pan, and not repeatable).
Quoted 12 lines · Mike Galbraith
Quoted 11 lines · Nick Piggin
Quoted 9 lines · Frederik Himpe
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task0
Quoted 5 lines
Quoted 3 lines
Well I've twice tried to submit a patch to print stacks for running tasks as well, but nobody seems interested. It would at least give a chance to see something.
I've hit same twice recently (not pan, and not repeatable).
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
Any luck getting traces?
kernel/sched.c +1 -2
@@ -4920,8 +4920,7 @@ static void show_task(struct task_structprintk (KERN_CONT "%5lu %5d %6d\n" , free,task_pid_nr (p),task_pid_nr (p->real_parent)); -if (state !=TASK_RUNNING ) -show_stack (p,NULL ); +show_stack (p,NULL ); }void show_state_filter (unsigned long state_filter)
Quoted 6 lines · Nick Piggin
Quoted 1 line · Mike Galbraith
I've hit same twice recently (not pan, and not repeatable).
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
SMP (P4/HT imitating real cores)
Quoted 1 line
Any luck getting traces?
We'll see. Armed.
-Mike
Quoted 5 lines · Mike Galbraith
Quoted 4 lines · Nick Piggin
Quoted 1 line · Mike Galbraith
I've hit same twice recently (not pan, and not repeatable).
Quoted 8 lines
Quoted 2 lines
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
SMP (P4/HT imitating real cores)
Quoted 1 line
Any luck getting traces?
We'll see. Armed.
Hm. ld just went loopy (but killable) in v2.6.24-6928-g9135f19. During kbuild, modpost segfaulted, restart build, ld goes gaga. Third attempt, build finished. Not what I hit before, but mentionable.
[ 674.589134] modpost[18588]: segfault at 3e8dc42c ip 0804a96d sp af982920 error 5 in modpost[8048000+9000] [ 674.589211] mm/memory.c:115: bad pgd 3e081163. [ 674.589214] mm/memory.c:115: bad pgd 3e0d2163. [ 674.589217] mm/memory.c:115: bad pgd 3eb01163.
[ 1407.322144] ======================= [ 1407.322144] ld R running 0 21963 21962 [ 1407.322144] db9d7f1c 00200086 c75f9020 b1814300 b0428300 b0428300 b0428300 c75f9280 [ 1407.322144] b1814300 00000001 db9d7000 00000000 d08c2f90 dba4f300 00000002 00000000 [ 1407.322144] b1810120 dba4f334 00200046 ffffffff db9d7000 c75f9020 db9d7f30 b02f333f [ 1407.322144] Call Trace: [ 1407.322144] [<b02f333f>] preempt_schedule_irq+0x45/0x5b [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b0104d6e>] need_resched+0x1f/0x21 [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b0117a5c>] ? do_page_fault+0x4c/0x470 [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b02f4a3a>] ? error_code+0x72/0x78 [ 1407.322144] [<b02f0000>] ? init_transmeta+0xcf/0x22f <== zzt P4
Quoted 19 lines · Mike Galbraith
Quoted 4 lines · Mike Galbraith
Quoted 3 lines · Nick Piggin
Quoted 1 line · Mike Galbraith
I've hit same twice recently (not pan, and not repeatable).
Quoted 8 lines
Quoted 2 lines
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
SMP (P4/HT imitating real cores)
Quoted 1 line
Any luck getting traces?
We'll see. Armed.
Hm. ld just went loopy (but killable) in v2.6.24-6928-g9135f19. During kbuild, modpost segfaulted, restart build, ld goes gaga. Third attempt, build finished. Not what I hit before, but mentionable.
Quoted 5 lines
[ 674.589134] modpost[18588]: segfault at 3e8dc42c ip 0804a96d sp af982920 error 5 in modpost[8048000+9000] [ 674.589211] mm/memory.c:115: bad pgd 3e081163. [ 674.589214] mm/memory.c:115: bad pgd 3e0d2163. [ 674.589217] mm/memory.c:115: bad pgd 3eb01163.
Hmm, this _could_ be bad memory. Or if it is very easy to reproduce with a particular kernel version, then it is probably a memory scribble from another part of the kernel :(
First thing I guess would be easy and helpful to run memtest86 for a while if you have time.
If that's clean, then I don't have another good option except to bisect the problem. Turning on DEBUG_VM, DEBUG_SLAB, DEBUG_LIST, DEBUG_PAGEALLOC, DEBUG_STACKOVERFLOW, DEBUG_RODATA might help catch it sooner... SLAB and PAGEALLOC could slow you down quite a bit though. And if the problem is quite reproduceable, then obviously don't touch your config ;)
Thanks,
Nick
Quoted 16 lines
[ 1407.322144] ======================= [ 1407.322144] ld R running 0 21963 21962 [ 1407.322144] db9d7f1c 00200086 c75f9020 b1814300 b0428300 b0428300
b0428300c75f9280[ 1407.322144]b181430000000001db9d700000000000d08c2f90dba4f30000000002 00000000 [ 1407.322144]b1810120dba4f334
00200046 ffffffffdb9d7000c75f9020db9d7f30b02f333f[ 1407.322144] Call Trace:[ 1407.322144] [<b02f333f>] preempt_schedule_irq+0x45/0x5b [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b0104d6e>] need_resched+0x1f/0x21 [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b0117a5c>] ? do_page_fault+0x4c/0x470 [ 1407.322144] [<b0117a10>] ? do_page_fault+0x0/0x470 [ 1407.322144] [<b02f4a3a>] ? error_code+0x72/0x78 [ 1407.322144] [<b02f0000>] ? init_transmeta+0xcf/0x22f <== zzt P4
kernel/sched.c static void show_task(struct task_struct · line 4923
Quoted 11 lines · Nick Piggin
Index:
linux-2.6/kernel/sched.c
===================================================================kernel/sched.c +1
-2@@ -4920,8 +4920,7 @@ static void show_task(struct task_structprintk (KERN_CONT "%5lu %5d %6d\n" , free,task_pid_nr (p),task_pid_nr (p->real_parent)); -if (state !=TASK_RUNNING ) -show_stack (p,NULL ); +show_stack (p,NULL );
thanks - applied to sched-devel.git. We'll see whether it causes any problems.
Ingo
kernel/sched.c static void show_task(struct task_struct · line 4924
Quoted 13 lines · Nick Piggin
Index:
linux-2.6/kernel/sched.c
===================================================================kernel/sched.c +1
-2@@ -4920,8 +4920,7 @@ static void show_task(struct task_structprintk (KERN_CONT "%5lu %5d %6d\n" , free,task_pid_nr (p),task_pid_nr (p->real_parent)); -if (state !=TASK_RUNNING ) -show_stack (p,NULL ); +show_stack (p,NULL ); }
Maybe something like this would be better?
if (state == TASK_RUNNING)
printk("running task, stack trace may be inaccurate\n");
show_stack(p, NULL);Just a thought....
Quoted 7 lines · Nick Piggin
Quoted 7 lines
Quoted 7 lines · Frederik Himpe
Quoted 6 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task0
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
Any luck getting traces?
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
Jan 26 14:17:43 Anastacia kernel: pan R running task 0 7564 1 Jan 26 14:17:43 Anastacia kernel: 000000003f5b3248 0000000000001000 ffffffff880c28b0 0000000000000000 Jan 26 14:17:43 Anastacia kernel: ffff81003f5b3248 ffff81002d1ed900 000000002d1ed900 0000000000000000 Jan 26 14:17:43 Anastacia kernel: ffff810016050dd0 fffff000fffff000 0000000000000000 ffff81002d1eda10 Jan 26 14:17:43 Anastacia kernel: Call Trace: Jan 26 14:17:43 Anastacia kernel: [_end+127964408/2129947720] :xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [unix_poll+0/176] unix_poll+0x0/0xb0 Jan 26 14:17:43 Anastacia kernel: [_end+127964408/2129947720] :xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [iov_iter_copy_from_user_atomic+65/160] iov_iter_copy_from_user_atomic+0x41/0xa0 Jan 26 14:17:43 Anastacia kernel: [iov_iter_copy_from_user_atomic+46/160] iov_iter_copy_from_user_atomic+0x2e/0xa0 Jan 26 14:17:43 Anastacia kernel: [generic_file_buffered_write+383/1728] generic_file_buffered_write+0x17f/0x6c0 Jan 26 14:17:43 Anastacia kernel: [current_fs_time+30/48] current_fs_time+0x1e/0x30 Jan 26 14:17:43 Anastacia kernel: [_end+127997742/2129947720] :xfs:xfs_write+0x676/0x910 Jan 26 14:17:43 Anastacia kernel: [find_lock_page+61/192] find_lock_page+0x3d/0xc0 Jan 26 14:17:43 Anastacia kernel: [_end+127981080/2129947720] :xfs:xfs_file_aio_write+0x0/0x50 Jan 26 14:17:43 Anastacia kernel: [do_sync_readv_writev+203/272] do_sync_readv_writev+0xcb/0x110 Jan 26 14:17:43 Anastacia kernel: [__do_fault+501/1056] __do_fault+0x1f5/0x420 Jan 26 14:17:43 Anastacia kernel: [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 26 14:17:43 Anastacia kernel: [handle_mm_fault+1344/2048] handle_mm_fault+0x540/0x800 Jan 26 14:17:43 Anastacia kernel: [rw_copy_check_uvector+157/336] rw_copy_check_uvector+0x9d/0x150 Jan 26 14:17:43 Anastacia kernel: [do_readv_writev+253/560] do_readv_writev+0xfd/0x230 Jan 26 14:17:43 Anastacia kernel: [sys_writev+83/144] sys_writev+0x53/0x90 Jan 26 14:17:43 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83 Jan 26 14:17:43 Anastacia kernel: Jan 26 14:17:43 Anastacia kernel: pan S 0000000000000000 0 7565 1 Jan 26 14:17:43 Anastacia kernel: ffff810001401c58 0000000000000086 ffff810001401bb8 ffff81003cd3a280 Jan 26 14:17:43 Anastacia kernel: ffff81003cd3a300 ffffffff80623980 ffffffff80623980 ffffffff80623980 Jan 26 14:17:43 Anastacia kernel: ffffffff8061fe80 ffffffff80623980 ffff810001bcc9a8 ffff8100299b34e8 Jan 26 14:17:43 Anastacia kernel: Call Trace: Jan 26 14:17:43 Anastacia kernel: [__qdisc_run+173/528] __qdisc_run+0xad/0x210 Jan 26 14:17:43 Anastacia kernel: [dev_queue_xmit+216/768] dev_queue_xmit+0xd8/0x300 Jan 26 14:17:43 Anastacia kernel: [futex_wait+838/912] futex_wait+0x346/0x390 Jan 26 14:17:43 Anastacia kernel: [tcp_connect+851/896] tcp_connect+0x353/0x380 Jan 26 14:17:43 Anastacia kernel: [tcp_v4_connect+914/1696] tcp_v4_connect+0x392/0x6a0 Jan 26 14:17:43 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [do_futex+287/3008] do_futex+0x11f/0xbc0 Jan 26 14:17:43 Anastacia kernel: [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20 Jan 26 14:17:43 Anastacia kernel: [cp_new_stat+229/256] cp_new_stat+0xe5/0x100 Jan 26 14:17:43 Anastacia kernel: [sys_futex+171/304] sys_futex+0xab/0x130 Jan 26 14:17:43 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83 Jan 26 14:17:43 Anastacia kernel: Jan 26 14:17:43 Anastacia kernel: pan S 0000000000000000 0 7566 1 Jan 26 14:17:43 Anastacia kernel: ffff8100013fdc58 0000000000000086 0000000000a492c0 0000000000a493c8 Jan 26 14:17:43 Anastacia kernel: 0000000000a494d0 ffffffff80623980 ffffffff80623980 ffffffff80623980 Jan 26 14:17:43 Anastacia kernel: ffffffff8061fe80 ffffffff80623980 ffff810001bcd8a8 ffff8100339756c8 Jan 26 14:17:43 Anastacia kernel: Call Trace: Jan 26 14:17:43 Anastacia kernel: [enqueue_entity+55/112] enqueue_entity+0x37/0x70 Jan 26 14:17:43 Anastacia kernel: [enqueue_task_fair+56/80] enqueue_task_fair+0x38/0x50 Jan 26 14:17:43 Anastacia kernel: [futex_wait+838/912] futex_wait+0x346/0x390 Jan 26 14:17:43 Anastacia kernel: [__wake_up+67/112] __wake_up+0x43/0x70 Jan 26 14:17:43 Anastacia kernel: [wake_futex+57/80] wake_futex+0x39/0x50 Jan 26 14:17:43 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [do_futex+287/3008] do_futex+0x11f/0xbc0 Jan 26 14:17:43 Anastacia kernel: [__up_read+33/176] __up_read+0x21/0xb0 Jan 26 14:17:43 Anastacia kernel: [do_page_fault+411/2000] do_page_fault+0x19b/0x7d0 Jan 26 14:17:43 Anastacia kernel: [sys_futex+171/304] sys_futex+0xab/0x130 Jan 26 14:17:43 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83 Jan 26 14:17:43 Anastacia kernel: Jan 26 14:17:43 Anastacia kernel: pan S 0000000000000000 0 7567 1 Jan 26 14:17:43 Anastacia kernel: ffff810001409c58 0000000000000086 0000000000a51618 0000000000a51720 Jan 26 14:17:43 Anastacia kernel: 0000000000a51828 ffffffff80623980 ffffffff80623980 ffffffff80623980 Jan 26 14:17:43 Anastacia kernel: ffffffff8061fe80 ffffffff80623980 ffff8100339758a8 ffff810033974f48 Jan 26 14:17:43 Anastacia kernel: Call Trace: Jan 26 14:17:43 Anastacia kernel: [enqueue_entity+55/112] enqueue_entity+0x37/0x70 Jan 26 14:17:43 Anastacia kernel: [enqueue_task_fair+56/80] enqueue_task_fair+0x38/0x50 Jan 26 14:17:43 Anastacia kernel: [futex_wait+838/912] futex_wait+0x346/0x390 Jan 26 14:17:43 Anastacia kernel: [__wake_up+67/112] __wake_up+0x43/0x70 Jan 26 14:17:43 Anastacia kernel: [wake_futex+57/80] wake_futex+0x39/0x50 Jan 26 14:17:43 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [do_futex+287/3008] do_futex+0x11f/0xbc0 Jan 26 14:17:43 Anastacia kernel: [__up_read+33/176] __up_read+0x21/0xb0 Jan 26 14:17:43 Anastacia kernel: [do_page_fault+411/2000] do_page_fault+0x19b/0x7d0 Jan 26 14:17:43 Anastacia kernel: [sys_futex+171/304] sys_futex+0xab/0x130 Jan 26 14:17:43 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83 Jan 26 14:17:43 Anastacia kernel: Jan 26 14:17:43 Anastacia kernel: pan S 0000000000000000 0 7568 1 Jan 26 14:17:43 Anastacia kernel: ffff8100013fbc58 0000000000000086 00002aaaabbe4320 ffffffffffffadbc Jan 26 14:17:43 Anastacia kernel: 0000000000000000 ffffffff80623980 ffffffff80623980 ffffffff80623980 Jan 26 14:17:43 Anastacia kernel: ffffffff8061fe80 ffffffff80623980 ffff810033975128 0000000001137ad0 Jan 26 14:17:43 Anastacia kernel: Call Trace: Jan 26 14:17:43 Anastacia kernel: [futex_wait+838/912] futex_wait+0x346/0x390 Jan 26 14:17:43 Anastacia kernel: [zone_statistics+125/128] zone_statistics+0x7d/0x80 Jan 26 14:17:43 Anastacia kernel: [__alloc_pages+170/976] __alloc_pages+0xaa/0x3d0 Jan 26 14:17:43 Anastacia kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [do_futex+287/3008] do_futex+0x11f/0xbc0 Jan 26 14:17:43 Anastacia kernel: [__up_read+33/176] __up_read+0x21/0xb0 Jan 26 14:17:43 Anastacia kernel: [do_page_fault+411/2000] do_page_fault+0x19b/0x7d0 Jan 26 14:17:43 Anastacia kernel: [sys_futex+171/304] sys_futex+0xab/0x130
SysRq : Show Regs
CPU 0:
Modules linked in: usb_storage af_packet nvidia(P) vboxdrv ipv6 fuse
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq cpufreq_ondemand video output tc1100_wmi sbs sbshc container dock battery ac binfmt_misc loop ext3 jbd dm_mirror sr_mod dm_mod pata_amd ata_generic sata_sil usbmouse usbhid ff_memless floppy usblp powernow_k8 freq_table snd_pcm_oss snd_mixer_oss snd_mpu401 snd_mpu401_uart snd_rawmidi ns558 gameport parport_pc snd_seq_device parport rtc_cmos pcspkr snd_intel8x0 k8temp snd_ac97_codec ohci1394 ac97_bus ieee1394 snd_pcm snd_timer skge ohci_hcd ehci_hcd snd soundcore usbcore forcedeth snd_page_alloc ssb fan pcmcia pcmcia_core i2c_nforce2 i2c_core button thermal processor sg evdev genrtc xfs scsi_wait_scan sd_mod sata_nv libata scsi_mod
Pid: 7564, comm: pan Tainted: P 2.6.24-desktop-0.rc8.2.1mdv #1
RIP: 0010:[<ffffffff802d5a57>] [<ffffffff802d5a57>] block_write_begin +0x87/0xe0 RSP: 0018:ffff81002e9b5ac8 EFLAGS: 00000286 RAX: ffff81003f5b3248 RBX: 00000000fffffff4 RCX: 0000000000000000 RDX: ffff81003f5b3248 RSI: 0000000000000000 RDI: ffff81002d1eda18 RBP: ffff81003f5b3248 R08: 0000000000000000 R09: ffff81002e9b5be0 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 R13: ffffffff880c28b0 R14: 0000000000001000 R15: 000000003f5b3248 FS: 00002b6bb3bf7960(0000) GS:ffffffff80589000(0000) knlGS:00000000f78568d0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b0537cf6000 CR3: 00000000391d2000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace: [<ffffffff802d5a4f>] block_write_begin+0x7f/0xe0 [<ffffffff880c2102>] :xfs:xfs_vm_write_begin+0x22/0x30 [<ffffffff880c28b0>] :xfs:xfs_get_blocks+0x0/0x10 [<ffffffff80280ba9>] generic_file_buffered_write+0x149/0x6c0 [<ffffffff80240a2e>] current_fs_time+0x1e/0x30 [<ffffffff880caae6>] :xfs:xfs_write+0x676/0x910 [<ffffffff8027f98d>] find_lock_page+0x3d/0xc0 [<ffffffff880c69d0>] :xfs:xfs_file_aio_write+0x0/0x50 [<ffffffff802aee9b>] do_sync_readv_writev+0xcb/0x110 [<ffffffff8028eb95>] __do_fault+0x1f5/0x420 [<ffffffff802522b0>] autoremove_wake_function+0x0/0x30 [<ffffffff80290b90>] handle_mm_fault+0x540/0x800 [<ffffffff802aecdd>] rw_copy_check_uvector+0x9d/0x150 [<ffffffff802af5fd>] do_readv_writev+0xfd/0x230 [<ffffffff802afc33>] sys_writev+0x53/0x90 [<ffffffff8020c36e>] system_call+0x7e/0x83
Signature · 1 line
Frederik Himpe <[email protected]>
Quoted 12 lines · Frederik Himpe
Quoted 11 lines · Nick Piggin
Quoted 10 lines
Quoted 10 lines · Frederik Himpe
Quoted 9 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task0
Quoted 7 lines
Quoted 4 lines
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
Any luck getting traces?
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
Possible. Though I think it is more likely to be a bug in the new deadlock avoidance code in the generic buffered write path. Dang... I wonder why this hasn't come up earlier. It looks like pan's use of writev might be tickling it.
How quickly can you reproduce this?
Can you use strace to see what the hanging syscall looks like?
Thanks,
Nick
Quoted 82 lines
Jan 26 14:17:43 Anastacia kernel: pan R running task 07564 1 Jan 26 14:17:43 Anastacia kernel:
000000003f5b3248
0000000000001000ffffffff880c28b00000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff81003f5b3248ffff81002d1ed900000000002d1ed9000000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff810016050dd0fffff000fffff0000000000000000000ffff81002d1eda10Jan 26 14:17:43 Anastacia kernel: Call Trace:Jan 26 14:17:43 Anastacia kernel: [_end+127964408/2129947720]:xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel:
[unix_poll+0/176]unix_poll+0x0/0xb0Jan 26 14:17:43 Anastacia kernel:
[_end+127964408/2129947720] :xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [iov_iter_copy_from_user_atomic+65/160]iov_iter_copy_from_user_atomic+0x41/0xa0Jan 26 14:17:43 Anastacia kernel:
[iov_iter_copy_from_user_atomic+46/160]iov_iter_copy_from_user_atomic+0x2e/0xa0Jan 26 14:17:43 Anastacia kernel:
[generic_file_buffered_write+383/1728]generic_file_buffered_write+0x17f/0x6c0 Jan 26 14:17:43 Anastacia kernel: [current_fs_time+30/48] current_fs_time+0x1e/0x30 Jan 26 14:17:43 Anastacia kernel: [_end+127997742/2129947720] :xfs:xfs_write+0x676/0x910 Jan 26 14:17:43 Anastacia kernel: [find_lock_page+61/192] find_lock_page+0x3d/0xc0 Jan 26 14:17:43 Anastacia kernel: [_end+127981080/2129947720] :xfs:xfs_file_aio_write+0x0/0x50 Jan 26 14:17:43 Anastacia kernel: [do_sync_readv_writev+203/272] do_sync_readv_writev+0xcb/0x110 Jan 26 14:17:43 Anastacia kernel: [__do_fault+501/1056] __do_fault+0x1f5/0x420 Jan 26 14:17:43 Anastacia kernel: [autoremove_wake_function+0/48] autoremove_wake_function+0x0/0x30 Jan 26 14:17:43 Anastacia kernel: [handle_mm_fault+1344/2048] handle_mm_fault+0x540/0x800 Jan 26 14:17:43 Anastacia kernel: [rw_copy_check_uvector+157/336] rw_copy_check_uvector+0x9d/0x150 Jan 26 14:17:43 Anastacia kernel: [do_readv_writev+253/560] do_readv_writev+0xfd/0x230 Jan 26 14:17:43 Anastacia kernel: [sys_writev+83/144] sys_writev+0x53/0x90 Jan 26 14:17:43 Anastacia kernel: [system_call+126/131] system_call+0x7e/0x83 Jan 26 14:17:43 Anastaciakernel:
SysRq : Show Regs
CPU 0:Modules linked in: usb_storage af_packet nvidia(P) vboxdrv ipv6 fuse
snd_seq_dummysnd_seq_osssnd_seq_midi_eventsnd_seqcpufreq_ondemandvideo outputtc1100_wmisbs sbshc container dock battery acbinfmt_miscloop ext3 jbddm_mirrorsr_moddm_modpata_amdata_genericsata_silusbmouse usbhidff_memlessfloppy usblppowernow_k8freq_tablesnd_pcm_osssnd_mixer_osssnd_mpu401snd_mpu401_uartsnd_rawmidins558 gameportparport_pcsnd_seq_deviceparportrtc_cmospcspkrsnd_intel8x0k8tempsnd_ac97_codecohci1394ac97_busieee1394snd_pcmsnd_timerskgeohci_hcdehci_hcdsnd soundcore usbcore forcedethsnd_page_allocssb fan pcmciapcmcia_corei2c_nforce2i2c_corebutton thermal processor sg evdev genrtc xfsscsi_wait_scansd_modsata_nvlibatascsi_mod
Pid: 7564, comm: pan Tainted: P 2.6.24-desktop-0.rc8.2.1mdv #1RIP: 0010:[<ffffffff802d5a57>] [<ffffffff802d5a57>] block_write_begin +0x87/0xe0 RSP: 0018:ffff81002e9b5ac8 EFLAGS: 00000286 RAX: ffff81003f5b3248 RBX: 00000000fffffff4 RCX: 0000000000000000 RDX: ffff81003f5b3248 RSI: 0000000000000000 RDI: ffff81002d1eda18 RBP: ffff81003f5b3248 R08: 0000000000000000 R09: ffff81002e9b5be0 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 R13: ffffffff880c28b0 R14: 0000000000001000 R15: 000000003f5b3248 FS: 00002b6bb3bf7960(0000) GS:ffffffff80589000(0000) knlGS:00000000f78568d0 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b0537cf6000 CR3: 00000000391d2000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400Call Trace: [<ffffffff802d5a4f>] block_write_begin+0x7f/0xe0 [<ffffffff880c2102>] :xfs:xfs_vm_write_begin+0x22/0x30 [<ffffffff880c28b0>] :xfs:xfs_get_blocks+0x0/0x10 [<ffffffff80280ba9>] generic_file_buffered_write+0x149/0x6c0 [<ffffffff80240a2e>] current_fs_time+0x1e/0x30 [<ffffffff880caae6>] :xfs:xfs_write+0x676/0x910 [<ffffffff8027f98d>] find_lock_page+0x3d/0xc0 [<ffffffff880c69d0>] :xfs:xfs_file_aio_write+0x0/0x50 [<ffffffff802aee9b>] do_sync_readv_writev+0xcb/0x110 [<ffffffff8028eb95>] __do_fault+0x1f5/0x420 [<ffffffff802522b0>] autoremove_wake_function+0x0/0x30 [<ffffffff80290b90>] handle_mm_fault+0x540/0x800 [<ffffffff802aecdd>] rw_copy_check_uvector+0x9d/0x150 [<ffffffff802af5fd>] do_readv_writev+0xfd/0x230 [<ffffffff802afc33>] sys_writev+0x53/0x90 [<ffffffff8020c36e>] system_call+0x7e/0x83
Quoted 3 lines · Nick Piggin <nickpiggin <at> yahoo.com.au> writes:
Quoted 2 lines · Frederik Himpe
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
I got the same issue using ext3
Quoted 6 lines
Possible. Though I think it is more likely to be a bug in the new deadlock avoidance code in the generic buffered write path. Dang... I wonder why this hasn't come up earlier. It looks like pan's use of writev might be tickling it.
How quickly can you reproduce this?
When I was using pan daily one month ago, I got it twice over a week
Quoted 1 line
Can you use strace to see what the hanging syscall looks like?
I tried last week during 5 hours without luck, I can try again
Quoted 12 lines · Pascal Terjan
Quoted 3 lines · Nick Piggin <nickpiggin <at> yahoo.com.au> writes:
Quoted 2 lines · Frederik Himpe
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
I got the same issue using ext3
Quoted 5 lines
Possible. Though I think it is more likely to be a bug in the new deadlock avoidance code in the generic buffered write path. Dang... I wonder why this hasn't come up earlier. It looks like pan's use of writev might be tickling it.
Quoted 7 lines
Quoted 1 line
How quickly can you reproduce this?
When I was using pan daily one month ago, I got it twice over a week
Quoted 1 line
Can you use strace to see what the hanging syscall looks like?
I tried last week during 5 hours without luck, I can try again
Dang, I didn't see any reports of this earlier :(
Anyway, I sent a patch to fix it in the original thread (can you reply-to-all please? just it a bit easier to keep threads together)
Thanks,
Nick
Quoted 28 lines · Frederik Himpe
Quoted 7 lines · Nick Piggin
Quoted 7 lines
Quoted 7 lines · Frederik Himpe
Quoted 6 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task0
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
Any luck getting traces?
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
Jan 26 14:17:43 Anastacia kernel: pan R running task 07564 1 Jan 26 14:17:43 Anastacia kernel:
000000003f5b3248
0000000000001000ffffffff880c28b00000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff81003f5b3248ffff81002d1ed900000000002d1ed9000000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff810016050dd0fffff000fffff0000000000000000000ffff81002d1eda10Jan 26 14:17:43 Anastacia kernel: Call Trace:Jan 26 14:17:43 Anastacia kernel: [_end+127964408/2129947720]
Quoted 8 lines
:xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel:
[unix_poll+0/176]unix_poll+0x0/0xb0Jan 26 14:17:43 Anastacia kernel:
[_end+127964408/2129947720] :xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [iov_iter_copy_from_user_atomic+65/160]iov_iter_copy_from_user_atomic+0x41/0xa0Jan 26 14:17:43 Anastacia kernel:
[iov_iter_copy_from_user_atomic+46/160]iov_iter_copy_from_user_atomic+0x2e/0xa0Jan 26 14:17:43 Anastacia kernel:
[generic_file_buffered_write+383/1728]
Well after trying a lot of writev combinations, I've reproduced a hang *hangs head*.
Does this help?
Zero length iovecs can go into an infinite loop in writev, because the iovec iterator does not always advance over them.
The sequence required to trigger this is not trivial. I think it requires that a zero-length iovec be followed by a non-zero-length iovec which causes a pagefault in the atomic usercopy. This causes the writev code to drop back into single-segment copy mode, which then tries to copy the 0 bytes of the zero-length iovec; a zero length copy looks like a failure though, so it loops.
Put a test into iov_iter_advance to catch zero-length iovecs. We could just put the test in the fallback path, but I feel it is more robust to skip over zero-length iovecs throughout the code (iovec iterator may be used in filesystems too, so it should be robust).
---
mm/filemap.c +6 -2
@@ -1733,7 +1733,11 @@ static void __iov_iter_advance_iov(strucconst struct iovec *iov = i->iov;size_t base = i->iov_offset; -while (bytes) { +/* +* The !iov->iov_len check ensures we skip over unlikely +* zero-length segments. +*/ +while (bytes || !iov->iov_len) {int copy =min (bytes, iov->iov_len - base); bytes -= copy;
@@ -2251,6 +2255,7 @@ again:cond_resched (); +iov_iter_advance (i, copied);if (unlikely (copied ==0 )) {/* * If we were unable to copy any data at all, we must
@@ -2264,7 +2269,6 @@ again: iov_iter_single_seg_count(i)); goto again; } -iov_iter_advance (i, copied); pos += copied; written += copied;
Quoted 38 lines · Nick Piggin
Quoted 28 lines · Frederik Himpe
Quoted 7 lines · Nick Piggin
Quoted 7 lines
Quoted 7 lines · Frederik Himpe
Quoted 6 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task0
Nasty. The attached patch is something really simple that can sometimes help. sysrq+p is also an option, if you're on a UP system.
Any luck getting traces?
I just succeeded to reproduce the problem with this patch. Does this smell like an XFS problem?
Jan 26 14:17:43 Anastacia kernel: pan R running task 07564 1 Jan 26 14:17:43 Anastacia kernel:
000000003f5b3248
0000000000001000ffffffff880c28b00000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff81003f5b3248ffff81002d1ed900000000002d1ed9000000000000000000 Jan 26 14:17:43 Anastacia kernel:ffff810016050dd0fffff000fffff0000000000000000000ffff81002d1eda10Jan 26 14:17:43 Anastacia kernel: Call Trace:Jan 26 14:17:43 Anastacia kernel: [_end+127964408/2129947720]Quoted 8 lines
:xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel:
[unix_poll+0/176]unix_poll+0x0/0xb0Jan 26 14:17:43 Anastacia kernel:
[_end+127964408/2129947720] :xfs:xfs_get_blocks+0x0/0x10 Jan 26 14:17:43 Anastacia kernel: [iov_iter_copy_from_user_atomic+65/160]iov_iter_copy_from_user_atomic+0x41/0xa0Jan 26 14:17:43 Anastacia kernel:
[iov_iter_copy_from_user_atomic+46/160]iov_iter_copy_from_user_atomic+0x2e/0xa0Jan 26 14:17:43 Anastacia kernel:
[generic_file_buffered_write+383/1728]
Quoted 4 lines
Well after trying a lot of writev combinations, I've reproduced a hang *hangs head*.
Does this help?
I'm currently running with this patch. The problem happens about two times a week, so it will take a few days to come to a conclusion whether it is fixed. I'll let you all know.
Thanks for the patch!
Signature · 1 line
Frederik Himpe <[email protected]>
Quoted 9 lines · Nick Piggin
Quoted 7 lines · Frederik Himpe
Quoted 6 lines · Nick Piggin
Quoted 6 lines
Quoted 6 lines · Frederik Himpe
Quoted 5 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 4 lines
Well after trying a lot of writev combinations, I've reproduced a hang *hangs head*.
Does this help?
Just to confirm: in four days of testing, I haven't seen the problem anymore, so it looks like this was indeed the right fix.
Thanks!
Signature · 1 line
Frederik Himpe <[email protected]>
Quoted 10 lines · Frederik Himpe
Quoted 9 lines · Nick Piggin
Quoted 7 lines · Frederik Himpe
Quoted 6 lines · Nick Piggin
Quoted 6 lines
Quoted 6 lines · Frederik Himpe
Quoted 5 lines
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 7 lines
Quoted 4 lines
Well after trying a lot of writev combinations, I've reproduced a hang *hangs head*.
Does this help?
Just to confirm: in four days of testing, I haven't seen the problem anymore, so it looks like this was indeed the right fix.
Thanks very much for reporting and testing. This patch needs to go into 2.6.24.stable and upstream.
Quoted 7 lines · Nick Piggin
Quoted 6 lines · Frederik Himpe
With Linux 2.6.24-rc8 I often have the problem that the pan usenet reader starts using 100% of CPU time after some time. When this happens, kill -9 does not work, and strace just hangs when trying to attach to the process. The same with gdb. ps shows the process as being in the R state.
Quoted 6 lines
Quoted 2 lines
I pressed Ctrl-Alt-SysRq-T, and this was shown for pan:
Jan 21 21:45:01 Anastacia kernel: pan R running task 0Well I've twice tried to submit a patch to print stacks for running tasks as well, but nobody seems interested. It would at least give a chance to see something.
i definitely remembering having done this myself a couple of times (it makes tons of sense to get _some_ info out of the system) but some problem in -mm kept reverting it. I dont remember the specifics ... it was some race.
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 3 lines · Nick Piggin
Well I've twice tried to submit a patch to print stacks for running tasks as well, but nobody seems interested. It would at least give a chance to see something.
i definitely remembering having done this myself a couple of times (it makes tons of sense to get _some_ info out of the system) but some problem in -mm kept reverting it. I dont remember the specifics ... it was some race.
Hmm, that's not unlikely. But there is nothing in the backtrace code which prevents a task from being woken up anyway, is there? I guess it will be more common now, but if we find a race we can try to fix the root cause.