XFS blocked task in xlog_cil_force_lsn
Hi,
today i've seen this call trace / block in XFS while running 3.0.27.
INFO: task xfsaild/sdc3:1818 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfsaild/sdc3 D ffff88041f374dc8 0 1818 2 0x00000000ffff88041edd3d30 0000000000000046 ffff8803ecaa8dc0 ffff88041f23fe28 ffff88041f374860 0000000000010800 ffff88041edd3fd8 ffff88041edd2010 ffff88041edd3fd8 0000000000010800 ffff88041f15c860 ffff88041f374860
Call Trace: [<ffffffff8159c94a>] schedule+0x3a/0x60 [<ffffffff812763a9>] xlog_cil_force_lsn+0xe9/0x110 [<ffffffff8104caf0>] ? try_to_wake_up+0x2a0/0x2a0 [<ffffffff81273922>] _xfs_log_force+0x62/0x250 [<ffffffff81273d23>] xfs_log_force+0x13/0x40 [<ffffffff81281ac7>] ? xfs_trans_ail_cursor_next+0x27/0x30 [<ffffffff81282237>] xfsaild+0x447/0x460 [<ffffffff81281df0>] ? xfs_trans_ail_init+0xb0/0xb0 [<ffffffff8106f646>] kthread+0x96/0xa0 [<ffffffff815a0214>] kernel_thread_helper+0x4/0x10 [<ffffffff8106f5b0>] ? kthread_worker_fn+0x130/0x130 [<ffffffff815a0210>] ? gs_change+0xb/0xb
Is there a known fix / patch available?
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 24 lines · Stefan Priebe - Profihost AG
Hi,
today i've seen this call trace / block in XFS while running 3.0.27.
INFO: task xfsaild/sdc3:1818 blocked for more than 120 seconds."echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
xfsaild/sdc3 Dffff88041f374dc80 1818 20x00000000ffff88041edd3d300000000000000046ffff8803ecaa8dc0ffff88041f23fe28ffff88041f3748600000000000010800ffff88041edd3fd8ffff88041edd2010ffff88041edd3fd80000000000010800ffff88041f15c860ffff88041f374860Call Trace: [<ffffffff8159c94a>] schedule+0x3a/0x60 [<ffffffff812763a9>] xlog_cil_force_lsn+0xe9/0x110 [<ffffffff8104caf0>] ? try_to_wake_up+0x2a0/0x2a0 [<ffffffff81273922>] _xfs_log_force+0x62/0x250 [<ffffffff81273d23>] xfs_log_force+0x13/0x40 [<ffffffff81281ac7>] ? xfs_trans_ail_cursor_next+0x27/0x30 [<ffffffff81282237>] xfsaild+0x447/0x460 [<ffffffff81281df0>] ? xfs_trans_ail_init+0xb0/0xb0 [<ffffffff8106f646>] kthread+0x96/0xa0 [<ffffffff815a0214>] kernel_thread_helper+0x4/0x10 [<ffffffff8106f5b0>] ? kthread_worker_fn+0x130/0x130 [<ffffffff815a0210>] ? gs_change+0xb/0xb
Quoted 1 line
Is there a known fix / patch available?
Your first question should be: is this a known problem?
And the answer to that is: I don't think so.
So, is the system actually hung? Were there IO errors around this time? did the system recover and continue to work normally? What's your storage hardware, application, workload, etc?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 17.04.2012 01:56, schrieb Dave Chinner:
Quoted 2 lines
Your first question should be: is this a known problem? And the answer to that is: I don't think so.
Sorry absolutely correct.
Quoted 1 line
So, is the system actually hung?
No. The kernel has ended the stalled request.
Quoted 1 line
Were there IO errors around this time?
No.
Quoted 1 line
did the system recover and continue to work normally?
Yes.
Quoted 1 line
What's your storage hardware, application, workload, etc?
It a Raid 10 of 1TB SATA Drives in front of a 3Ware Controller doing heavy rsync backups.
Greets
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Stefan Priebe - Profihost AG
Am 17.04.2012 01:56, schrieb Dave Chinner:
Quoted 2 lines
Your first question should be: is this a known problem? And the answer to that is: I don't think so.
Sorry absolutely correct.
Quoted 1 line
So, is the system actually hung?
No. The kernel has ended the stalled request.
Quoted 1 line
Were there IO errors around this time?
Quoted 8 lines
No.
Quoted 1 line
did the system recover and continue to work normally?
Yes.
Quoted 1 line
What's your storage hardware, application, workload, etc?
It a Raid 10 of 1TB SATA Drives in front of a 3Ware Controller doing heavy rsync backups.
Hmmm. It sounds like it is just taking a long time for the IO to complete, not that there is any other problem. If the IO is realy, really slow, the a log force is definitely a place it coul dstall for some time.
Does the RAID controller have a BBWC? What IO scheduler are you using? Does mounting with nobarriers reduce the stall time? Can you reduce the IO load so that it doesn't end up in this state?
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Am 17.04.2012 10:32, schrieb Dave Chinner:
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
What's your storage hardware, application, workload, etc?
It a Raid 10 of 1TB SATA Drives in front of a 3Ware Controller doing heavy rsync backups.
Hmmm. It sounds like it is just taking a long time for the IO to complete, not that there is any other problem. If the IO is realy, really slow, the a log force is definitely a place it coul dstall for some time.
OK as i've seen this just once it just might be that the controller / disks were too slow in this case.
Quoted 1 line
Does the RAID controller have a BBWC?
No but doesn't matter ;-) If it crashes i can just repair the FS and restart the backup again.
Quoted 1 line
What IO scheduler are you using?
[root@backup1 ~]# cat /sys/block/sdc/queue/scheduler [noop] deadline cfq
Quoted 1 line
Does mounting with nobarriers reduce the stall time?
It is already mounted with nobarrier:/dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota)
Quoted 1 line
Can you reduce the IO load so that it doesn't end up in this state?
Can try that by just doing less backups per time period.
Stefan
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs