2.6.24-rc6 reproducible raid5 hang

Sorry if this breaks threaded mail readers, I only just subscribed to the list so don;t have the original post to reply to.

I believe I'm having the same problem.

Regarding XFS on a raid5 md array:

Kernels 2.6.22-14 (Ubuntu Gutsy generic and server builds) *and* 2.6.24-rc8 (pure build from virgin sources) compiled for amd64 arch.

Raid 5 configured across 4 x 500GB SATA disks (Nforce nv_sata driver, Asus M2N-E mobo, Athlon X64, 4GB RAM

MD Chunk size is 1024k. This is allocated to an LVM2 PV, then sliced up. Taking one sample logical volume of 150GB I ran

mkfs.xfs -d su=1024k,sw=3 -L vol_linux /dev/vg00/vol_linux

I then found that putting high write load on that filesystem cause a hang. High load could be a little as a single rsync of a mirror of Ubunty Gutsy (many 10's of GB) from my old server to here. Hang would happen in a few hours typically.

I could generate relatively quick hangs by running xfs_fsr (defragger) in parallel.

Trying the workaround up upping /sys/block/md1/md/stripe_cache_size to 4096 seems (fingers crossed) to have helped. Been running the rsync again, plus xfs_fst + a few dd's of 11 GB to the same filesystem.

I did notice also that the write speed increased dramatically with a bigger stripe_cache_size.

A more detailed analysis of the problem indicated that, after the hang:

I could log in;

One CPU core was stuck in 100% IO wait.
The other core was useable, with care. So I managed to get a SysRQ T and
one place the system appeared blocked was via this path:

[ 2039.466258] xfs_fsr       D 0000000000000000     0  7324   7308
[ 2039.466260]  ffff810119399858 0000000000000082 0000000000000000 0000000000000046
[ 2039.466263]  ffff810110d6c680 ffff8101102ba998 ffff8101102ba770 ffffffff8054e5e0
[ 2039.466265]  ffff8101102ba998 000000010014a1e6 ffffffffffffffff ffff810110ddcb30
[ 2039.466268] Call Trace:
[ 2039.466277]  [<ffffffff8808a26b>] :raid456:get_active_stripe+0x1cb/0x610
[ 2039.466282]  [<ffffffff80234000>] default_wake_function+0x0/0x10
[ 2039.466289]  [<ffffffff88090ff8>] :raid456:make_request+0x1f8/0x610
[ 2039.466293]  [<ffffffff80251c20>] autoremove_wake_function+0x0/0x30
[ 2039.466295]  [<ffffffff80331121>] __up_read+0x21/0xb0
[ 2039.466300]  [<ffffffff8031f336>] generic_make_request+0x1d6/0x3d0
[ 2039.466303]  [<ffffffff80280bad>] vm_normal_page+0x3d/0xc0
[ 2039.466307]  [<ffffffff8031f59f>] submit_bio+0x6f/0xf0
[ 2039.466311]  [<ffffffff802c98cc>] dio_bio_submit+0x5c/0x90
[ 2039.466313]  [<ffffffff802c9943>] dio_send_cur_page+0x43/0xa0
[ 2039.466316]  [<ffffffff802c99ee>] submit_page_section+0x4e/0x150
[ 2039.466319]  [<ffffffff802ca2e2>] __blockdev_direct_IO+0x742/0xb50
[ 2039.466342]  [<ffffffff8832e9a2>] :xfs:xfs_vm_direct_IO+0x182/0x190
[ 2039.466357]  [<ffffffff8832edb0>] :xfs:xfs_get_blocks_direct+0x0/0x20
[ 2039.466370]  [<ffffffff8832e350>] :xfs:xfs_end_io_direct+0x0/0x80
[ 2039.466375]  [<ffffffff80444fb5>] __wait_on_bit_lock+0x65/0x80
[ 2039.466380]  [<ffffffff80272883>] generic_file_direct_IO+0xe3/0x190
[ 2039.466385]  [<ffffffff802729a4>] generic_file_direct_write+0x74/0x150
[ 2039.466402]  [<ffffffff88336db2>] :xfs:xfs_write+0x492/0x8f0
[ 2039.466421]  [<ffffffff883099bc>] :xfs:xfs_iunlock+0x2c/0xb0
[ 2039.466437]  [<ffffffff88336866>] :xfs:xfs_read+0x186/0x240
[ 2039.466443]  [<ffffffff8029e5b9>] do_sync_write+0xd9/0x120
[ 2039.466448]  [<ffffffff80251c20>] autoremove_wake_function+0x0/0x30
[ 2039.466457]  [<ffffffff8029eead>] vfs_write+0xdd/0x190
[ 2039.466461]  [<ffffffff8029f5b3>] sys_write+0x53/0x90
[ 2039.466465]  [<ffffffff8020c29e>] system_call+0x7e/0x83

However, I'm of the opinion that the system should not deadlock, even if tunable parameters are unfavourable. I'm happy with the workaround (indeed the system performs better).

However, it will take me a week's worth of testing before I'm willing to commission this as my new fileserver.

So, if there is anything anyone would like me to try, I'm happy to volunteer as a guinea pig :)

Yes, I can build and patch kernels. But I'm not hot at debugging kernels so if kernel core dumps or whatever are needed, please point me at the right document or hint as to which commands I need to read about.

Cheers

Tim

Tim Southerwood ([email protected]) wrote on 23 January 2008 13:37:

This has been corrected already, install Neil's patches. It worked for several people under high stress, including us.

Hi

I just coerced the patch into 2.6.23.14, reset /sys/block/md1/md/stripe_cache_size to default (256) and rebooted.

I can confirm that after 2 hours of heavy bashing[1] the system has not hung. Looks good - many thanks. But I will run with a stripe_cache_size of 4096 in practise as it improves write speen on my configuration about 2.5 times.

Cheers

Tim

[1] Rsync > 50GB to raid pluf xfs_fsr + dd 11GB of /dev/zero to same filesystem.

Subtitle: Patch to mainline yet?

Hi

I don't see evidence of Neil's patch in 2.6.24, so I applied it by hand on my server.

Was that the correct thing to do, or did this issue get fixed in a different way that I wouldn't have spotted? I had a look at the git logs but it was not obvious - please pardon my ignorance, I'm not familiar enough with the code.

Many thanks,

Tim

Tim Southerwood ([email protected]) wrote on 28 January 2008 17:29:

I applied all 4 pending patches to .24. It's been better than .22 and .23... Unfortunately the bitmap and rai1 patch don't go in .22.16.

Neil, have these been sent up against 24-stable and 23-stable?

Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 

Am Dienstag, 29. Januar 2008 23:58 schrieb Bill Davidsen:

.. and .22-stable ?

Also, is this a xfs-on-raid5 bug or would it also happen with ext3-on-raid5 ?

regards
Burkhard