centos and xfs filesystem shutdowns

Desperately seeking advice :)

The setup:

* CentOS 5.2
* kmod-xfs-0.4-2 from centosplus repo
* nfs exporting -> xfs filesystem -> lvm volume -> iscsi target
* each filesystem is about 750GB and we mount 5 on each server.
* each filesystem contains 3-20 million small files.
* mount options are as follows _netdev,noatime,uqnoenforce,gqnoenforce,ihashsize=262139,rw

The crash:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff883ff56b

Call Trace:
 [<ffffffff883fdc7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f
 [<ffffffff883ff56b>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff8840c2ad>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88429e46>] :xfs:xfs_itruncate_finish+0x172/0x2b3
 [<ffffffff8843fe87>] :xfs:xfs_setattr+0x7fe/0xd63
 [<ffffffff8853e5ff>] :nfsd:exp_get_by_name+0x5b/0x71
 [<ffffffff8844a704>] :xfs:xfs_vn_setattr+0x11e/0x141
 [<ffffffff8002c9ac>] notify_change+0x145/0x2e0
 [<ffffffff8853c2a5>] :nfsd:nfsd_setattr+0x34f/0x3fa
 [<ffffffff88542620>] :nfsd:nfsd3_proc_setattr+0x98/0xa4
 [<ffffffff885381db>] :nfsd:nfsd_dispatch+0xd8/0x1d6
 [<ffffffff884c94fb>] :sunrpc:svc_process+0x454/0x71b
 [<ffffffff800645ec>] __down_read+0x12/0x92
 [<ffffffff885385a1>] :nfsd:nfsd+0x0/0x2cb
 [<ffffffff88538746>] :nfsd:nfsd+0x1a5/0x2cb
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff885385a1>] :nfsd:nfsd+0x0/0x2cb
 [<ffffffff885385a1>] :nfsd:nfsd+0x0/0x2cb
 [<ffffffff8005dfa7>] child_rip+0x0/0x11

xfs_force_shutdown(dm-8,0x8) called from line 4267 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_bmap.c. Return address = 0xffffffff8840c2ea Filesystem "dm-8": Corruption of in-memory data detected. Shutting down filesystem: dm-8
Please umount the filesystem, and rectify the problem(s)

Subsequent recovery:

Filesystem "dm-9": Disabling barriers, not supported by the underlying device
XFS mounting filesystem dm-9
Starting XFS recovery on filesystem: dm-9 (logdev: internal)
XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1561 of file /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c. Caller 0xffffffff883fd56b

Call Trace:
 [<ffffffff883fbc7e>] :xfs:xfs_free_ag_extent+0x19f/0x67f
 [<ffffffff883fd56b>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff884320e5>] :xfs:xlog_recover_finish+0x15a/0x244
 [<ffffffff88435b00>] :xfs:xfs_mountfs+0xa24/0xc30
 [<ffffffff8000c31a>] _atomic_dec_and_lock+0x39/0x57
 [<ffffffff8843b748>] :xfs:xfs_mount+0x762/0x83b
 [<ffffffff8844ac79>] :xfs:xfs_fs_fill_super+0x0/0x1e3
 [<ffffffff8844acf7>] :xfs:xfs_fs_fill_super+0x7e/0x1e3
 [<ffffffff80064553>] __down_write_nested+0x12/0x92
 [<ffffffff80122410>] selinux_sb_alloc_security+0x3e/0x82
 [<ffffffff800e29c1>] get_filesystem+0x12/0x3b
 [<ffffffff800da854>] sget+0x365/0x377
 [<ffffffff800da1a0>] set_bdev_super+0x0/0xf
 [<ffffffff800da1af>] test_bdev_super+0x0/0xd
 [<ffffffff800db163>] get_sb_bdev+0x10a/0x164
 [<ffffffff80122e04>] selinux_sb_copy_data+0x1a1/0x1c5
 [<ffffffff800dab00>] vfs_kern_mount+0x93/0x11a
 [<ffffffff800dabc9>] do_kern_mount+0x36/0x4d
 [<ffffffff800e42fb>] do_mount+0x6a7/0x717
 [<ffffffff8002cb60>] mntput_no_expire+0x19/0x89
 [<ffffffff8000e80b>] link_path_walk+0xd3/0xe5
 [<ffffffff8003c397>] do_unlinkat+0xe8/0x141
 [<ffffffff8002371c>] __user_walk_fd+0x41/0x4c
 [<ffffffff800c4edc>] zone_statistics+0x3e/0x6d
 [<ffffffff8000f095>] __alloc_pages+0x65/0x2ce
 [<ffffffff8003c397>] do_unlinkat+0xe8/0x141
 [<ffffffff8004bd19>] sys_mount+0x8a/0xcd
 [<ffffffff8005d116>] system_call+0x7e/0x83

Ending XFS recovery on filesystem: dm-9 (logdev: internal)

The story:

Been getting these corruptions for a while now over the span of 6 different machines and a few months. It's gotten a tad crazy lately though with 2 crashes on 2 different filesystems and machines within the span of 3 days.

In looking up portions of the backtrace I found many recommendations to stress/memtest etc to ensure the hardware is solid, I'll of which we've been doing diligently. In fact we've used so many different machines and sticks of ecc memory at this point I can pretty confidently rule it out.

Since our iscsi storage takes nightly snapshots, I've used these and passed them through xfs_repair, xfs_check thinking there was some kind of issue and they always (in 3 repair/checks after 3 different crashes)
seem to come up perfectly clean. These file systems are relatively new
as well, being created in March 2008.

The crash is always exactly the same across different machines. In fact the first 5 lines look very similar to http://oss.sgi.com/archives/xfs/2007-11/msg00041.html in that it always mentions setattr.

I noticed a newer xfs rpm in http://sandeen.net/rhel5_xfs/ is that worth a shot?

Any suggestions would be very much appreciated.

Quoting Matthew Kent <[email protected]>:

I also encountered lot of crashes and hangups with Centos 5.2 and XFS filesystem.

Solution that worked for me was recompiling latest vanilla kernel and installed it into Centos 5.2 system. Maybe some third party provides latest vanilla kernel for Centos 5.2 in rpm format?

--
Eero