XFS corruption [2.6.30.5+patches.2.6.30.tgz]
Hello!
In a backup server attached to a Coraid shelf via AOE and DM, we saw 2.6.30.5+patches.2.6.30.tgz die with a corruption problem. The kernel logged many similar errors, and then stopped responding some time later.
This particular storage is written to entirely by "cp" and "rsync", and is responsible for storing backups over NFS. The file system was created only a few days ago, and the system had not rebooted since mkfs.xfs. This is one of six volumes at 3 TB each.
This file system stores a hardlink-based backup (eg: where each day's backups are hardlinked except where files change), and this crash occurred during a run of "cp -al" between backup trees:
Sep 3 00:05:22 backup01 kernel: ffff880125072000: 54 03 62 d0 83 cf 00 00 00 6c 16 47 00 6c 16 47 T.b......l.G.l.G Sep 3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff803fec85 Sep 3 00:05:22 backup01 kernel: Sep 3 00:05:22 backup01 kernel: Pid: 2449, comm: cp Not tainted 2.6.30.5-hw-fixedxfs #1 Sep 3 00:05:22 backup01 kernel: Call Trace: Sep 3 00:05:22 backup01 kernel: [<ffffffff8040a83e>] xfs_error_report+0x3e/0x40 Sep 3 00:05:22 backup01 kernel: [<ffffffff803fec85>] ? xfs_da_read_buf+0x25/0x30 Sep 3 00:05:22 backup01 kernel: [<ffffffff8040a898>] xfs_corruption_error+0x58/0x70 Sep 3 00:05:22 backup01 kernel: [<ffffffff803febbd>] xfs_da_do_buf+0x65d/0x6b0 Sep 3 00:05:22 backup01 kernel: [<ffffffff803fec85>] ? xfs_da_read_buf+0x25/0x30 Sep 3 00:05:22 backup01 kernel: [<ffffffff80705c47>] ? __down_read+0x17/0xc7 Sep 3 00:05:22 backup01 kernel: [<ffffffff8028a4ea>] ? get_page_from_freelist+0x30a/0x480 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0 Sep 3 00:05:22 backup01 kernel: [<ffffffff803fec85>] xfs_da_read_buf+0x25/0x30 Sep 3 00:05:22 backup01 kernel: [<ffffffff804026ea>] ? xfs_dir2_block_getdents+0x7a/0x1e0 Sep 3 00:05:22 backup01 kernel: [<ffffffff804026ea>] xfs_dir2_block_getdents+0x7a/0x1e0 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0 Sep 3 00:05:22 backup01 kernel: [<ffffffff804013f1>] xfs_readdir+0xd1/0xe0 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0 Sep 3 00:05:22 backup01 kernel: [<ffffffff80431d6a>] xfs_file_readdir+0x3a/0x50 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c1eb1>] vfs_readdir+0xa1/0xc0 Sep 3 00:05:22 backup01 kernel: [<ffffffff802c2141>] sys_getdents+0x81/0xd0 Sep 3 00:05:22 backup01 kernel: [<ffffffff80706365>] ? page_fault+0x25/0x30 Sep 3 00:05:22 backup01 kernel: [<ffffffff8020be02>] system_call_fastpath+0x16/0x1b Sep 3 00:05:22 backup01 kernel: ffff880228059000: f7 b6 37 2e cf dc e2 ea 00 00 00 00 00 00 00 00 ..7............. Sep 3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff803fec85 (same backtrace) Sep 3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff803fec85
(same backtrace)
Full kern.log including result from attempt to mount after reboot:
http://0x.ca/sim/ref/2.6.30.5-hw-fixedxfs/kern_log_0.txt
I've left the file system in this state for debugging purposes. I can run xfs_repair or metadata dumps, etc., on demand.
Cheers,
Simon-
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Hmm, I suspect the corruption here came from the RAID 6 implementation. I see corruption on another volume on this volume group now. We happened to test a two drive RAID 6 failure/rebuild shortly before this occurred.
So, perhaps it's not worth looking at this further other than perhaps the part where the kernel hung. :)
Simon-
Quoted 5 lines · Simon Kirby
Hello!
In a backup server attached to a Coraid shelf via AOE and DM, we saw 2.6.30.5+patches.2.6.30.tgz die with a corruption problem. The kernel logged many similar errors, and then stopped responding some time later.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs