Skip to contentSkip to conversation

XFS corruption [2.6.30.5+patches.2.6.30.tgz]

DISCUSSION Simon Kirby 1 participant 2 messages Sep 7, 2009

lore
Share & tools
lore
#0 lore raw

Hello!

In a backup server attached to a Coraid shelf via AOE and DM, we saw 2.6.30.5+patches.2.6.30.tgz die with a corruption problem. The kernel logged many similar errors, and then stopped responding some time later.

This particular storage is written to entirely by "cp" and "rsync", and is responsible for storing backups over NFS. The file system was created only a few days ago, and the system had not rebooted since mkfs.xfs. This is one of six volumes at 3 TB each.

This file system stores a hardlink-based backup (eg: where each day's backups are hardlinked except where files change), and this crash occurred during a run of "cp -al" between backup trees:

Sep  3 00:05:22 backup01 kernel: ffff880125072000: 54 03 62 d0 83 cf 00 00 00 6c 16 47 00 6c 16 47  T.b......l.G.l.G
Sep  3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff803fec85
Sep  3 00:05:22 backup01 kernel:
Sep  3 00:05:22 backup01 kernel: Pid: 2449, comm: cp Not tainted 2.6.30.5-hw-fixedxfs #1
Sep  3 00:05:22 backup01 kernel: Call Trace:
Sep  3 00:05:22 backup01 kernel: [<ffffffff8040a83e>] xfs_error_report+0x3e/0x40
Sep  3 00:05:22 backup01 kernel: [<ffffffff803fec85>] ? xfs_da_read_buf+0x25/0x30
Sep  3 00:05:22 backup01 kernel: [<ffffffff8040a898>] xfs_corruption_error+0x58/0x70
Sep  3 00:05:22 backup01 kernel: [<ffffffff803febbd>] xfs_da_do_buf+0x65d/0x6b0  
Sep  3 00:05:22 backup01 kernel: [<ffffffff803fec85>] ? xfs_da_read_buf+0x25/0x30   
Sep  3 00:05:22 backup01 kernel: [<ffffffff80705c47>] ? __down_read+0x17/0xc7  
Sep  3 00:05:22 backup01 kernel: [<ffffffff8028a4ea>] ? get_page_from_freelist+0x30a/0x480
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0
Sep  3 00:05:22 backup01 kernel: [<ffffffff803fec85>] xfs_da_read_buf+0x25/0x30
Sep  3 00:05:22 backup01 kernel: [<ffffffff804026ea>] ? xfs_dir2_block_getdents+0x7a/0x1e0
Sep  3 00:05:22 backup01 kernel: [<ffffffff804026ea>] xfs_dir2_block_getdents+0x7a/0x1e0
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0
Sep  3 00:05:22 backup01 kernel: [<ffffffff804013f1>] xfs_readdir+0xd1/0xe0
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c1d30>] ? filldir+0x0/0xe0
Sep  3 00:05:22 backup01 kernel: [<ffffffff80431d6a>] xfs_file_readdir+0x3a/0x50
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c1eb1>] vfs_readdir+0xa1/0xc0
Sep  3 00:05:22 backup01 kernel: [<ffffffff802c2141>] sys_getdents+0x81/0xd0
Sep  3 00:05:22 backup01 kernel: [<ffffffff80706365>] ? page_fault+0x25/0x30
Sep  3 00:05:22 backup01 kernel: [<ffffffff8020be02>] system_call_fastpath+0x16/0x1b
Sep  3 00:05:22 backup01 kernel: ffff880228059000: f7 b6 37 2e cf dc e2 ea 00 00 00 00 00 00 00 00  ..7.............
Sep  3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff803fec85
(same backtrace)
Sep  3 00:05:22 backup01 kernel: Filesystem "dm-59": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff803fec85

(same backtrace)

Full kern.log including result from attempt to mount after reboot:

	http://0x.ca/sim/ref/2.6.30.5-hw-fixedxfs/kern_log_0.txt

I've left the file system in this state for debugging purposes. I can run xfs_repair or metadata dumps, etc., on demand.

Cheers,

Simon-

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

Hmm, I suspect the corruption here came from the RAID 6 implementation. I see corruption on another volume on this volume group now. We happened to test a two drive RAID 6 failure/rebuild shortly before this occurred.

So, perhaps it's not worth looking at this further other than perhaps the part where the kernel hung. :)

Simon-

Quoted 5 lines · Simon Kirby

Hello!

In a backup server attached to a Coraid shelf via AOE and DM, we saw 2.6.30.5+patches.2.6.30.tgz die with a corruption problem. The kernel logged many similar errors, and then stopped responding some time later.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs