3.8-rc5 xfs corruption
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
(Sorry to post to xfs mailing lists but unsure about which one is the best for this.)
I have seen something like this once during testing on a system with a EMC VNX FC/multipath back-end. Kernel config file is here,
http://people.redhat.com/qcai/stable/.config
[ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo [ 3025.071743] XFS (dm-1): Internal error xfs_bmbt_verify at line 747 of file fs/xfs/xfs_bmap_btree.c. Caller 0xffffffffa01c375e [ 3025.071743] [ 3025.084589] Pid: 275, comm: xfsaild/dm-1 Tainted: GF 3.8.0-rc5 #1 [ 3025.091636] Call Trace: [ 3025.094176] [<ffffffffa019910f>] xfs_error_report+0x3f/0x50 [xfs] [ 3025.100415] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.107163] [<ffffffffa019917e>] xfs_corruption_error+0x5e/0x90 [xfs] [ 3025.113744] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.120495] [<ffffffffa01c3606>] xfs_bmbt_verify+0x76/0x1c0 [xfs] [ 3025.126737] [<ffffffffa01c375e>] ? xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.133481] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.139714] [<ffffffffa01c375e>] xfs_bmbt_write_verify+0xe/0x10 [xfs] [ 3025.146287] [<ffffffffa019713e>] _xfs_buf_ioapply+0x5e/0x370 [xfs] [ 3025.152565] [<ffffffff81097c70>] ? try_to_wake_up+0x2d0/0x2d0 [ 3025.158442] [<ffffffffa01975e5>] ? xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.164669] [<ffffffffa019752a>] xfs_buf_iorequest+0x4a/0xa0 [xfs] [ 3025.170979] [<ffffffffa01975e5>] xfs_bdstrat_cb+0x65/0xd0 [xfs] [ 3025.177032] [<ffffffffa01977c1>] __xfs_buf_delwri_submit+0x171/0x1e0 [xfs] [ 3025.184038] [<ffffffffa0198110>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.191491] [<ffffffffa01f1100>] ? xfs_trans_ail_cursor_first+0x80/0xb0 [xfs] [ 3025.198755] [<ffffffffa0198110>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [ 3025.206030] [<ffffffffa01f1352>] xfsaild+0x222/0x5e0 [xfs] [ 3025.211665] [<ffffffffa01f1130>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs] [ 3025.218890] [<ffffffff81085b10>] kthread+0xc0/0xd0 [ 3025.223784] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.230317] [<ffffffff816176ec>] ret_from_fork+0x7c/0xb0 [ 3025.235721] [<ffffffff81085a50>] ? kthread_create_on_node+0x120/0x120 [ 3025.242248] XFS (dm-1): Corruption detected. Unmount and run xfs_repair [ 3025.248865] XFS (dm-1): xfs_do_force_shutdown(0x8) called from line 1340 of file fs/xfs/xfs_buf.c. Return address = 0xffffffffa0197411
Does this ring any bell?
Regards,
CAI Qian
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · CAI Qian
Hello,
(Sorry to post to xfs mailing lists but unsure about which one is the best for this.)
Trimmed to just [email protected].
Quoted 2 lines
I have seen something like this once during testing on a system with a EMC VNX FC/multipath back-end.
This is a trace from the verifier code that was added in 3.8-rc1 so I doubt it has anything to do with any problem you've seen in the past....
Can you tell us what workload you were running and what hardware you are using as per:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
As it is, if you mounted the filesystem after this problem was detected, log recovery probably propagated it to disk. I'd suggest that you run xfs_repair -n on the device and post the output so we can see if any corruption has actaully made it to disk. If no corruption made it to disk, it's possible that we've got the incorrect verifier attached to the buffer.
Quoted 1 line
[ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c 69 62 2f 6d 6f ../../usr/lib/mo
The start of a block contains a path and the only
type of block that can contain this format of metadata is remote symlink block. Remote symlink blocks don't have a verifier attached to them as there is nothing that can currently be used to verify them as correct.
I can't see exactly how this can occur as stale buffers have the verifier ops cleared before being returned to the new user, and newly allocated xfs_bufs are zeroed before being initialised. I really need to know what you are doing to be able to get to the bottom of it....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
----- Original Message -----
Quoted 6 lines
From: "Dave Chinner" <[email protected]>
To: "CAI Qian" <[email protected]>
Cc: [email protected], [email protected], "linux-kernel" <[email protected]>
Sent: Thursday, January 31, 2013 12:07:48 PM
Subject: Re: 3.8-rc5 xfs corruption
Quoted 8 lines
Quoted 5 lines · CAI Qian
Hello,
(Sorry to post to xfs mailing lists but unsure about which one is the
best for this.)Trimmed to just [email protected].
Thanks for quick response, Dave.
Quoted 5 lines
Quoted 4 lines
I have seen something like this once during testing on a system with a
EMC VNX FC/multipath back-end.
Quoted 8 lines
This is a trace from the verifier code that was added in 3.8-rc1 so I doubt it has anything to do with any problem you've seen in the past....
Can you tell us what workload you were running and what hardware you are using as per:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
This was the system,
- AMD Opteron(tm) Processor 4130 (1 socket, 4 cores)
- PowerEdge R415
- 8G memory
- mptsas local disks
Software version,
- xfsprogs-3.1.10
The workload was running some fs_mark, syscalls tests, some nfs/cifs connectathon tests, memory, libhugetlbfs tests, and some dynamic debug (Documentation/dynamic-debug-howto.txt) tests.
Quoted 7 lines
As it is, if you mounted the filesystem after this problem was detected, log recovery probably propagated it to disk. I'd suggest that you run
xfs_repair-n on the device and post the output so we can see if any corruption has actaully made it to disk. If no corruption made it to disk, it's possible that we've got the incorrect verifier attached to the buffer.
The system was taken away from me, so I can only occupy it again later if needed.
Regards,
CAI Qian
Quoted 23 lines
Quoted 3 lines
[ 3025.063024] ffff8801a0d50000: 2e 2e 2f 2e 2e 2f 75 73 72 2f 6c69 62 2f 6d 6f ../../usr/lib/mo
The start of a block contains a path and the only
type of block that can contain this format of metadata is remote symlink block. Remote symlink blocks don't have a verifier attached to them as there is nothing that can currently be used to verify them as correct.I can't see exactly how this can occur as stale buffers have the verifier ops cleared before being returned to the new user, and newly allocated
xfs_bufsare zeroed before being initialised. I really need to know what you are doing to be able to get to the bottom of it....Cheers,
Dave.
--
Dave Chinner
[email protected]