xfs corruptions

Hello,

I'm presently debugging the error handler of the MPT fusion driver and therefore causing errors on the disk (Infortrend scsi hardware raids). When I later on try to delete files and directories having been created before and during the failures, "rm -fr" simply says directory not empty. No message in dmesg about it, but xfs_repair reports errors, see below. Once xfs_repair has done its jobs, removing these directories works fine. But this shouldn't happen, should it? This is with 2.6.26

root@beo-11:~# xfs_repair /dev/inf/box-3a/disc
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
imap claims a free inode 1073741964 is in use, correcting imap and clearing
inode
cleared inode 1073741964
        - agno = 2
imap claims a free inode 2147483788 is in use, correcting imap and clearing
inode
cleared inode 2147483788
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
        - agno = 30
        - agno = 31
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
entry "9B769A18" in shortform directory 1073741962 references free inode
1073741964
junking entry "9B769A18" in directory inode 1073741962
        - agno = 2
entry "E95A1D2D" in shortform directory 2147483786 references free inode
2147483788
junking entry "E95A1D2D" in directory inode 2147483786
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - agno = 16
        - agno = 17
        - agno = 18
        - agno = 19
        - agno = 20
        - agno = 21
        - agno = 22
        - agno = 23
        - agno = 24
        - agno = 25
        - agno = 26
        - agno = 27
        - agno = 28
        - agno = 29
        - agno = 30
        - agno = 31
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...

Thanks,
Bernd

So we have an inode that is marked free in the AGI btree, but apparently still in use according to the shortform directory that referenced it.

There are two possibilities here:

The first possibility is the inode btree buffer containing the record indicating the inode is free/used never got written to disk while the other metadata blocks made it to disk. Seeing as the filesystem didn't hang here, it implies that the buffer was written so that on I/O completion the tail of the log could move forward. That is, the I/O was issued, no error was reported, but the I/O never made it to disk. If there was an error, you should see something like:

Warning: Device <bdevname>, XFS metadata write error, block 0x456 in <bdev>

in the syslog indicating a write error. In this case it woul dbe non-fatal and XFS would try to write it again a little later.

The second possibility is that the write of the inode containing the shortform directory to disk did not actually hit the disk, but that implies unlinks had already taken place well before the 'rm -rf' was executed. Perhaps your workload does that....

However, both cases imply that an I/O was indicated as completing successfully when they did not get written to disk, and that points to a bug in the error handling in the underlying driver.

That being said - it could be a bug in the XFS error handling that is causing this, but XFS tends to be pretty noisy when errors occur. I guess that you need to add more tracing to indicate when errors are induced so we can check if errors are being created against the same buffers that the inconsistent state is being found in. That will help point out if the errors are being reported to XFS or not correctly.

Cheers,

Dave.

Dave Chinner
[email protected]