Repeated XFS corruption on RAID-10 on Adaptec 51245
Hey all,
I have problems with an XFS volume. Upon discovering the message "kernel: XFS (sda3): corrupt inode 3714097 (bad size 16437 for local fork, size = 60)."
I ran xfs_repair /dev/sda3 (/dev/sda3 was unmounted). It reported to have fixed some errors.
However, after a while in normal operation, another XFS corruption occurred on /dev/sda3. I noticed that repeatedly calling xfs_repair will always report and fix new errors, even if the volume is not mounted in between, e.g., "rebuilding directory inode XXX" with different (new) values of XXX.
/dev/sda is a 12 TB RAID-10 volume on an Adaptec 51245 controller. All disks are online and none is reported faulty.
Naive, I would assume that running xfs_repair once would fix all errors. My guess is that the underlying RAID volume (Adaptec 51245 RAID 10) is somehow invalid (although I can not find any indicators confirming this). Any suggestions?
I am running CentOS 6.2 (=RHEL 6.2) with kernel 2.6.32-220.17.1.el6.x86_64 (most recent) and all OS updates installed. Controller Firmware is the most recent (18948), driver version is 1.1-5. HDDs are 2x WD2001FASS, 10x WD2002FAEX.
Thanks in advance,
Chris
Signature · 9 lines
Christian J. Dietrich Institute for Internet Security - if(is) Westfälische Hochschule University of Applied Sciences https://www.internet-sicherheit.de _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Fri, 08 Jun 2012 11:45:28 +0200
"Christian J. Dietrich" <[email protected]> écrivait:
Quoted 5 lines
I am running CentOS 6.2 (=RHEL 6.2) with kernel 2.6.32-220.17.1.el6.x86_64 (most recent) and all OS updates installed. Controller Firmware is the most recent (18948), driver version is 1.1-5. HDDs are 2x WD2001FASS, 10x WD2002FAEX.
I currently manage many servers (about 50) with Adaptec 5xx5 raid cards. The only case of data corruption I've met was with WD drives.
Therefore it's most probably related to the WD drives. WD desktop drives are well known for being (voluntarily) crippled for RAID operation. They will almost always create all sort of weird problems when running under high IO load.
It is of utmost importance that you at least take care of setting TLER in the correct mode on the drives. Beware that apparently newer WD drives don't even allow setting TLER properly anymore (I said "crippled").
http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsEmmanuel,
Am 08.06.2012 12:51, schrieb Emmanuel Florac:
Quoted 16 lines
"Christian J. Dietrich" <[email protected]> écrivait:
Quoted 5 lines
I am running CentOS 6.2 (=RHEL 6.2) with kernel 2.6.32-220.17.1.el6.x86_64 (most recent) and all OS updates installed. Controller Firmware is the most recent (18948), driver version is 1.1-5. HDDs are 2x WD2001FASS, 10x WD2002FAEX.
I currently manage many servers (about 50) with Adaptec 5xx5 raid cards. The only case of data corruption I've met was with WD drives.
Therefore it's most probably related to the WD drives. WD desktop drives are well known for being (voluntarily) crippled for RAID operation. They will almost always create all sort of weird problems when running under high IO load.
Quoted 6 lines
It is of utmost importance that you at least take care of setting TLER in the correct mode on the drives. Beware that apparently newer WD drives don't even allow setting TLER properly anymore (I said "crippled").
Indeed, it seems to be related to the WD HDDs, I am in contact with Adaptec support and will dig deeper. A couple of disks have comparably high CommandAborts.
Probably, I will rebuild the RAID volume using disks with proper TLER support and activate TLER.
Thanks for your help,
Chris
Signature · 10 lines
Christian J. Dietrich Institute for Internet Security - if(is) Westfälische Hochschule University of Applied Sciences https://www.internet-sicherheit.de _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Le Fri, 08 Jun 2012 21:36:58 +0200 vous écriviez:
Quoted 3 lines
Indeed, it seems to be related to the WD HDDs, I am in contact with Adaptec support and will dig deeper. A couple of disks have comparably high CommandAborts.
I'm afraid there's not much they can do, but let us know if they come back with any constructive suggestion.
Quoted 2 lines
Probably, I will rebuild the RAID volume using disks with proper TLER support and activate TLER.
I suppose you may turn off the system, connect the drives to some PC and run the TLER tool; after that it should work better without even rebuilding the array.
good luck,
Signature · 11 lines
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <[email protected]>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfsQuoted 11 lines · Emmanuel Florac
Le Fri, 08 Jun 2012 21:36:58 +0200 vous écriviez:
Quoted 3 lines
Indeed, it seems to be related to the WD HDDs, I am in contact with Adaptec support and will dig deeper. A couple of disks have comparably high CommandAborts.
I'm afraid there's not much they can do, but let us know if they come back with any constructive suggestion.
Quoted 2 lines
Probably, I will rebuild the RAID volume using disks with proper TLER support and activate TLER.
There's nothing to activate. If a drive has TLER/ERC/etc (any SAS or enterprise SATA) it's turned on/enabled at the factory.
Quoted 3 lines
I suppose you may turn off the system, connect the drives to some PC and run the TLER tool; after that it should work better without even rebuilding the array.
wdidle will only change the TLER timeout on firmware that allows it, which means older drives. Many/most of WD's newer consumer drives apparently do not allow this.
Worth noting, the price premium of the 1TB RE4 over the 1TB Black is currently ZERO at Newegg-- both drives are $120. No premium for TLER, enterprise features.
The 2TB RE4 is $230, the 2TB Black is $210, a $20 premium for enterprise features.
I didn't compare to the WD Green drives because they are squarely low performance consumer junk designed NOT to be accesses more than TO be accessed. WD's anticipated mode of operation for such drives is to be idle with the heads parked over 99% of the time. This simply is not suitable for RAID use, and they boldly tell us that. Many ignore the warning then shed tears when arrays die...
At Newegg's current prices, likely other vendors as well, there is no meaningful price premium for a WD 7.2k enterprise SATA drive over a WD 7.2K performance oriented consumer drive. Thus, if the preference is for WD gear, no sysadmin has a valid excuse for not buying the RE series drives.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 14 lines · Christian J. Dietrich
Hey all,
I have problems with an XFS volume. Upon discovering the message "kernel: XFS (sda3): corrupt inode 3714097 (bad size 16437 for local fork, size = 60)."
I ranxfs_repair/dev/sda3(/dev/sda3was unmounted). It reported to have fixed some errors.
However, after a while in normal operation, another XFS corruption occurred on/dev/sda3. I noticed that repeatedly callingxfs_repairwill always report and fix new errors, even if the volume is not mounted in between, e.g., "rebuilding directory inode XXX" with different (new) values of XXX.
Quoted 7 lines
/dev/sdais a 12 TB RAID-10 volume on an Adaptec 51245 controller. All disks are online and none is reported faulty.Naive, I would assume that running
xfs_repaironce would fix all errors. My guess is that the underlying RAID volume (Adaptec 51245 RAID 10) is somehow invalid (although I can not find any indicators confirming this). Any suggestions?
If you suspect a problem with repair, you can try:
# umount /dev/sda # xfs_metadump -o /dev/sda - | xfs_mdrestore - filesystem.img # xfs_repair filesystem.img # xfs_repair filesystem.img
The image shouldn't take too much space, but 12T might take a little while to dump.
If repair doesn't fix everything the first time please let us know.
Quoted 2 lines
I am running CentOS 6.2 (=RHEL 6.2) with kernel 2.6.32-220.17.1.el6.x86_64 (most recent) and all OS updates installed.
please make sure that you aren't running the old kmod-xfs (or was it xfs-kmod?) rpm.
-Eric
Quoted 6 lines
Controller Firmware is the most recent (18948), driver version is 1.1-5. HDDs are 2x WD2001FASS, 10x WD2002FAEX.
Thanks in advance,
Chris
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Eric,
thanks for your suggestions.
Am 08.06.2012 15:56, schrieb Eric Sandeen:
Quoted 7 lines
Quoted 4 lines
Naive, I would assume that running
xfs_repaironce would fix all errors. My guess is that the underlying RAID volume (Adaptec 51245 RAID 10) is somehow invalid (although I can not find any indicators confirming this). Any suggestions?If you suspect a problem with repair, you can try:
Quoted 8 lines
# umount /dev/sda # xfs_metadump -o /dev/sda - | xfs_mdrestore - filesystem.img # xfs_repair filesystem.img # xfs_repair filesystem.imgThe image shouldn't take too much space, but 12T might take a little while to dump.
If repair doesn't fix everything the first time please let us know.
Thanks, I can confirm that xfs_repair fixes everything on the first run.
Quoted 2 lines
please make sure that you aren't running the old kmod-xfs (or was it xfs-kmod?) rpm.
No, I am using the default driver that comes with the distribution.
Thanks again, I consider the problem to be related to the combination of controller and HDDs (and no longer to xfs) and will dig deeper into that direction.
Chris
Signature · 10 lines
Christian J. Dietrich Institute for Internet Security - if(is) Westfälische Hochschule University of Applied Sciences https://www.internet-sicherheit.de _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs