XFS: Internal error XFS_WANT_CORRUPTED_RETURN
Powered up my desktop this morning and noticed I couldn't cd into ~/Mail dmesg didn't look good. "XFS: Internal error XFS_WANT_CORRUPTED_RETURN"
http://codemonkey.org.uk/junk/xfs-1.txt
I rebooted into single user mode, and ran xfs_repair on /dev/sda3 (/home). It fixed up a bunch of stuff, but ended up eating ~/.procmailrc entirely (no sign of it in lost & found), and a bunch of filenames got garbled 'december' became 'decemcer' for eg. Looks like a couple kernel trees ended up in lost & found.
After rebooting back into multi-user mode, I looked in dmesg again to be sure and this time sda2 was complaining..
http://codemonkey.org.uk/junk/xfs-2.txt
Same drill, reboot, xfs_repair. Looks like a bunch of man pages ended up in lost & found.
Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
Dave
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Dave Jones
Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
It may reveal nothing useful, but please report the results from 'smartctl -x /dev/sda' and if not found install smartmontools package.
Chris Murphy
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Chris Murphy
Quoted 2 lines · Dave Jones
Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
It may reveal nothing useful, but please report the results from 'smartctl -x
/dev/sda' and if not found install smartmontools package.
I meant it when I said 'old' and 'crappy'.
It doesn't even support the interesting SMART commands.
Dave
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.11.10-300.fc20.x86_64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: G.SKILL 64GB SSD Serial Number: MK08085207D640017 Firmware Version: 02.10104 User Capacity: 64,105,742,336 bytes [64.1 GB] Sector Size: 512 bytes logical/physical Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS, ATA/ATAPI-7 T13/1532D revision 4a Local Time is: Wed Dec 11 13:56:50 2013 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Unavailable Rd look-ahead is: Disabled Write cache is: Disabled ATA Security is: Unavailable Wt Cache Reorder: Unavailable
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x00) Offline data collection not supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode. Supports SMART auto save timer.
Error logging capability: (0x00) Error logging NOT supported.
No General Purpose Logging support.
SMART Attributes Data Structure revision number: 1280 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
12 Power_Cycle_Count -O--CK 100 100 000 - 5447
9 Power_On_Hours -O--CK 100 100 000 - 0
194 Temperature_Celsius POS--- 032 100 000 - 0
229 Unknown_Attribute -O---- 100 000 000 - 260003199309804
232 Available_Reservd_Space -O---- 100 048 000 - 9028846498104
233 Media_Wearout_Indicator -O---- 100 000 000 - 1122231520092
234 Unknown_Attribute -O---- 100 000 000 - 782120273690
235 Unknown_Attribute -O---- 100 000 000 - 1006826557
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warningRead SMART Log Directory failed: scsi error aborted command
ATA_READ_LOG_EXT (addr=0x00:0x00, page=0, n=1) failed: scsi error aborted command Read GP Log Directory failed
SMART Extended Comprehensive Error Log (GP Log 0x03) not supported
SMART Error Log not supported
SMART Extended Self-test Log (GP Log 0x07) not supported
SMART Self-test Log not supported
Selective Self-tests/Logging not supported
SCT Commands not supported
Device Statistics (GP Log 0x04) not supported
ATA_READ_LOG_EXT (addr=0x11:0x00, page=0, n=1) failed: scsi error aborted command Read SATA Phy Event Counters failed
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Dave Jones
Powered up my desktop this morning and noticed I couldn't cd into ~/Mail dmesg didn't look good. "XFS: Internal error
XFS_WANT_CORRUPTED_RETURN"
http://codemonkey.org.uk/junk/xfs-1.txt
They came from xfs_dir3_block_verify() on read IO completion, which indicates that the corruption was on disk and in the directory structure. Yeah, definitely a verifier error:
XFS (sda3): metadata I/O error: block 0x2e790 ("xfs_trans_read_buf_map") error 117 numblks 8
Are you running a CRC enabled filesystem? (i.e. mkfs.xfs -m crc=1)
Is there any evidence that this verifier has fired in the past on write? If not, then it's a good chance that it's a media error causing this, because the same verifier runs when the metadata is written to ensure we are not writing bas stuff to disk.
Quoted 5 lines
I rebooted into single user mode, and ran
xfs_repairon/dev/sda3(/home). It fixed up a bunch of stuff, but ended up eating ~/.procmailrc entirely (no sign of it in lost & found), and a bunch of filenames got garbled 'december' became 'decemcer' for eg. Looks like a couple kernel trees ended up in lost & found.
Single bit errors in directory names? That really does point towards media errors, not a filesystem error being the cause.
Quoted 4 lines
After rebooting back into multi-user mode, I looked in dmesg again to be sure and this time sda2 was complaining..
Exaclty the same - directory blocks failing read verification.
Quoted 3 lines
Same drill, reboot,
xfs_repair. Looks like a bunch of man pages ended up in lost & found.Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
I'd seriously be considering replacing the SSD as the first step. If you then see failures on a known good drive, we'll need to dig further.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Dave Jones
Quoted 6 lines · Chris Murphy
Quoted 2 lines · Dave Jones
Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
It may reveal nothing useful, but please report the results from 'smartctl -x
/dev/sda' and if not found install smartmontools package.I meant it when I said 'old' and 'crappy'.
It doesn't even support the interesting SMART commands.
Oh well, was worth a shot. The Available_Reservd_Space and Media_Wearout_Indicator could be useful, but I don't know how trustworthy they are when both say they're at 100 which is normally where these values start. Yet they have high, and without reference, meaningless, raw values. The Available_Reservd_Space value is currently 100 but its worst value was 48 which is sorta interesting that it dipped down at one point. That seems to imply it gave up some reserved sectors. I'd expect that once replaced that'd be it, and this value should only go down.
I suspect we've only just begun to see the myriad ways in which SSDs could fail. I ran across this article earlier today:
http://techreport.com/review/25681/the-ssd-endurance-experiment-testing-data-retention-at-300tb
What I thought was eye opening was a hashed file failing multiple times in a row with *different* hash values, being allowed to rest unpowered for five days and then passing. Eeek. Talk about a great setup for a lot of weird transient problems with that kind of reversal. What I can't tell is if there were read errors report to the SATA driver, or if (different) bad data from a particular page was sent to the driver.
Chris Murphy
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 9 lines · Dave Chinner
Quoted 3 lines · Dave Jones
Powered up my desktop this morning and noticed I couldn't cd into ~/Mail dmesg didn't look good. "XFS: Internal error
XFS_WANT_CORRUPTED_RETURN"
http://codemonkey.org.uk/junk/xfs-1.txtThey came from
xfs_dir3_block_verify()on read IO completion, which indicates that the corruption was on disk and in the directory structure. Yeah, definitely a verifier error:
Quoted 8 lines
XFS (sda3): metadata I/O error: block
0x2e790("xfs_trans_read_buf_map") error 117 numblks 8Are you running a CRC enabled filesystem? (i.e. mkfs.xfs -m crc=1)
Is there any evidence that this verifier has fired in the past on write? If not, then it's a good chance that it's a media error causing this, because the same verifier runs when the metadata is written to ensure we are not writing bas stuff to disk.
Dave C, have you given any thought to how to make the verifier errors more actionable? If davej throws up his hands, the rest of the world is obviously in trouble. ;)
To the inexperienced this looks like a "crash" thanks to the backtrace. I do understand that it's necessary for bug reports, but I wonder if we could preface it with something informative or instructive.
We also don't get a block number or inode number, although you or I can dig the inode number out of the hexdump, in this case.
We also don't get any details of what the values in the failed check were; not from the check macro itself or from the hexdump, necessarily, since it only prints the first handful of bytes.
Any ideas here?
-Eric
Quoted 20 lines
Quoted 5 lines
I rebooted into single user mode, and ran
xfs_repairon/dev/sda3(/home). It fixed up a bunch of stuff, but ended up eating ~/.procmailrc entirely (no sign of it in lost & found), and a bunch of filenames got garbled 'december' became 'decemcer' for eg. Looks like a couple kernel trees ended up in lost & found.Single bit errors in directory names? That really does point towards media errors, not a filesystem error being the cause.
Quoted 4 lines
After rebooting back into multi-user mode, I looked in dmesg again to be sure and this time sda2 was complaining..
Exaclty the same - directory blocks failing read verification.
Quoted 3 lines
Same drill, reboot,
xfs_repair. Looks like a bunch of man pages ended up in lost & found.Thoughts ? Could sda be dying ? (It is a fairly old crappy ssd)
Quoted 8 lines
I'd seriously be considering replacing the SSD as the first step. If you then see failures on a known good drive, we'll need to dig further.
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 27 lines · Eric Sandeen
Quoted 9 lines · Dave Chinner
Quoted 3 lines · Dave Jones
Powered up my desktop this morning and noticed I couldn't cd into ~/Mail dmesg didn't look good. "XFS: Internal error
XFS_WANT_CORRUPTED_RETURN"
http://codemonkey.org.uk/junk/xfs-1.txtThey came from
xfs_dir3_block_verify()on read IO completion, which indicates that the corruption was on disk and in the directory structure. Yeah, definitely a verifier error:Quoted 8 lines
XFS (sda3): metadata I/O error: block
0x2e790("xfs_trans_read_buf_map") error 117 numblks 8Are you running a CRC enabled filesystem? (i.e. mkfs.xfs -m crc=1)
Is there any evidence that this verifier has fired in the past on write? If not, then it's a good chance that it's a media error causing this, because the same verifier runs when the metadata is written to ensure we are not writing bas stuff to disk.
Dave C, have you given any thought to how to make the verifier errors more actionable? If davej throws up his hands, the rest of the world is obviously in trouble. ;)
To the inexperienced this looks like a "crash" thanks to the backtrace. I do understand that it's necessary for bug reports, but I wonder if we could preface it with something informative or instructive.
Quoted 6 lines
We also don't get a block number or inode number, although you or I can dig the inode number out of the hexdump, in this case.
We also don't get any details of what the values in the failed check were; not from the check macro itself or from the hexdump, necessarily, since it only prints the first handful of bytes.
This morning, same ssd spewed a bunch of other errors when find ran over a kernel tree..
http://paste.fedoraproject.org/61189/38686344
in that case I did get a block number (the irony of the failing block # is not lost on me)
As soon as the new one arrives, I'll try some destructive tests on the failing one.
I'm just happy it's stayed alive long enough for me to get the data off it. When my Intel SSD failed earlier this year it was just a brick.
Dave
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 14 lines · Eric Sandeen
Quoted 13 lines · Dave Chinner
Quoted 3 lines · Dave Jones
Powered up my desktop this morning and noticed I couldn't cd into ~/Mail dmesg didn't look good. "XFS: Internal error
XFS_WANT_CORRUPTED_RETURN"
http://codemonkey.org.uk/junk/xfs-1.txtThey came from
xfs_dir3_block_verify()on read IO completion, which indicates that the corruption was on disk and in the directory structure. Yeah, definitely a verifier error:XFS (sda3): metadata I/O error: block
0x2e790("xfs_trans_read_buf_map") error 117 numblks 8Are you running a CRC enabled filesystem? (i.e. mkfs.xfs -m crc=1)
Quoted 8 lines
Quoted 4 lines
Is there any evidence that this verifier has fired in the past on write? If not, then it's a good chance that it's a media error causing this, because the same verifier runs when the metadata is written to ensure we are not writing bas stuff to disk.
Dave C, have you given any thought to how to make the verifier errors more actionable? If davej throws up his hands, the rest of the world is obviously in trouble. ;)
The verifier behaviour is effectively boiler plate code.
Quoted 3 lines
To the inexperienced this looks like a "crash" thanks to the backtrace. I do understand that it's necessary for bug reports, but I wonder if we could preface it with something informative or instructive.
Yup, It was done like that so it would scare people and they'd report verifier failures so that we had good visibility of problems they were detecting. So, from that perspective they are doing exactly what they were intended to do.
In reality, the incidence of verifiers detecting corruption is no different from the long term historical trends of corruptions being reported. The only difference is that we are catching them immediately as the come off disk, rather than later on in the code when we can't tell if the problem is a code bug or an IO error. So, again, the verifiers are doing exactly what they were intended to do.
Quoted 2 lines
We also don't get a block number or inode number, although you or I can dig the inode number out of the hexdump, in this case.
That comes from the higher layer error message. We don't get it from the verifier simply because the boilerplate code doesn't report it.
Quoted 3 lines
We also don't get any details of what the values in the failed check were; not from the check macro itself or from the hexdump, necessarily, since it only prints the first handful of bytes.
In most cases, the handful (64) of bytes is more than sufficient - it is big enough to contain the entire self-describing header for the object that failed, and that is enough to validate whether the corruption is a bad metadata block or something internal to the metadata structure itself. i.e. the hexdump has actually been carefully sized to balance between scary noise and useful for debugging.
That said, we need to do some work on the verifiers - they need to be converted to use WANT_CORRUPTED_RETURN or a similar new corruption report. That way we know exactly what verifier test failed from the line of code it dumped from. A couple of the verifiers already do this (in the directory code), but the rest need to be converted across, too. We can easily add a more custom info to the failure by doing this (e.g. block number, whether it is a read or write verifier failure, etc); if we do this correctly then the stack trace that is currently being dumped can go away.
We also need to distinguish between CRC validation errors and object format validation errors. We need this in userspace for xfs_repair, and it could replace the custom code in xfs_db that does this, so the kernel code needs to have it put in place first.
IOWs, there's a bunch of verifier improvements that are in the works that should help this situation.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
On 12/11/2013 6:19 PM, Chris Murphy wrote:
...
Quoted 11 lines
I suspect we've only just begun to see the myriad ways in which SSDs could fail. I ran across this article earlier today:
http://techreport.com/review/25681/the-ssd-endurance-experiment-testing-data-retention-at-300tbWhat I thought was eye opening was a hashed file failing multiple
times in a row with *different* hash values, being allowed to rest unpowered for five days and then passing. Eeek. Talk about a great setup for a lot of weird transient problems with that kind of reversal. What I can't tell is if there were read errors report to the SATA driver, or if (different) bad data from a particular page was sent to the driver.
The drive that exhibited this problem, the Samsung 840, is (one of) the first on the market to use triple level cell NAND. The drive is marketed at consumers only. The anomaly occurred after 100 TB of writes, well beyond what is expected for a consumer drive. After the anomaly occurred the drive ran flawlessly up to 300 TB.
The rest of the drives, including the Samsung 840 Pro, use two cell MLC NAND, and none of them have shown problems in their testing. They've been flawless. So I disagree with your statement "we've only just begun to see the myriad ways in which SSDs could fail".
What we have here is what we've always had. A manufacturer using a bleeding edge technology didn't have all the bugs identified and fixed with the first rev of the product. This isn't a problem with SSDs in general, but one manufacturer, one new drive model, using a brand new NAND type.
Signature · 6 lines
Stan _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs